8000 Migrate Dockerfiles from external repository by radusuciu · Pull Request #7303 · OpenMS/OpenMS · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Migrate Dockerfiles from external repository #7303

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 60 commits into from
Nov 7, 2024

Conversation

radusuciu
Copy link
Contributor
@radusuciu radusuciu commented Jan 16, 2024

Description

This PR migrates the Dockerfiles currently housed in the OpenMS/dockerfiles as initially outlined in OpenMS/dockerfiles#28. At the same time I'd like where possible optimize the images and related workflows to achieve minimal image size and build times

Some notes, outstanding questions, and issues. This is basically what I want to finish before this is ready for review/testing (though you're welcome to test and note issues ahead of time).

  • add a runtime stage which just copies all of the executables and only installs the bare minimum runtime libraries
  • fix use of OPENMS_BRANCH and OPENMS_TAG
  • pyopenms target isn't building due to missing python-dev package. I copied this straight from the original dockerfile so I assume that that must've been broken as well
  • adjust software LABEL for each stage
  • fix related workflows, including singularity builds
  • use GHA cache
  • add .dockerignore

Checklist

  • Make sure that 8000 you are listed in the AUTHORS file
  • Add relevant changes and new features to the CHANGELOG file
  • I have commented my code, particularly in hard-to-understand areas
  • New and existing unit tests pass locally with my changes
  • Updated or added python bindings for changed or new classes (Tick if no updates were necessary.)

@radusuciu radusuciu marked this pull request as draft January 16, 2024 18:07
@radusuciu
Copy link
Contributor Author

Two questions that haven't been addressed in the previous PRs/discussions:

  1. Is the addition of a top-level dockerfiles folder okay?
  2. Do we want to bring in the KNIME related dockerfiles as well?

@jpfeuffer, @timosachsenberg

this is used in used for two things:
- selecting the base contrib image version
- selecting the branch from the OpenMS repository
@radusuciu
Copy link
Contributor Author

So I've made good progress on most of the items outlined so far but the Singularity conversion isn't as simple as I was hoping. Though Singularity also has this concept, and spython converts it mostly fine (mostly because it doesn't do a good job of propagating ARG defaults between stages), I don't think there's an equivalent to docker build --target STAGE. Recent versions of Singularity have better support for running docker images directly without having to re-build but I think this requires the user to use the --oci option when running the container, which is not ideal. I have a few more ideas to try but please let me know if you have any thoughts!

@radusuciu
Copy link
Contributor Author

Update on the singularity issue mentioned above: instead of converting the Dockerfile to a Singularity .def, I'm attempting to convert the already built docker image directly: sudo singularity build library.sif docker://ghcr.io/${{ needs.deploy-docker.outputs.downcase_repo }}-library:${{ needs.deploy-docker.outputs.tag_name }} and have adjusted the workflows accordingly so that the singularity build job depends on the docker build job (and re-uses some metadata where possible).

I am not 100% sure if this is equivalent - this needs some testing. If this does not work then I see two choices:

  1. Abandon the docker multi-stage build. I would prefer to avoid this as this is a perfect application of a multi-stage build.
  2. Use spython to select the stages before converting to .def. I've tested this and it works -- the more annoying issue is that the way I'm using ARG in the new multi-stage Dockerfile will require some additional patching. This was already done in the existing workflow with sed so it's not too big of a blocker, but I'd prefer to avoid this if possible.

@jpfeuffer
Copy link
Contributor

There is no way to have build args or environment vars for singularity. And yes apparently no way to run a single stage.

I'm a bit hesitant of converting the ready images. It was a bit of the reason why we build them here. It sometimes failed. Not sure if it was because of on-the-fly conversions at the user and their environments but could also be due to incompatibilitied in the docker layers. I'm not sure.. might work. We could try..

An alternative could be to have a "from-scratch" singularity def file template that can be configured with jinja templates or something (for both args and stages). In this case, we should move the build steps to small scripts that will be mounted and executed, such that we do not have to maintain both files.

@radusuciu
Copy link
Contributor Author
radusuciu commented Jan 18, 2024

Thank you for the context - I'll see if I can put something together, and if it doesn't work out maybe we can scale back the optimization ambitions and just do a direct migration, or you/others can take a stab at it.

Also, I should mention that I don't have much experience with singularity and don't understand deeply enough why converting an already built docker image might be flaky (and what the pitfalls are..), so I trust your experience of course. I did try and convert the current images though, and can verify that it does work or at least, I encounter no errors when doing the following:

docker run --rm -v $PWD:/go quay.io/singularity/singularity:v3.11.5 build executables.sif docker://ghcr.io/openms/openms-executables
docker run --rm -v $PWD:/go -it --privileged quay.io/singularity/singularity:v3.11.5 run executables.sif IsobaricAnalyzer --help

(yes, I'm running singularity in docker lol)

Is there a chance that newer versions of singularity are better in this regard? It does seem that docker/OCI compatibility has been a priority for them.

@jpfeuffer
Copy link
Contributor

I guess we can try with the converted containers and come up with a solution if they fail. Apparently, according to your external issue, they are pretty confident about the conversion process.

@radusuciu
Copy link
Contributor Author
radusuciu commented Jan 19, 2024

I guess we can try with the converted containers and come up with a solution if they fail. Apparently, according to your external issue, they are pretty confident about the conversion process.

I tried to learn a bit more about how it works, I think the layer flattening should be pretty robust. There's also an open issue with a kinda old PR for building directly SIF from Dockerfiles. Basically I think if we have issues it'd also be worth raising them with the Singularity project.

I would like to test the conversion process in a GitHub action though, in my fork. One thing that I can think of that would lead to sporadic failures would be worker diskspace.. I've hit this a few times when building larger containers and had to either slim them down or spread them across different jobs.

EDIT: The conversion/build works, and takes ~7 minutes: https://github.com/radusuciu/OpenMS/actions/runs/7587340359/job/20667464502

@jpfeuffer
Copy link
Contributor

Yep I think the disk space and permissions on clusters is what made us build them here for other people so they don't have to do it on the fly. I guess if we can convert in a controlled environment, such as here it should really be fine. I'm very happy to merge. Happy to see that it works out on your fork as well.

@radusuciu
Copy link
Contributor Author
radusuciu commented Jan 19, 2024

Alright, cool! I'm now focusing on seeing if there's any fat to trim from the images. For example, the main repository .git and THIRDPARTY submodules (note: everything needed for the container is already copied to /thirdparty) which total up to ~2GB in the current executables image:

$ docker run -it --rm ghcr.io/openms/openms-executables:3.1.0 du -sh /OpenMS/{THIRDPARTY,.git}/
1.1G    /OpenMS/THIRDPARTY/
970M    /OpenMS/.git/

That's out of a total of 5.3GB so that's a pretty big chunk:

$ docker image inspect -f "{{ .Size }}" ghcr.io/openms/openms-executables:3.1.0 | numfmt --to=si
5.3G

@radusuciu
Copy link
Contributor Author

Before PR is ready for review, I had a few questions for you (or any other maintainers):

  1. Is there any test-suite we can run on the executables/tools alone? I'm asking because I've already played around with a minimal runtime stage which is just executables and dependencies from apt (ie. no -dev packages, no unnecessary build tools), and while I've confirmed that they run without errors, I would love if it I could run the same test suite that is run during build (this would also be a separate stage)
  2. Are you currently doing anything with the pyOpenMS docker container: https://github.com/OpenMS/dockerfiles/blob/master/pyOpenMS/Dockerfile. If not, I will not migrate this, though I'm happy to contribute a separate PR later if there is a potential use.
  3. Are you currently doing anything with the KNIME docker containers: https://github.com/OpenMS/dockerfiles/tree/master/KNIME. If not, I will not migrate this, though I'm happy to contribute a separate PR later if there is a potential use.
  4. I forget what motivated me but I introduced an alternative tag for openms-executables: openms-tools. Do you have any opinion there?

@timosachsenberg
Copy link
Contributor
timosachsenberg commented Jan 22, 2024
  1. Is there any test-suite we can run on the executables/tools alone? I'm asking because I've already played around with a minimal runtime stage which is just executables and dependencies from apt (ie. no -dev packages, no unnecessary build tools), and while I've confirmed that they run without errors, I would love if it I could run the same test suite that is run during build (this would also be a separate stage)

hmm I think this is not easily done because installing tests is a bit non-standard for cmake projects.
The test project is created here https://github.com/OpenMS/OpenMS/blob/develop/src/tests/topp/CMakeLists.txt
And it seems to me it is not easy to pull that out of the project.

What others have done: they take one or few of the tests (preferably of a stable tool, maybe OpenMSInfo) and execute it in the container to check if loading of libraries etc. works.

Edit: fixed sentence

@jpfeuffer
Copy link
Contributor
jpfeuffer commented Jan 22, 2024

You can just copy the test executables and run them. Some of them will have hard coded paths to compare outputs to expected outputs. Those files must be copied over to the same path. Not sure if those were relative paths. Other tests will work out of the box.

You can also copy over all ctest related files and with Cmake 3.20 run ctest --test-dir new_dir_on_target_machine. Figuring out what you need might be non-trivial. We never tried this. It would be cool to have a Cmake install configuration that installs expected test files as well but I think it's a bit of work.

I wouldn't say our tests are non-standard at all.
I think testing a full test suite on a target (non-build) machine is not toooo common yet and not well supported in CMake in general (until maybe recently)

@radusuciu
Copy link
Contributor Author
radusuciu commented Jan 22, 2024

Thanks for the responses, I think I've got it! I've reduced the tools/executables runtime to below 1GB.

$ docker image inspect -f "{{ .Size }}" tools-slim | numfmt --to=si
911M

The relevant stage:

FROM ubuntu:22.04 AS tools-slim
ARG OPENMS_LIBRARY_DIR
ARG OPENMS_LIBRARY_BUILD_DIR
ARG CONTRIB_BUILD_DIR
ARG OPENMS_LIBRARY_DIR

RUN apt-get update \
  && apt-get install -y --no-install-recommends --no-install-suggests \
    libqt5opengl5 \
    libsvm3 \
    libzip4 \
    zlib1g \
    libbz2-1.0 \
    libgomp1 \
    libqt5svg5 \
    libxerces-c3.2 \
    libboost-date-time1.74-dev  \
    libboost-iostreams1.74-dev \
    libboost-regex1.74-dev \
    libboost-math1.74-dev \
    libboost-random1.74-dev \
  && rm -rf /var/lib/apt/lists/*

COPY --from=tools ${CONTRIB_BUILD_DIR}/bin ${CONTRIB_BUILD_DIR}/bin
COPY --from=tools /thirdparty /thirdparty
COPY --from=tools ${OPENMS_LIBRARY_BUILD_DIR}/bin ${OPENMS_LIBRARY_BUILD_DIR}/bin
COPY --from=tools ${OPENMS_LIBRARY_BUILD_DIR}/lib ${OPENMS_LIBRARY_BUILD_DIR}/lib
COPY --from=tools ${OPENMS_LIBRARY_DIR}/share ${OPENMS_LIBRARY_DIR}/share

What's missing is the PATH env. Though I think it might be worthwhile to move things from the various build dirs to more standard system locations (this is where we might need to much about with ctest arguments). I'm testing in a separate stage based on tools-slim where I install cmake and the test files, and just got here:

root@US320024WSD:/openms-build# ctest -R TOPP_

[snipped stdout]

100% tests passed, 0 tests failed out of 1278

😃

I'll update PR today/tomorrow, but I think this is very promising and IMO a worthwhile optimization if it holds.

@jpfeuffer
Copy link
Contributor

Whaat? That's awesome!

a few other modifications:
- added dockerignore file
- clone thirdparty repo instead of using submodule since I've added .git to the dockergignore
@radusuciu
Copy link
Contributor Author

Okay, so the slimmed tools image is at 1.3GB, I accidently the /thirdparty before, but still, pretty good. I think we're fairly close to the minimum now - take a look at the dive layer summary:

│ Layers ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Cmp   Size  Command                                                                                                                
     78 MB  FROM 9ee4303fd391873                                                                                                   
    553 MB  RUN |5 OPENMS_LIBRARY_DIR=/OpenMS OPENMS_LIBRARY_BUILD_DIR=/openms-build CONTRIB_BUILD_DIR=/contrib-build OPENMS_LIBRAR
    9.4 MB  COPY /contrib-build/bin /contrib-build/bin # buildkit                                                                  
    370 MB  COPY /thirdparty /thirdparty # buildkit                                                                                
     19 MB  COPY /openms-build/bin /openms-build/bin # buildkit                                                                    
     49 MB  COPY /openms-build/lib /openms-build/lib # buildkit                                                                    
    202 MB  COPY /OpenMS/share /OpenMS/share # buildkit                                                                            

Notes on the above:

  • the runtime deps (553MB) are the biggest contributor, and I don't think these can be trimmed much more (note the full command is cut off but you can see it in the Dockerfile)
  • the base system is only 78MB. Earlier I suggested trying out debian-slim, but that's only a few MB smaller, so not sure that's worth it at this point
  • /OpenMS/share/OpenMS/examples is 136MB, so maybe worth ditching

All binaries/files are in the original build folders still, I'll see if I can tidy that up, but that's almost an aesthetic concern.

Copy link
Contributor
@jpfeuffer jpfeuffer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some quick comments

@jpfeuffer
Copy link
Contributor
jpfeuffer commented Jan 23, 2024

Regarding runtime binaries.
If you do not make install the binaries and libs won't be stripped. Another path to explore.
Make install will install out of the build directory though. Which could mess up your nice tests.
Maybe run your own strip.
Pros: Smaller binaries
Cons: probably no function info in stack traces etc.

radusuciu added a commit to radusuciu/docker-openms that referenced this pull request Feb 7, 2024
radusuciu and others added 4 commits February 8, 2024 09:41
it is sufficient for caching purposes to load it
runners have varying amounts of free space, and these images can be quite big and sometimes juuuust fail to build if the runner happens to have less free space.
@radusuciu
Copy link
Contributor Author

Thanks for keeping the PR up to date. It's pretty much ready for merge but I would recommend a final review. I've been using a similar image that I build here https://github.com/radusuciu/docker-openms/ without problems.

There are two things I'd call your attention to:

  1. I added a new tag openms-tools, but kept the original as well openms-executables.. maybe you prefer to not change this or to not have two.
  2. I have not tested this latest version with a release tag. In this case, the workflow builds the contrib image. I can test this to make sure there are no issues.

@timosachsenberg
Copy link
Contributor

looks very good to me in general.
One part where I am a bit lost is the contrib.
In the Dockerfile we use system packages and in the action we use the submodule.
Is there a reason why this is done this way e.g. to preserve both ways to provide dependencies?

@radusuciu
Copy link
Contributor Author

looks very good to me in general. One part where I am a bit lost is the contrib. In the Dockerfile we use system packages and in the action we use the submodule. Is there a reason why this is done this way e.g. to preserve both ways to provide dependencies?

Hey, sorry for delay, I just got back from a trip.

This was mainly for easy testing prior to OpenMS/contrib#136 being merged. It might still be helpful to have as a stage in the same Dockerfile as opposed to a separate Dockerfile in the sub-module when how changes to the contrib packgages affect builds of the images here.. but this also introduces the risk of some confusion for future maintenance.

I don't feel super-strongly, but I'm inclined to say let's remove it.

@timosachsenberg
Copy link
Contributor

@jpfeuffer what is your call here?

@jpfeuffer
Copy link
Contributor
jpfeuffer commented Apr 9, 2024

If you want to avoid contrib containers you can also build remaining dependencies in the corresponding action (CI, pyopenms, this docker action)
Even better, add it to cmake and build automatically if not found, as we do for json and SQLite.
You can also cache the deps but our caches are already at the limit due to compiler caches

@jpfeuffer
Copy link
Contributor
jpfeuffer commented Apr 9, 2024

Ah I misremembered the end of the thread here. I think you can remove.
The inability to get the full contrib on manylinux does not stop us from building the Ubuntu containers here.

(For which we do not need any contrib base containers anymore)

@radusuciu
Copy link
Contributor Author
radusuciu commented Apr 30, 2024

I've moved the unused contrib stage from the Dockerfile. Some of the points I raised in the comment above might still warrant comment -- let me know or we can merge and deal with them separately.

Thanks!

@poshul
Copy link
Collaborator
poshul commented Sep 5, 2024

Is this ready to merge? I'm trying to set up devcontainer image that we can use for development and came across this (which is going to be used as a basis for my setup)

@radusuciu
Copy link
Contributor Author

Quoting myself

There are two things I'd call your attention to:

  1. I added a new tag openms-tools, but kept the original as well openms-executables.. maybe you prefer to not change this or to not have two.

This is minor, and well covered in the README I included in the containing directory: https://github.com/OpenMS/OpenMS/blob/54a90800c60e9ffd9a1ec250671480277a25bb74/dockerfiles/README.md

  1. I have not tested this latest version with a release tag. In this case, the workflow builds the contrib image. I can test this to make sure there are no issues.

I just tagged a fake release, let's see if it runs https://github.com/radusuciu/OpenMS/actions/runs/10724213483

@radusuciu
Copy link
Contributor Author

Not necessarily related to the PR but looks like we failed two tests

#37 31.24           Start 2122: TOPP_THERMORAWFILEPARSER_1
#37 31.66  652/2353 Test #2122: TOPP_THERMORAWFILEPARSER_1 .................................***Failed    0.43 sec
#37 31.67 RawFileReader reading tool. Copyright 2016 by Thermo Fisher Scientific, Inc. All rights reserved
#37 31.67 Standard output: Running: mono /opt/OpenMS/thirdparty/ThermoRawFileParser/ThermoRawFileParser.exe -i=/tmp/OpenMS/src/tests/topp/THIRDPARTY/ginkgotoxin-ms-switching.raw --output_file=ginkgotoxin-ms-switching_out_tmp.mzML -f=2 -e
#37 31.67 
#37 31.67 Standard error: Unexpected extra arguments
#37 31.67 Error - usage is:
#37 31.67   -h, --help                 Prints out the options.
#37 31.67       --version              Prints out the version of the executable.
#37 31.67   -i, --input=VALUE          The raw file input (Required). Specify this or an
#37 31.67                                input directory -d.
#37 31.67   -d, --input_directory=VALUE
#37 31.67                              The directory containing the raw files (Required).
#37 31.67                                Specify this or an input raw file -i.
#37 31.67   -b, --output=VALUE         The output file. Specify this or an output
#37 31.67                                directory -o. Specifying neither writes to the
#37 31.67                                input directory.
#37 31.67   -o, --output_directory=VALUE
#37 31.67                              The output directory. Specify this or an output
#37 31.67                                file -b. Specifying neither writes to the input
#37 31.67                                directory.
#37 31.67   -s, --stdout               Write to standard output. Cannot be combined with
#37 31.67                                file or directory output. Implies silent logging,
#37 31.67                                 i.e. logging level 0
#37 31.67   -f, --format=VALUE         The spectra output format: 0 for MGF, 1 for mzML,
#37 31.67                                2 for indexed mzML, 3 for Parquet, 4 for None (
#37 31.67                                no output); both numeric and text (case
#37 31.67                                insensitive) value recognized. Defaults to
#37 31.67                                indexed mzML if no format is specified.
#37 31.67   -m, --metadata=VALUE       The metadata output format: 0 for JSON, 1 for TXT,
#37 31.67                                2 for None (no output); both numeric and text (
#37 31.67                                case insensitive) value recognized. Defaults to
#37 31.67                                None
#37 31.67   -c, --metadata_output_file=VALUE
#37 31.67                              The metadata output file. By default the metadata
#37 31.67                                file is written to the output directory.
#37 31.67   -g, --gzip                 GZip the output file.
#37 31.67   -p, --noPeakPicking[=VALUE]
#37 31.67                              Don't use the peak picking provided by the native
#37 31.67                                Thermo library. By default peak picking is
#37 31.67                                enabled. Optional argument allows disabling peak
#37 31.67                                peaking only for selected MS levels and should
#37 31.67                                be a comma-separated list of integers (1,2,3)
#37 31.67                                and/or intervals (1-3), open-end intervals (1-)
#37 31.67                                are allowed
#37 31.67   -z, --noZlibCompression    Don't use zlib compression for the m/z ratios and
#37 31.67                                intensities. By default zlib compression is
#37 31.67                                enabled.
#37 31.67   -a, --allDetectors         Extract additional detector data: UV/PDA etc
#37 31.67   -l, --logging=VALUE        Optional logging level: 0 for silent, 1 for
#37 31.67                                verbose, 2 for default, 3 for warning, 4 for
#37 31.67                                error; both numeric and text (case insensitive)
#37 31.67                                value recognized.
#37 31.67   -e, --ignoreInstrumentErrors
#37 31.67                              Ignore missing properties by the instrument.
#37 31.67   -x, --excludeExceptionData Exclude reference and exception data
#37 31.67   -L, --msLevel=VALUE        Select MS levels (MS1, MS2, etc) included in the
#37 31.67                                output, should be a comma-separated list of
#37 31.67                                integers (1,2,3) and/or intervals (1-3), open-
#37 31.67                                end intervals (1-) are allowed
#37 31.67   -P, --mgfPrecursor         Include precursor scan number in MGF file TITLE
#37 31.67   -N, --noiseData            Include noise data in mzML output
#37 31.67   -w, --warningsAreErrors    Return non-zero exit code for warnings; default
#37 31.67                                only for errors
#37 31.67   -u, --s3_url[=VALUE]       Optional property to write directly the data into
#37 31.67                                S3 Storage.
#37 31.67   -k, --s3_accesskeyid[=VALUE]
#37 31.67                              Optional key for the S3 bucket to write the file
#37 31.67                                output.
#37 31.67   -t, --s3_secretaccesskey[=VALUE]
#37 31.67                              Optional key for the S3 bucket to write the file
#37 31.67                                output.
#37 31.67   -n, --s3_bucketName[=VALUE]
#37 31.67                              S3 bucket name
#37 31.67 Process 'mono' did not finish successfully (exit code: 255). Please check the log.

and

#37 97.67 2195/2353 Test #2123: TOPP_THERMORAWFILEPARSER_1_out .............................***Failed    0.02 sec
#37 97.67 Warning: Parameters file version (3.2.0) does not match the version of this tool (3.2.0-pre-HEAD-2024-09-05).
#37 97.67 Your current parameters are still valid, but there might be new valid values or even new parameters. Upgrading the INI might be useful.
#37 97.67 Cannot read input file given from parameter '-in1'!
#37 97.67 Error: File not found (the file 'ginkgotoxin-ms-switching_out_tmp.mzML' could not be found)
#37 97.67 

@poshul
Copy link
Collaborator
poshul commented Nov 7, 2024

So the error is due to a change in the command line params to ThermoRawFileParser we just need to make sure to that the test cmake file matches the correct version of the executable.

@poshul poshul merged commit 9be7d02 into OpenMS:develop Nov 7, 2024
12 checks passed
@timosachsenberg
Copy link
Contributor
timosachsenberg commented Nov 15, 2024

@poshul what changed?

  • Needs to be fixed in cmake scripts
  • This probably needs to be fixed in FileConverter as well.

Deployment fails currently

@timosachsenberg
Copy link
Contributor

fixed in #7673

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants
0