Description
Add support for multiple (named) build-contexts
Related issues:
- Add with relative path to parent directory fails with "Forbidden path" #2745 "Add with relative path to parent directory fails with "Forbidden path""
- Dockerfile ADD relative symlinks support #18789 Dockerfile ADD relative symlinks support
- Add support for specifying .dockerignore file with -i/--ignore #12886 "Add support for specifying .dockerignore file with -i/--ignore"
- probably others
Problem statement
Take the following directory structure for a project;
project
├── .dockerignore
├── Dockerfile
├── ginormous
│ ├── big-file-1
│ ├── big-file-2
│ └── big-file-xx
├── Makefile
├── service1
│ ├── Dockerfile
│ └── src
│ ├── Makefile
│ ├── source-file-1
│ ├── source-file-2
│ └── source-file-xx
├── service2
│ ├── Dockerfile
│ └── src
│ ├── Makefile
│ ├── source-file-1
│ ├── source-file-2
│ └── source-file-xx
├── common
│ └── src
│ ├── 0-various
│ ├── 1-files
│ ├── 2-used-by
│ ├── 3-service-1-and-2
└── common-2
└── src
├── 0-various
├── 1-files
├── 2-used-by
├── 3-service-1-and-2
In the above;
- the
Dockerfileat the root of the project usesginormous, andshared service1has a Dockerfile, and source-files used to build the service inservice1/srcservice2has a Dockerfile, and source-files used to build the service inservice2/src- both
service1andservice2share some code/resources, located incommonandcommon-2 - nor
service1, norservice2useginormous(a big directory)
Challenges with this example
Building service1 and 2 is a challenge;
- When building a Dockerfile, all files used have to be within the build-context. This means that in the project structure above, the only context that can be used is the root
projectdirectory. Doing so results in the entire project, includingginormousto be sent to the daemon (even though it's not used at all). Relative paths outside of the build-context cannot be used (also see Add with relative path to parent directory fails with "Forbidden path" #2745) - Similarly; when contructing/sending the build-context, docker won't resolve symlinks, and copy symlinks as-is, so putting symlinks to
commonandcommon-2insideproject1andproject2will not resolve this problem. - Only a single
.dockerignoreis supported, so it's not possible to "conditionally" exclude files (e.g. when buildingservice1, exclude theginormousandservice2directories, and vice-versa)
Proposal: allow multiple (named) build-contexts
I propose to add support for multiple build contexts, implemented as a --context flag on COPY and ADD, and a --context <name>=<path> option on the docker image build subcommand.
For example, to build project1, the Dockerfile could look like this:
FROM baseimage
# no --context option set: use the default build-context
COPY . /build/service1/src/
# use the build-context named "common"
COPY --context=common . /build/common/src/
# use the build-context named "common-2"
ADD --context=common-2 . /build/common-2/src/
RUN cd /build && make && make installWhen building the Dockerfile, docker expects two named build-contexts to be provided, in addition to the default (positional) build-context:
From within the project directory:
docker build \
-f ./service1/Dockerfile \
--context common=./common/src \
--context common-2=./common-2/src \
./service1In the above:
-f ./service1/Dockerfileis the Dockerfile used to build the image- the
./common/srcdirectory is used as build-context "common" - the
./common-2/srcdirectory is used as build-context "common-2" - the
./service1directory is used as default build-context
Only the common/src, common-2/src and service1 directories are uploaded to the daemon. All other directories are not part of the build-context, so won't be uploaded.
Similarly, when building from within the project/service1 directory:
docker build \
--context common=../common/src \
--context common-2=../common-2/src \
.- No
-fis provided, so the Dockerfile in the current directory is used to build the image. - the
../common/srcdirectory is used as build-context "common" - the
../common-2/srcdirectory is used as build-context "common-2" - the current (
.) directory is used as default build-context
Both relative and absolute paths can be used to specify the location of a build-context, so all of these are valid:
--context foo=~/go/src/github.com/foobar/foo
--context bar=/dev/sdb/share/bar/
--context bar=./some/dir
--context baz=../../../foobar
Validation
Validation: missing build-contexts
Before building (and sending the build-context), docker validates if all build-contexts are provided. If a context is missing, an error is produced, and the build is aborted. Trying to build the Dockerfile from the example above without specifying any build-context:
docker build .
Error: missing build-context "common"
Error: missing build-context "common-2"In a multi-stage build, only contexts that are required for the stages that are built should be taken into account. For example:
FROM baseimage AS stage-one
COPY --context=one /subdir/foo /target/dir
FROM busybox AS stage-two
ADD --context=two /foo.tar.gz /target
FROM scratch AS final
COPY --from=stage-one /foo /bar
COPY --from=stage-two /bar /baz
COPY --context=config /config.ini /config.iniGiven the Dockerfile above:
Building just stage-one:
docker build --target=stage-one .
Error: missing build-context "one"Building up until stage-two:
docker build --target=stage-two .
Error: missing build-context "one"
Error: missing build-context "two"Building the whole Dockerfile:
docker build .
Error: missing build-context "one"
Error: missing build-context "two"The default build context (positional argument) is never optional:
docker build --context --context two=./two
"docker build" requires exactly 1 argument.
See 'docker build --help'.
Usage: docker build [OPTIONS] PATH | URL | - [flags]
Build an image from a DockerfileValidation: unused build-contexts
Similar to unused --build-arg, specifying a build-context that is not used will produce a warning. When determining which contexts are expected/used in
Given the Dockerfile from the previous example:
docker build --target=stage-one --context --context two=./two .
Sending build context to Docker daemon 2.048kB
Step 1/4 : FROM baseimage AS stage-one
.....
Removing intermediate container edc9bb0f15dc
---> a77418a9ddad
[Warning] One or more contexts [two] were not consumed
Successfully built a77418a9ddadValidation: conflicting options
The --context and --from options cannot be combined. Using both will produce an error:
FROM baseimage AS stage-one
RUN echo foo
FROM busybox
COPY --from=stage-one --context=foo /foo /bardocker build --context=one .
Sending build context to Docker daemon 2.048kB
Error response from daemon: Dockerfile error line 5: conflicting options '--from' and '--context'The --context option can be combined with the --chown option.
Validation: context names
Build context names follow these rules:
- lowercase Alphanumeric characters (a-z, 0-9)
- punctuation symbols: dashes and underscores
- must start, and end, with an alphanumberic character
- consecutive punctuation symbols are not allowed
Specifying an invalid context-name (either on the command-line, or inside a Dockerfile) produces an error.