Configuration files for services #32336

aaronlehmann · 2017-04-04T01:43:49Z

This adds a docker config subcommand and related APIs that allows "config" objects to be attached to services. Config files can be mounted inside services' containers, avoiding the need to bake configuration into images.

Configuration files are similar to secrets, and in fact the CLI and API show few differences between the two. The principal differences so far are:

Secrets are always redacted at the API level, so the payload cannot be obtained through an API call after they are created.
Secrets are restricted to the /run/secrets directory inside the container, as a design choice. Config files can be mounted anywhere.

We are expecting to introduce some additional features for configs in the future that may cause the feature to diverge further from secrets. Although secrets and configs are very similar right now, we decided that creating a parallel API and subcommand was better than overloading secrets for use by things that do not have the same confidentiality requirements.

I suspect one thing that will come up in the review process is duplication of code between secrets and configs. This is definitely an issue, and I would have liked to reduce the level of duplication. However, with separate APIs and CLI subcommands, a certain amount of code duplication is unavoidable. Since the APIs use different types, and Go doesn't support generics, it's difficult to reuse code between the two implementations. Also, I didn't want to prematurely refactor things, since it's very possible that configs and secrets will diverge more soon (perhaps even during the review process). But if there are any suggestions for easy wins on reducing duplicated code, they are certainly welcome.

This feature is wanted for 17.05.

Marked WIP because this depends on moby/swarmkit#2036 (which should be merged shortly).

ping @dhiltgen @dnephin @aluzzardi @diogomonica

dnephin · 2017-04-05T15:34:38Z

I would rather see secrets implemented as a --secret flag on config to enable the "secret behaviour". Having two objects that are nearly identical is really unfortunate. I'm not sure what we do about this now that secrets already exist. Maybe docker secrets becomes a hidden alias for docker config which sets that flag?

Secrets are always redacted at the API level, so the payload cannot be obtained through an API call after they are created.

Unless I run this ?

docker service create --secret foo --name printsecrets alpine:3.5 cat /run/secrets/foo
docker service logs printsecrets

Secrets are restricted to the /run/secrets directory inside the container, as a design choice. Config files can be mounted anywhere.

Do you know where I can learn more about this design choice?

diogomonica · 2017-04-05T17:16:22Z

@dnephin there are a lot of things packed in there. Ultimately we want to make sure people deal with sensitive data in a different way than they deal with configs.

Having two distinct cmd-line arguments and two distinct APIs allows us to separate the roles of managing secrets from managing configs. It's easy to imagine a team being in charge of secrets and another team just being responsible for their own configs.
Making secrets restrictive where they can be found creates consistency of where they live across containers, and allows security teams to reason better about permissions/access to sensitive material inside of the container.
Configs will diverge significantly from secrets. It doesn't make a lot of sense to template a secret, but it makes sense to template a config, for example. Also it allows us to have different experiences to deal with sensitive data (secret data is never returned to the user after set), than we have with configs (you can always inspect the material inside.

I agree that a lot of these reasons are nuanced, but we feel like separating them is the right way to move forward.

diogomonica · 2017-04-05T21:13:28Z

container/container_unix.go

@@ -168,6 +169,21 @@ func (container *Container) SecretMountPath() string {
 	return filepath.Join(container.Root, "secrets")
 }

+// ConfigsDirPath returns the path to the directory where configs are stored on
+// disk.
+func (container *Container) ConfigsDirPath() string {


Wouldn't this be /configs/?

This is where they are stored on the host filesystem, for mounting into the container.

diogomonica · 2017-04-05T21:14:59Z

container/container_unix.go

+}
+
+// ConfigFilePath returns the path to the on-disk location of a config.
+func (container *Container) ConfigFilePath(configRefTarget *swarmtypes.ConfigReferenceFileTarget) (string, error) {


Weren't configs supposed to be mountable anywhere?

This is referring to the path on the host. It gets bind-mounted at the target path inside the container.

aaronlehmann · 2017-04-05T21:29:52Z

moby/swarmkit#2036 was merged upstream. Updated to incorporate proper swarmkit vendor. No longer marked WIP.

dnephin · 2017-04-06T18:45:03Z

secret data is never returned to the user after set

Sure it is, we've just made it slightly inconvenient.

docker service create --secret foo --name printsecrets alpine:3.5 cat /run/secrets/foo
docker service logs printsecrets

diogomonica · 2017-04-06T22:01:27Z

@dnephin I think you would agree those are separate concerns. Not allowing accidental secret retrieval by using the API (specially on the list secrets endpoint, which would return all secrets in plaintext), is not the same thing as a user intentionally deploying services with legitimate secret access to exfiltrate them.

aaronlehmann · 2017-04-06T23:43:22Z

I've rebased and marked configs as an experimental feature. The API endpoints, CLI subcommand, flags for service create / service update, and integration tests only run in experimental mode. PTAL

dnephin · 2017-04-07T17:44:22Z

@dnephin I think you would agree those are separate concerns.

Sorry, separate from what? I was responding to this statement "secret data is never returned to the user after set", which I believe I've demonstrated to be incorrect. We do allow secret data to be returned, we've just made it (afaik unnecessarily) inconvenient.

Not allowing accidental secret retrieval by using the API (specially on the list secrets endpoint, which would return all secrets in plaintext)

Why would the list endpoint returns secret text? Most list endpoints do not return full data. They only return "summary data". I don't think the "list secret" endpoint needs to change. It would be equally appropriate for a "list configs" endpoint as it is now.

... is not the same thing as a user intentionally deploying services with legitimate secret access to exfiltrate them.

I agree, but how accidental is docker secret inspect foo ? That seems like it's just as intentional.

This was just a response to "Also it allows us to have different experiences to deal with sensitive data", which is what I'm challenging.

I think for many users the distinction between a config and a secret is not clear. Many popular web frameworks expect credentials to be read from their config file. If users have passwords in their config files they should be using secrets, but they may also want to template that same config file. So I think it's quite possible that users will want to template secrets as well.

Having two distinct cmd-line arguments and two distinct APIs allows us to separate the roles of managing secrets from managing configs.

Even if this was a common case, it does not require two separate API endpoints or cmd-line arguments. Users are free to implement whatever process makes sense to them.

Configs will diverge significantly from secrets

So far I don't see an argument for this:

hiding them from the user - easily bypassed with a service create/logs, so not really significant (unless I'm misunderstanding this point)
templating - just as easily applied to "configs" that contain sensitive data , which would be considered "secrets"

dnephin · 2017-04-07T18:01:35Z

Another issue that came up during the maintainers PR review session (that I guess wasn't mentioned here yet) is that docker config doesn't really do what you would expect. I would have expected that command to configure docker or dockerd, not just services.

Couldn't we just use secrets as they are now. Add a proper secret inspect and if/when the implementation actually requires some significant difference between these types we can add --sensitive=false (or some similar flag) to change the behaviour? Maybe that's already the case with the path restrictions?

aluzzardi · 2017-04-07T18:24:56Z

Couldn't we just use secrets as they are now. Add a proper secret inspect and if/when the implementation actually requires some significant difference between these types we can add --sensitive=false (or some similar flag) to change the behaviour? Maybe that's already the case with the path restrictions?

Secrets and Configs are going to diverge more and more
Configs will be supporting templates and things like that
Secrets will support types
Having config secrets doesn't make sense. We could have had secret configs, but it's way too late now.

aaronlehmann · 2017-04-07T19:29:39Z

Added config CLI unit tests inspired by #32289.

cpuguy83 · 2017-04-07T19:57:41Z

Having these as experimental I think makes sense. It gives us time to let people test the feature before 17.06-stable and potentially pull it if it's not quite right.
Other than object/subcommand naming, I think it's good.

One suggestion made by @thaJeztah I think was docker stash (though that's a verb I would expect to be docker sash <name> -< <data>).
Not sure what else to call it.

diogomonica · 2017-04-07T20:48:04Z

Sorry, separate from what? I was responding to this statement "secret data is never returned to the user after set", which I believe I've demonstrated to be incorrect. We do allow secret data to be returned, we've just made it (afaik unnecessarily) inconvenient.

No, you demonstrated that you can deploy an app that can exfiltrate secrets that have been handed to it, you didn't demonstrate the API allows you to leak secrets. That is what I mean by separate concerns.

If you really want to protect your swarm from maliciously deployed containers, you'll have to deploy an AuthZ plugin that only runs signed code and segment the concerns.

I agree, but how accidental is docker secret inspect foo ? That seems like it's just as intentional.

Minimizing the ways secrets can get leaked from your swarm is a worth goal.

I think for many users the distinction between a config and a secret is not clear. Many popular web frameworks expect credentials to be read from their config file. If users have passwords in their config files they should be using secrets, but they may also want to template that same config file. So I think it's quite possible that users will want to template secrets as well.

This is why we would like to template configs, so you can include secrets in them without having to handle configs themselves as sensitive.

This was just a response to "Also it allows us to have different experiences to deal with sensitive data", which is what I'm challenging.

You're absolutely entitled to have your opinion; ours (secrets team) has been that production setups always have distinct treatment/confidentiality/apis/control around what they deem to be secret data, and therefore, we should keep them separate.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>

aaronlehmann · 2017-05-11T17:25:15Z

Rebased

vieux · 2017-05-11T18:02:57Z

LGTM

mavenugo · 2017-05-11T20:23:23Z

Janky failure (TestSwarmClusterRotateUnlockKey) is unrelated and this PR has gone through multiple rebases and I think it is time to put it out of its misery and merge the PR :)

vieux · 2017-05-11T20:26:02Z

@aaronlehmann 🎉 🎉

This adds bash completion for - docker#45 - moby/moby#32336 Signed-off-by: Harald Albers <github@albersweb.de>

This adds bash completion for - docker/cli#45 - moby/moby#32336 Signed-off-by: Harald Albers <github@albersweb.de> (cherry picked from commit c40952b3058db3c6e44bca8c518d888b83878ce5) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

This adds bash completion for - docker#45 - moby/moby#32336 Signed-off-by: Harald Albers <github@albersweb.de>

This adds bash completion for - docker/cli#45 - moby/moby#32336 Signed-off-by: Harald Albers <github@albersweb.de> (cherry picked from commit c40952b3058db3c6e44bca8c518d888b83878ce5) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

This adds bash completion for - docker/cli#45 - moby/moby#32336 Signed-off-by: Harald Albers <github@albersweb.de> (cherry picked from commit c40952b) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Replicate relevant mutations to the in-memory ACID store. Readers will then be able to query container state without locking. Signed-off-by: Fabio Kung <fabio.kung@gmail.com> (cherry picked from commit eed4c7b) Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com> Conflicts: - daemon/daemon.go: missing moby#33241 - container/container_unix.go: missing moby#32336 - container/container_windows.go, daemon/container_operations.go: minor conflicts Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

GordonTheTurtle added the status/0-triage label Apr 4, 2017

cpuguy83 added status/1-design-review and removed status/0-triage labels Apr 4, 2017

aaronlehmann force-pushed the configs branch from 63e3e4c to 9aeb5d3 Compare April 4, 2017 17:54

friism mentioned this pull request Apr 5, 2017

services: Add support for Credential Spec and SELinux #32339

Merged

diogomonica reviewed Apr 5, 2017

View reviewed changes

8000 diogomonica reviewed Apr 5, 2017

View reviewed changes

aaronlehmann force-pushed the configs branch from 9aeb5d3 to f085f69 Compare April 5, 2017 21:28

aaronlehmann changed the title ~~[WIP] Configuration files for services~~ Configuration files for services Apr 5, 2017

aaronlehmann force-pushed the configs branch from f085f69 to c6e63f2 Compare April 5, 2017 21:29

aaronlehmann force-pushed the configs branch from c6e63f2 to 6e62dd5 Compare April 6, 2017 23:40

vdemeester requested review from vdemeester, ehazlett, dnephin and thaJeztah April 7, 2017 14:10

aaronlehmann force-pushed the configs branch from 6e62dd5 to faba31f Compare April 7, 2017 19:29

vieux changed the title ~~Configuration files for services~~ [experimental] Configuration files for services Apr 7, 2017

vieux added this to the 17.05.0 milestone Apr 7, 2017

aaronlehmann added 4 commits May 11, 2017 10:08

Update CLI docs and add opts/config.go

a58cc35

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>

Add config support to executor backend

9e9fc7b

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>

Vendor CLI fork for integration tests

1f1befd

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>

Add integration test coverage for configs

d2c5b6e

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>

aaronlehmann force-pushed the configs branch from f320c99 to d2c5b6e Compare May 11, 2017 17:19

vieux merged commit 69c35da into moby:master May 11, 2017

thaJeztah added impact/api impact/changelog impact/documentation labels May 11, 2017

mdlinville mentioned this pull request Jun 2, 2017

Document service configs docker/docs#3482

Closed

albers added a commit to albers/docker-cli that referenced this pull request Jul 3, 2017

Add bash completion for docker config command family

43f8677

This adds bash completion for - docker#45 - moby/moby#32336 Signed-off-by: Harald Albers <github@albersweb.de>

thojkooi mentioned this pull request Jul 6, 2017

Support configs for Swarm mode services portainer/portainer#988

Closed

albers added a commit to albers/docker-cli that referenced this pull request Jul 7, 2017

Add bash completion for docker config command family

c40952b

This adds bash completion for - docker#45 - moby/moby#32336 Signed-off-by: Harald Albers <github@albersweb.de>

allencloud mentioned this pull request Jul 12, 2017

docker-py supports swarm config object operation docker/docker-py#1681

Closed

thaJeztah mentioned this pull request Jul 25, 2017

Feature Request: Please provides -v option for docker stack rm to remove all volumes from the stack #29158

Open

alshabib pushed a commit to alshabib/cli that referenced this pull request Aug 1, 2017

Add bash completion for docker config command family

5c901b2

This adds bash completion for - docker#45 - moby/moby#32336 Signed-off-by: Harald Albers <github@albersweb.de>

riyazdf pushed a commit to riyazdf/cli that referenced this pull request Aug 2, 2017

Add bash completion for docker config command family

3e21102

This adds bash completion for - docker#45 - moby/moby#32336 Signed-off-by: Harald Albers <github@albersweb.de>

thaJeztah mentioned this pull request Sep 16, 2017

Copying files to container on run before executing ENTRYPOINT/CMD #26944

Closed

kolyshkin mentioned this pull request May 2, 2020

backport: No container locks on docker ps projectatomic/docker#375

Merged

thaJeztah mentioned this pull request Oct 20, 2024

daemon: remove configsSupported, secretsSupported utilities #48703

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Configuration files for services #32336

Configuration files for services #32336

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Configuration files for services #32336

Configuration files for services #32336

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!