Open
Description
Description
dockerd option shutdown-timeout does not seem to work as expected on service shutdown.
Steps to reproduce the issue:
- setup systemd with a configuration that includes the
--shutdown-timeout=30option. In my case
ExecStart=/usr/bin/dockerd -D -H unix:///var/run/docker-${USER_NAME}.sock --userns-remap=default --exec-root=/var/run/docker/${USER_NAME} -g /home/${USER_NAME}/docker/ --pidfile=/var/run/docker-${USER_NAME}.pid --insecure-registry=dregistry:5000 --fixed-cidr=${DOCKER_NETWORK} --shutdown-timeout=30
- Run a container which is not handling SIGTERM by default, so it always gets killed.
docker run -d -ti --name=zkui goodguide/zk-web:latest
- stop
docker.servicemeasuring time statistics
time systemctl stop docker.service
real 0m11.846s
user 0m0.000s
sys 0m0.000s
Describe the results you received:
journalctl logs mention this
"start clean shutdown of all containers with a 30 seconds timeout...
but container gets stopped after 10 seconds (instead of 30).
msg="Container 23e4f3e7529471356ae0069197e77f6939d7d8176d49422c59b9b93d3afbb63b failed to exit within 10 seconds of signal 15 - using the force"
Apr 04 09:30:35 qa22 dockerd[28264]: time="2017-04-04T09:30:35.316828071-07:00" level=debug msg="Sending kill signal 9 to container
Apr 04 09:30:25 qa22 systemd[1]: Stopping Docker Application Container Engine...
Apr 04 09:30:25 qa22 dockerd[28264]: time="2017-04-04T09:30:25.315742282-07:00" level=info msg="Processing signal 'terminated'"
Apr 04 09:30:25 qa22 dockerd[28264]: time="2017-04-04T09:30:25.315887135-07:00" level=debug msg="start clean shutdown of all containers with a 30 seconds timeout..."
Apr 04 09:30:25 qa22 dockerd[28264]: time="2017-04-04T09:30:25.315980272-07:00" level=debug msg="stopping 23e4f3e7529471356ae0069197e77f6939d7d8176d49422c59b9b93d3afbb63b"
Apr 04 09:30:25 qa22 dockerd[28264]: time="2017-04-04T09:30:25.316012651-07:00" level=debug msg="Sending kill signal 15 to container 23e4f3e7529471356ae0069197e77f6939d7d8176d49422c59b9b93d3afbb63b"
Apr 04 09:30:35 qa22 dockerd[28264]: time="2017-04-04T09:30:35.316780128-07:00" level=info msg="Container 23e4f3e7529471356ae0069197e77f6939d7d8176d49422c59b9b93d3afbb63b failed to exit within 10 seconds of signal 15 - using the force"
Apr 04 09:30:35 qa22 dockerd[28264]: time="2017-04-04T09:30:35.316828071-07:00" level=debug msg="Sending kill signal 9 to container 23e4f3e7529471356ae0069197e77f6939d7d8176d49422c59b9b93d3afbb63b"
Apr 04 09:30:35 qa22 dockerd[28264]: time="2017-04-04T09:30:35.44054447-07:00" level=debug msg="containerd: process exited" id=23e4f3e7529471356ae0069197e77f6939d7d8176d49422c59b9b93d3afbb63b pid=init status=137 systemPid=3870
Apr 04 09:30:35 qa22 dockerd[28264]: time="2017-04-04T09:30:35.446673243-07:00" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"exit\", Id:\"23e4f3e7529471356ae0069197e77f6939d7d8176d49422c59b9b93d3afbb63b\", Status:0x89, Pid:\"init\", Timestamp:(*timestamp.Timestamp)(0xc421372b50)}"
Apr 04 09:30:35 qa22 dockerd[28264]: time="2017-04-04T09:30:35.447517766-07:00" level=debug msg="Revoking external connectivity on endpoint zkui (5bd33a9b2b7571fbf24661ae18fdedc0f67ab412a4bd0b1604fcb1d192809441)"
Apr 04 09:30:35 qa22 dockerd[28264]: time="2017-04-04T09:30:35.623431528-07:00" level=debug msg="Releasing addresses for endpoint zkui's interface on network bridge"
Apr 04 09:30:35 qa22 dockerd[28264]: time="2017-04-04T09:30:35.623473594-07:00" level=debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16/172.17.15.0/24, 172.17.15.19)"
Apr 04 09:30:36 qa22 dockerd[28264]: time="2017-04-04T09:30:36.092134694-07:00" level=debug msg="GetMountID id: 23e4f3e7529471356ae0069197e77f6939d7d8176d49422c59b9b93d3afbb63b -> mountID: ecfc1dff8eae821378ed76e5e6d6813f9ee733edfe635da43e0472022b241385"
Apr 04 09:30:36 qa22 dockerd[28264]: time="2017-04-04T09:30:36.092170115-07:00" level=debug msg="Cleaning up old mountid ecfc1dff8eae821378ed76e5e6d6813f9ee733edfe635da43e0472022b241385: start."
Apr 04 09:30:36 qa22 dockerd[28264]: time="2017-04-04T09:30:36.092612747-07:00" level=debug msg="Cleaning up old mountid ecfc1dff8eae821378ed76e5e6d6813f9ee733edfe635da43e0472022b241385: done."
Apr 04 09:30:36 qa22 dockerd[28264]: time="2017-04-04T09:30:36.092629491-07:00" level=debug msg="container stopped 23e4f3e7529471356ae0069197e77f6939d7d8176d49422c59b9b93d3afbb63b"
Apr 04 09:30:36 qa22 dockerd[28264]: time="2017-04-04T09:30:36.117212427-07:00" level=debug msg="Cleaning up old mountid : start."
Apr 04 09:30:36 qa22 dockerd[28264]: time="2017-04-04T09:30:36.117318455-07:00" level=debug msg="Unix socket /run/docker/libnetwork/bfa6d768c2dd0afb1c12c2e69b9c0b2781361d2de54afd8a327f0e2c024b0cab.sock doesn't exist. cannot accept client connections"
Apr 04 09:30:36 qa22 dockerd[28264]: time="2017-04-04T09:30:36.117564226-07:00" level=debug msg="Cleaning up old mountid : done."
Apr 04 09:30:36 qa22 dockerd[28264]: time="2017-04-04T09:30:36.117588043-07:00" level=debug msg="Clean shutdown succeeded"
Apr 04 09:30:36 qa22 dockerd[28264]: time="2017-04-04T09:30:36.117850074-07:00" level=info msg="stopping containerd after receiving terminated"
Apr 04 09:30:36 qa22 dockerd[28264]: time="2017-04-04T09:30:36.411530576-07:00" level=debug msg="libcontainerd: containerd health check returned error: rpc error: code = 9 desc = grpc: the client connection is closing"
Apr 04 09:30:37 qa22 systemd[1]: Stopped Docker Application Container Engine.
container stopped
Describe the results you expected:
I was expecting command time systemctl stop docker-litservices.service to finish after 30 seconds.
Even if I run the container with a container level timeout of 60s using docker run --stop-timeout=60, still time systemctl stop docker-litservices.service is shutting down after 60s (instead of 30s).
it seems that the container level timeout option stop-timeout, overpowers the shutdown-timeout dockerd value and the latter is not making any difference at all. Is this normal?
Which timeout is the dockerd/ --shutdown-time option is setting?
Additional information you deem important (e.g. issue happens only occasionally):
--shutdown-timeout exists in man/help page but is not documented in the documentation
[~]>dockerd --help | grep shut
--shutdown-timeout int Set the default shutdown timeout (default 15)
Output of docker version:
# docker --version
Docker version 17.03.1-ce, build c6d412e
Output of docker info:
[~]>docker info
Containers: 22
Running: 20
Paused: 0
Stopped: 2
Images: 6
Server Version: 17.03.1-ce
Storage Driver: aufs
Root Dir: /home/services/docker/1083040.1083040/aufs
Backing Filesystem: extfs
Dirs: 143
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
userns
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 94.6 GiB
Name: dev2
ID: J34K:LKP5:Y6A5:IWIO:KT5H:WMRH:YSZJ:6QOA:EHYP:ZLBH:XGOA:EANL
Docker Root Dir: /home/services/docker/1083040.1083040
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 135
Goroutines: 102
System Time: 2017-04-04T09:36:55.73437204-07:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Experimental: false
Insecure Registries:
dregistry:5000
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
Physical debian
SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux