Description
Description
We recently had several issues with our cloud provider which lead us to several docker swarms failing with strange errors. One thing we observied that from time to time IP adresses in the swarm overlay network are assigned duplicate.
We made several attempts to reproduce this issue and finally found a way to reproduce the duplicate ip adress issue.
Steps to reproduce the issue:
- Create a docker swarm with several nodes (five in our example)
- Start some Services in the swarm with at least two replicas
- Start some containers on swarm nodes, that are attached to the swarm overlay network
- kill the docker process (kill -9 <main_pid_from systemctl status docker>) and/or kill some service tasks running on the same node (kill -9 <main_pid_from docker inspect>)
- do a "docker network inspect -v <overlay_network>" on the node
- start and stop some services in the swarm
Describe the results you received:
I can see that ip adresses are assigned to more than one service task.
Describe the results you expected:
I would expect that every service task is assigned a unique ip adress.
Additional information you deem important (e.g. issue happens only occasionally):
What we tried to resolve this issue:
- let affected node leave the swarm (docker swarm leave, docker node demote, docker node rm)
- stop the docker daemon on the affected node
- rm -rf /var/lib/docker/*
- start the docker daemon
- rejoin the swarm
there are still duplicate ip adresses (ocker network inspect -v <overlay_network>)
next try:
- let node leave the swarm again
- stop the host
- delete the host
- create a new host (new host name, new ip address)
- let new host join swarm as manager
again, there are duplicate ip adresses which are assigned to the host that was completely removed from the swarm (see attached file overlay.txt)
overlay.txt
Output of docker version:
Client:
Version: 17.05.0-ce
API version: 1.29
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:06:25 2017
OS/Arch: linux/amd64
Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:06:25 2017
OS/Arch: linux/amd64
Experimental: false
Client:
Version: 17.05.0-ce
API version: 1.29
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:06:25 2017
OS/Arch: linux/amd64
Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:06:25 2017
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
see attached file docker_info.txt
docker_info.txt
Additional environment details (AWS, VirtualBox, physical, etc.):
Environment:
OS: CentOS 7 running in OpenStack in a public cloud.