Closed
Description
Description
When using Flannel VXLAN for overlay network, for ingress traffic interface is flannel.1 not docker0.
Docker V28.0 creates below rule for published port, which is blocking the traffic since the ingress traffic interface is flannel.1 for Flannel VXLAN overlay network.
# Warning: iptables-legacy tables present, use iptables-legacy to see them
33 1980 DROP tcp -- !docker0 * 0.0.0.0/0 10.1.215.5 tcp dpt:2181
VM1#
VM2# nc -zv 10.1.215.5 2181
nc: connect to 10.1.215.5 port 2181 (tcp) failed: Connection timed out
if we change the interface from docker0 to flannel.1, then Inter host container communication is working fine.
VM1# iptables -L -t raw -v -n | grep 2181
# Warning: iptables-legacy tables present, use iptables-legacy to see them
0 0 DROP tcp -- !flannel.1 * 0.0.0.0/0 10.1.215.5 tcp dpt:2181
VM1#
VM2# nc -zv 10.1.215.5 2181
Connection to 10.1.215.5 2181 port [tcp/*] succeeded!
VM2#
Below is the complete details:
VM1# docker -v
Docker version 28.0.4, build b8034c0
VM1# ./flanneld.bin -version
v0.24.2
VM1# docker run -d \
--name zookeeper \
-p 2181:2181 \
zookeeper:latest
VM1# docker exec -it b4e41c31b0a3 bash
root@b4e41c31b0a3:/apache-zookeeper-3.9.3-bin# netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:37105 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:2181 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN -
root@b4e41c31b0a3:/apache-zookeeper-3.9.3-bin# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1410
inet 10.1.215.5 netmask 255.255.255.0 broadcast 10.1.215.255
ether 3a:6d:76:38:ed:64 txqueuelen 0 (Ethernet)
RX packets 3157 bytes 39176309 (39.1 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2776 bytes 189334 (189.3 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
VM1# iptables -t raw -L -v -n |grep 2181
# Warning: iptables-legacy tables present, use iptables-legacy to see them
33 1980 DROP tcp -- !docker0 * 0.0.0.0/0 10.1.215.5 tcp dpt:2181
VM1#
VM2# nc -zv 10.1.215.5 2181
nc: connect to 10.1.215.5 port 2181 (tcp) failed: Connection timed out
VM1# iptables -t raw -L PREROUTING --line-numbers
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 DROP tcp -- anywhere search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com tcp dpt:8983
2 DROP udp -- anywhere search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com udp dpt:8983
3 DROP tcp -- anywhere search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com tcp dpt:17777
4 DROP udp -- anywhere search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com udp dpt:17777
5 DROP tcp -- anywhere search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com tcp dpt:18983
6 DROP udp -- anywhere search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com udp dpt:18983
7 DROP tcp -- anywhere 10.1.215.5 tcp dpt:2181
VM1# iptables -t raw -D PREROUTING 7
VM1# iptables -t raw -L PREROUTING --line-numbers
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 DROP tcp -- anywhere search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com tcp dpt:8983
2 DROP udp -- anywhere search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com udp dpt:8983
3 DROP tcp -- anywhere search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com tcp dpt:17777
4 DROP udp -- anywhere search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com udp dpt:17777
5 DROP tcp -- anywhere search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com tcp dpt:18983
6 DROP udp -- anywhere search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com udp dpt:18983
VM1#
VM1# sudo iptables -t raw -I PREROUTING ! -i flannel.1 -d 10.1.215.5 -p tcp --dport 2181 -j DROP
VM1# iptables -L -t raw -v -n | grep 2181
# Warning: iptables-legacy tables present, use iptables-legacy to see them
0 0 DROP tcp -- !flannel.1 * 0.0.0.0/0 10.1.215.5 tcp dpt:2181
VM1#
VM2# nc -zv 10.1.215.5 2181
Connection to 10.1.215.5 2181 port [tcp/*] succeeded!
VM2#
VM1# grep -ir vxlan *
grep: flanneld.bin: binary file matches
flannel-service.sh:local/cobalt/etcdctl.bin put /coreos.com/network/config "{ \"Network\": \"${CONTAINER_SUBNET}\", \"Backend\": {\"Type\": \"vxlan\"} }"
VM1# cat /etc/docker/daemon.json
{
"dns": ["10.178.186.45"],
"dns-opts": ["ndots:3"],
"bip": "10.1.215.1/24",
"selinux-enabled": true,
"mtu": 1410,
"ip-masq": false
}
VM1# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1410
inet 10.1.215.1 netmask 255.255.255.0 broadcast 10.1.215.255
ether 9e:0f:5e:01:1e:20 txqueuelen 0 (Ethernet)
RX packets 5391236 bytes 1859152760 (1.8 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3921247 bytes 3539225416 (3.5 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1460
inet 10.178.186.45 netmask 255.255.255.255 broadcast 0.0.0.0
ether 42:01:0a:b2:ba:2d txqueuelen 1000 (Ethernet)
RX packets 86457869 bytes 27792601309 (27.7 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 64012273 bytes 204858975727 (204.8 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1410
inet 10.1.215.0 netmask 255.255.255.255 broadcast 0.0.0.0
ether 02:a6:fd:1f:3e:9c txqueuelen 0 (Ethernet)
RX packets 211250 bytes 1721911041 (1.7 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 242377 bytes 36963774 (36.9 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
VM1# netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.178.184.1 0.0.0.0 UG 0 0 0 ens4
10.1.1.0 10.1.1.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.2.0 10.1.2.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.3.0 10.1.3.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.5.0 10.1.5.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.7.0 10.1.7.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.9.0 10.1.9.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.13.0 10.1.13.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.14.0 10.1.14.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.17.0 10.1.17.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.18.0 10.1.18.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.22.0 10.1.22.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.24.0 10.1.24.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.26.0 10.1.26.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.27.0 10.1.27.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.30.0 10.1.30.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.32.0 10.1.32.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.35.0 10.1.35.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.37.0 10.1.37.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.39.0 10.1.39.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.40.0 10.1.40.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.46.0 10.1.46.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.47.0 10.1.47.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.49.0 10.1.49.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.52.0 10.1.52.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.53.0 10.1.53.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.57.0 10.1.57.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.60.0 10.1.60.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.63.0 10.1.63.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.66.0 10.1.66.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.70.0 10.1.70.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.76.0 10.1.76.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.79.0 10.1.79.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.86.0 10.1.86.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.90.0 10.1.90.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.91.0 10.1.91.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.93.0 10.1.93.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.96.0 10.1.96.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.102.0 10.1.102.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.103.0 10.1.103.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.104.0 10.1.104.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.105.0 10.1.105.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.110.0 10.1.110.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.114.0 10.1.114.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.122.0 10.1.122.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.123.0 10.1.123.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.126.0 10.1.126.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.128.0 10.1.128.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.131.0 10.1.131.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.132.0 10.1.132.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.136.0 10.1.136.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.137.0 10.1.137.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.138.0 10.1.138.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.140.0 10.1.140.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.141.0 10.1.141.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.144.0 10.1.144.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.146.0 10.1.146.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.147.0 10.1.147.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.151.0 10.1.151.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.152.0 10.1.152.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.155.0 10.1.155.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.158.0 10.1.158.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.164.0 10.1.164.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.168.0 10.1.168.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.170.0 10.1.170.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.171.0 10.1.171.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.180.0 10.1.180.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.181.0 10.1.181.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.184.0 10.1.184.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.189.0 10.1.189.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.190.0 10.1.190.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.192.0 10.1.192.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.193.0 10.1.193.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.194.0 10.1.194.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.198.0 10.1.198.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.199.0 10.1.199.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.200.0 10.1.200.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.203.0 10.1.203.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.204.0 10.1.204.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.208.0 10.1.208.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.211.0 10.1.211.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.215.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
10.1.216.0 10.1.216.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.217.0 10.1.217.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.220.0 10.1.220.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.226.0 10.1.226.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.231.0 10.1.231.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.232.0 10.1.232.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.233.0 10.1.233.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.240.0 10.1.240.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.246.0 10.1.246.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.249.0 10.1.249.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.250.0 10.1.250.0 255.255.255.0 UG 0 0 0 flannel.1
10.178.184.1 0.0.0.0 255.255.255.255 UH 0 0 0 ens4
169.254.169.254 10.178.184.1 255.255.255.255 UGH 0 0 0 ens4
Reproduce
-
install Flannel vxlan
-
docker run zookeeper with published port
--name zookeeper \
-p 2181:2181 \
zookeeper:latest```
3. docker v28 creates below iptable rule.
```VM1# iptables -t raw -L -v -n |grep 2181
# Warning: iptables-legacy tables present, use iptables-legacy to see them
33 1980 DROP tcp -- !docker0 * 0.0.0.0/0 10.1.215.5 tcp dpt:2181
VM1#```
4. login to different host & test zookeeper container ip on port 2181 reachability, it will fail.
```VM2# nc -zv 10.1.215.5 2181
nc: connect to 10.1.215.5 port 2181 (tcp) failed: Connection timed out
Expected behavior
No response
docker version
# docker version
Client: Docker Engine - Community
Version: 28.0.4
API version: 1.48
Go version: go1.23.7
Git commit: b8034c0
Built: Tue Mar 25 15:07:11 2025
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 28.0.4
API version: 1.48 (minimum version 1.24)
Go version: go1.23.7
Git commit: 6430e49
Built: Tue Mar 25 15:07:11 2025
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.27
GitCommit: 05044ec0a9a75232cad458027ca83437aae3f4da
runc:
Version: 1.2.5
GitCommit: v1.2.5-0-g59923ef
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
# docker info
Client: Docker Engine - Community
Version: 28.0.4
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.22.0
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.34.0
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 5
Running: 5
Paused: 0
Stopped: 0
Images: 5
Server Version: 28.0.4
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
runc version: v1.2.5-0-g59923ef
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-1025-gcp
Operating System: Ubuntu 22.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 251.9GiB
Name: search3-v2-devfarm-ch-jcr3
ID: 2d4e91b6-f09e-4d40-bde4-695729d5ee3f
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: falseAdditional Info
No response
Metadata
Metadata
Assignees
Type
Projects
Status
Todo