Hi,
I recognized that suddenly docker containers can not reach out to the internet any longer. This causes that e.g. wordpress is not able to pull any updates anylonger or nextcloud can not install any apps.
IP forwarding is already set.
sysctl net.ipv4.conf.all.forwarding=1
Does anyone has an idea?
Warm regards!
Andy
So I checked some things. The issue might not be the DNS as resolution works.
apt-get update
Err:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Could not connect to archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out
Err:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Unable to connect to archive.ubuntu.com:http:
Err:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Unable to connect to archive.ubuntu.com:http:
Err:4 http://security.ubuntu.com/ubuntu bionic-security InRelease
Could not connect to security.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.39), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.38), connection timed out
Reading package lists... Done
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/InRelease Could not connect to archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease Unable to connect to archive.ubuntu.com:http:
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease Unable to connect to archive.ubuntu.com:http:
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease Could not connect to security.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.39), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.38), connection timed out
W: Some index files failed to download. They have been ignored, or old ones used instead.
This is my iptables where the nextcloud container is 172.17.0.2.
sudo iptables -t nat -L -v
Chain PREROUTING (policy ACCEPT 203 packets, 14092 bytes)
pkts bytes target prot opt in out source destination
2804 169K DOCKER all -- any any anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 119 packets, 7001 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 182 packets, 11170 bytes)
pkts bytes target prot opt in out source destination
1279 78907 DOCKER all -- any any anywhere !loopback/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 246 packets, 17025 bytes)
pkts bytes target prot opt in out source destination
18 1080 MASQUERADE all -- any !docker0 172.17.0.0/16 anywhere
0 0 MASQUERADE all -- any !br-8692ab5ba72f 172.16.0.0/24 anywhere
8 516 MASQUERADE all -- any !br-ab3d4919ad39 172.24.0.0/16 anywhere
0 0 MASQUERADE all -- any !br-2bbe458bd139 172.19.0.0/16 anywhere
4 240 MASQUERADE all -- any !br-bcc782403b81 172.16.2.0/24 anywhere
0 0 MASQUERADE all -- any !br-d9cd84d2f643 172.18.0.0/16 anywhere
0 0 MASQUERADE all -- any !br-87db0b73962e 172.20.0.0/16 anywhere
0 0 MASQUERADE tcp -- any any 172.16.2.2 172.16.2.2 tcp dpt:3000
0 0 MASQUERADE tcp -- any any 172.16.0.3 172.16.0.3 tcp dpt:http-alt
0 0 MASQUERADE all -- any br0 10.1.2.0/24 anywhere
0 0 MASQUERADE tcp -- any any 172.17.0.1 172.17.0.1 tcp dpt:https
0 0 MASQUERADE tcp -- any any 172.17.0.2 172.17.0.2 tcp dpt:http
0 0 MASQUERADE tcp -- any any 172.17.0.3 172.17.0.3 tcp dpt:https
0 0 MASQUERADE tcp -- any any 172.17.0.3 172.17.0.3 tcp dpt:http
0 0 MASQUERADE tcp -- any any 172.17.0.4 172.17.0.4 tcp dpt:8777
0 0 MASQUERADE tcp -- any any 172.24.0.3 172.24.0.3 tcp dpt:http
0 0 MASQUERADE udp -- any any 172.24.0.5 172.24.0.5 udp dpt:10000
0 0 MASQUERADE tcp -- any any 172.24.0.5 172.24.0.5 tcp dpt:4443
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- br-87db0b73962e any anywhere anywhere
0 0 RETURN all -- br-d9cd84d2f643 any anywhere anywhere
21 1342 RETURN all -- br-bcc782403b81 any anywhere anywhere
0 0 RETURN all -- br-2bbe458bd139 any anywhere anywhere
8 504 RETURN all -- br-ab3d4919ad39 any anywhere anywhere
0 0 RETURN all -- br-8692ab5ba72f any anywhere anywhere
1473 88541 RETURN all -- docker0 any anywhere anywhere
0 0 DNAT tcp -- !br-bcc782403b81 any anywhere anywhere tcp dpt:40005 to:172.16.2.2:3000
0 0 DNAT tcp -- !br-8692ab5ba72f any anywhere anywhere tcp dpt:40002 to:172.16.0.3:8080
0 0 DNAT tcp -- !docker0 any anywhere anywhere tcp dpt:5443 to:172.17.0.1:443
0 0 DNAT tcp -- !docker0 any anywhere anywhere tcp dpt:40000 to:172.17.0.2:80
0 0 DNAT tcp -- !docker0 any anywhere anywhere tcp dpt:40003 to:172.17.0.3:443
0 0 DNAT tcp -- !docker0 any anywhere anywhere tcp dpt:40001 to:172.17.0.3:80
0 0 DNAT tcp -- !docker0 any anywhere anywhere tcp dpt:8777 to:172.17.0.4:8777
0 0 DNAT tcp -- !br-ab3d4919ad39 any anywhere anywhere tcp dpt:8888 to:172.24.0.3:80
0 0 DNAT udp -- !br-ab3d4919ad39 any anywhere anywhere udp dpt:10000 to:172.24.0.5:10000
0 0 DNAT tcp -- !br-ab3d4919ad39 any anywhere anywhere tcp dpt:4443 to:172.24.0.5:4443
Any help is appreciated!
Andy
Hi Andy,
from the first sight I would say you have some serious internet problems. You may check this with e.g. traceroute univention.de (install it first).
IP-Forwarding and iptables look good so far, for reference (from my test system with 1 container):
Chain PREROUTING (policy ACCEPT 4167 packets, 803K bytes)
pkts bytes target prot opt in out source destination
1577 96808 DOCKER all -- any any anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 377 packets, 23236 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 408 packets, 26052 bytes)
pkts bytes target prot opt in out source destination
5 300 DOCKER all -- any any anywhere !loopback/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 408 packets, 26052 bytes)
pkts bytes target prot opt in out source destination
97 6109 MASQUERADE all -- any !docker0 172.17.0.0/16 anywhere
0 0 MASQUERADE tcp -- any any 172.17.0.1 172.17.0.1 tcp dpt:http
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
836 50160 RETURN all -- docker0 any anywhere anywhere
0 0 DNAT tcp -- !docker0 any anywhere anywhere tcp dpt:40001 to:172.17.0.1:80
Good luck!
Hi Oliver,
Thx for the reply.
Due to the lacking internet connection I downloaded and copied the .deb file into my nextcloud data folder and installed traceroute 2.1 with dpkg -i.
Here is the output of traceroute univention.de from within the container
root@nextc-12338939:/var/lib/univention-appcenter/apps/nextcloud/data/nextcloud-data# traceroute univention.de
traceroute to univention.de (78.47.199.152), 30 hops max, 60 byte packets
1 172.17.42.1 (172.17.42.1) 0.051 ms 0.028 ms 0.023 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
This is the traceroute from the server itself
traceroute univention.de
traceroute to univention.de (78.47.199.152), 30 hops max, 60 byte packets
1 * * *
2 192.168.178.251 (192.168.178.251) 0.799 ms 0.818 ms 0.798 ms
3 100.98.160.3 (100.98.160.3) 15.119 ms 13.369 ms 15.091 ms
4 rudo7-t000-123.net.encoline.de (5.102.160.90) 13.034 ms 13.044 ms 13.251 ms
5 rudo6-t001-122.net.encoline.de (5.102.160.84) 14.947 ms 14.947 ms 14.929 ms
6 decix2-gw.hetzner.com (80.81.193.164) 39.985 ms * 39.354 ms
7 core11.nbg1.hetzner.com (213.239.245.34) 42.674 ms core12.nbg1.hetzner.com (213.239.245.246) 40.426 ms 41.475 ms
8 ex9k2.dc1.nbg1.hetzner.com (213.239.203.218) 40.816 ms ex9k2.dc1.nbg1.hetzner.com (213.239.203.214) 40.746 ms ex9k2.dc1.nbg1.hetzner.com (213.239.203.218) 40.241 ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
Andy
So I found the issue. We have had zeroshell running as a VM on the master ucs as a fail over load balancer between 2 internet connections. This works for the UCS and all DHCP clients without any issue. But somehow it is not working for containers. This might be an IP address range issue.
I checked everything in the zeroshell config but I can not find a reason why this is happening.
If I set the default gateway in the nextwork settings of the UCS to the zeroshell VM the UCS can reach the internet and containers can reach the whole LAN but not the internet.
If I set the default gateway in the network settings of the UCS to the hardware router the UCS can reach the internet and containers can reach the internet as well as the LAN.
Is there some setting necessary in the docker configuration in this case? As far as I know the UCS is NATing everything.
And somehow when I change the gateway of the UCS via the web frontend new clients are not receiving an IP address from the DHCP service running on the UCS. Ony a reboot of the server is solving this issue.
Andy