Containers can not reach the internet

Hi,

I recognized that suddenly docker containers can not reach out to the internet any longer. This causes that e.g. wordpress is not able to pull any updates anylonger or nextcloud can not install any apps.

IP forwarding is already set.

sysctl net.ipv4.conf.all.forwarding=1

Does anyone has an idea?

Warm regards!

Andy

So I checked some things. The issue might not be the DNS as resolution works.

apt-get update
Err:1 http://archive.ubuntu.com/ubuntu bionic InRelease
  Could not connect to archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out
Err:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
  Unable to connect to archive.ubuntu.com:http:
Err:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
  Unable to connect to archive.ubuntu.com:http:
Err:4 http://security.ubuntu.com/ubuntu bionic-security InRelease
  Could not connect to security.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.39), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.38), connection timed out
Reading package lists... Done
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/InRelease  Could not connect to archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease  Unable to connect to archive.ubuntu.com:http:
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease  Unable to connect to archive.ubuntu.com:http:
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease  Could not connect to security.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.39), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.38), connection timed out
W: Some index files failed to download. They have been ignored, or old ones used instead.

This is my iptables where the nextcloud container is 172.17.0.2.

sudo iptables -t nat -L -v
Chain PREROUTING (policy ACCEPT 203 packets, 14092 bytes)
 pkts bytes target     prot opt in     out     source               destination
 2804  169K DOCKER     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 119 packets, 7001 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 182 packets, 11170 bytes)
 pkts bytes target     prot opt in     out     source               destination
 1279 78907 DOCKER     all  --  any    any     anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 246 packets, 17025 bytes)
 pkts bytes target     prot opt in     out     source               destination
   18  1080 MASQUERADE  all  --  any    !docker0  172.17.0.0/16        anywhere
    0     0 MASQUERADE  all  --  any    !br-8692ab5ba72f  172.16.0.0/24        anywhere
    8   516 MASQUERADE  all  --  any    !br-ab3d4919ad39  172.24.0.0/16        anywhere
    0     0 MASQUERADE  all  --  any    !br-2bbe458bd139  172.19.0.0/16        anywhere
    4   240 MASQUERADE  all  --  any    !br-bcc782403b81  172.16.2.0/24        anywhere
    0     0 MASQUERADE  all  --  any    !br-d9cd84d2f643  172.18.0.0/16        anywhere
    0     0 MASQUERADE  all  --  any    !br-87db0b73962e  172.20.0.0/16        anywhere
    0     0 MASQUERADE  tcp  --  any    any     172.16.2.2           172.16.2.2           tcp dpt:3000
    0     0 MASQUERADE  tcp  --  any    any     172.16.0.3           172.16.0.3           tcp dpt:http-alt
    0     0 MASQUERADE  all  --  any    br0     10.1.2.0/24          anywhere
    0     0 MASQUERADE  tcp  --  any    any     172.17.0.1           172.17.0.1           tcp dpt:https
    0     0 MASQUERADE  tcp  --  any    any     172.17.0.2           172.17.0.2           tcp dpt:http
    0     0 MASQUERADE  tcp  --  any    any     172.17.0.3           172.17.0.3           tcp dpt:https
    0     0 MASQUERADE  tcp  --  any    any     172.17.0.3           172.17.0.3           tcp dpt:http
    0     0 MASQUERADE  tcp  --  any    any     172.17.0.4           172.17.0.4           tcp dpt:8777
    0     0 MASQUERADE  tcp  --  any    any     172.24.0.3           172.24.0.3           tcp dpt:http
    0     0 MASQUERADE  udp  --  any    any     172.24.0.5           172.24.0.5           udp dpt:10000
    0     0 MASQUERADE  tcp  --  any    any     172.24.0.5           172.24.0.5           tcp dpt:4443

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  br-87db0b73962e any     anywhere             anywhere
    0     0 RETURN     all  --  br-d9cd84d2f643 any     anywhere             anywhere
   21  1342 RETURN     all  --  br-bcc782403b81 any     anywhere             anywhere
    0     0 RETURN     all  --  br-2bbe458bd139 any     anywhere             anywhere
    8   504 RETURN     all  --  br-ab3d4919ad39 any     anywhere             anywhere
    0     0 RETURN     all  --  br-8692ab5ba72f any     anywhere             anywhere
 1473 88541 RETURN     all  --  docker0 any     anywhere             anywhere
    0     0 DNAT       tcp  --  !br-bcc782403b81 any     anywhere             anywhere             tcp dpt:40005 to:172.16.2.2:3000
    0     0 DNAT       tcp  --  !br-8692ab5ba72f any     anywhere             anywhere             tcp dpt:40002 to:172.16.0.3:8080
    0     0 DNAT       tcp  --  !docker0 any     anywhere             anywhere             tcp dpt:5443 to:172.17.0.1:443
    0     0 DNAT       tcp  --  !docker0 any     anywhere             anywhere             tcp dpt:40000 to:172.17.0.2:80
    0     0 DNAT       tcp  --  !docker0 any     anywhere             anywhere             tcp dpt:40003 to:172.17.0.3:443
    0     0 DNAT       tcp  --  !docker0 any     anywhere             anywhere             tcp dpt:40001 to:172.17.0.3:80
    0     0 DNAT       tcp  --  !docker0 any     anywhere             anywhere             tcp dpt:8777 to:172.17.0.4:8777
    0     0 DNAT       tcp  --  !br-ab3d4919ad39 any     anywhere             anywhere             tcp dpt:8888 to:172.24.0.3:80
    0     0 DNAT       udp  --  !br-ab3d4919ad39 any     anywhere             anywhere             udp dpt:10000 to:172.24.0.5:10000
    0     0 DNAT       tcp  --  !br-ab3d4919ad39 any     anywhere             anywhere             tcp dpt:4443 to:172.24.0.5:4443

Any help is appreciated!

Andy

Hi Andy,

from the first sight I would say you have some serious internet problems. You may check this with e.g. traceroute univention.de (install it first).

IP-Forwarding and iptables look good so far, for reference (from my test system with 1 container):

Chain PREROUTING (policy ACCEPT 4167 packets, 803K bytes)
 pkts bytes target     prot opt in     out     source               destination
 1577 96808 DOCKER     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 377 packets, 23236 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 408 packets, 26052 bytes)
 pkts bytes target     prot opt in     out     source               destination
    5   300 DOCKER     all  --  any    any     anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 408 packets, 26052 bytes)
 pkts bytes target     prot opt in     out     source               destination
   97  6109 MASQUERADE  all  --  any    !docker0  172.17.0.0/16        anywhere
    0     0 MASQUERADE  tcp  --  any    any     172.17.0.1           172.17.0.1           tcp dpt:http

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
  836 50160 RETURN     all  --  docker0 any     anywhere             anywhere
    0     0 DNAT       tcp  --  !docker0 any     anywhere             anywhere             tcp dpt:40001 to:172.17.0.1:80

Good luck!

Hi Oliver,

Thx for the reply.

Due to the lacking internet connection I downloaded and copied the .deb file into my nextcloud data folder and installed traceroute 2.1 with dpkg -i.

Here is the output of traceroute univention.de from within the container

root@nextc-12338939:/var/lib/univention-appcenter/apps/nextcloud/data/nextcloud-data# traceroute univention.de
traceroute to univention.de (78.47.199.152), 30 hops max, 60 byte packets
 1  172.17.42.1 (172.17.42.1)  0.051 ms  0.028 ms  0.023 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

This is the traceroute from the server itself

traceroute univention.de
traceroute to univention.de (78.47.199.152), 30 hops max, 60 byte packets
 1  * * *
 2  192.168.178.251 (192.168.178.251)  0.799 ms  0.818 ms  0.798 ms
 3  100.98.160.3 (100.98.160.3)  15.119 ms  13.369 ms  15.091 ms
 4  rudo7-t000-123.net.encoline.de (5.102.160.90)  13.034 ms  13.044 ms  13.251 ms
 5  rudo6-t001-122.net.encoline.de (5.102.160.84)  14.947 ms  14.947 ms  14.929 ms
 6  decix2-gw.hetzner.com (80.81.193.164)  39.985 ms *  39.354 ms
 7  core11.nbg1.hetzner.com (213.239.245.34)  42.674 ms core12.nbg1.hetzner.com (213.239.245.246)  40.426 ms  41.475 ms
 8  ex9k2.dc1.nbg1.hetzner.com (213.239.203.218)  40.816 ms ex9k2.dc1.nbg1.hetzner.com (213.239.203.214)  40.746 ms ex9k2.dc1.nbg1.hetzner.com (213.239.203.218)  40.241 ms
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

Andy

So I found the issue. We have had zeroshell running as a VM on the master ucs as a fail over load balancer between 2 internet connections. This works for the UCS and all DHCP clients without any issue. But somehow it is not working for containers. This might be an IP address range issue.

I checked everything in the zeroshell config but I can not find a reason why this is happening.

If I set the default gateway in the nextwork settings of the UCS to the zeroshell VM the UCS can reach the internet and containers can reach the whole LAN but not the internet.

If I set the default gateway in the network settings of the UCS to the hardware router the UCS can reach the internet and containers can reach the internet as well as the LAN.

Is there some setting necessary in the docker configuration in this case? As far as I know the UCS is NATing everything.

And somehow when I change the gateway of the UCS via the web frontend new clients are not receiving an IP address from the DHCP service running on the UCS. Ony a reboot of the server is solving this issue.

Andy