Prometheus fails to pull Node Exporter metrics after upgrade to UCS 5.2-2
Summary:
After upgrading to UCS 5.2-2, Prometheus running in a Docker container on a UCS system is unable to pull metrics from other UCS systems where the Prometheus Node Exporter is installed. The issue originates from missing firewall scripts and an incomplete iptables configuration due to the absence of the univention-firewall package.
Environment:
- UCS Version: 5.2-2 errata181
- Infrastructure: 9 UCS systems
- Prometheus and Grafana: Installed on UCS Backup Node
- Prometheus Node Exporter: Installed and running on all UCS systems
Systems Involved:
ucs5replica.univention.intranet
UCS: 5.2-2 errata181
Installed: pkgdb=15.0, prometheus-node-exporter=2.0.1
ucs5backup.univention.intranet
UCS: 5.2-2 errata181
Installed: admin-dashboard=5.0, prometheus=2.53.3
Problem Description:
After the update to UCS 5.2-2, Prometheus cannot pull metrics from systems running the Prometheus Node Exporter.
The Prometheus web interface shows the following error message:
Get "https://ucs5replica.univention.intranet:443/metrics-node/metrics/": context deadline exceeded
Prometheus attempts to pull metrics from all Node Exporters via HTTP. Since the Prometheus server itself runs as a Docker container, the problem likely resides in the network or firewall configuration on the UCS Backup system hosting the Prometheus container.
Investigation:
1. Verifying Connectivity with cURL
A test request can be executed directly with username:passwordon the UCS Backup system (outside the Docker container) to confirm whether metrics can be retrieved manually:
curl -u Administrator:univention https://ucs5replica.univention.intranet:443/metrics-node/metrics
Observed Output (Failure Case)
HTTP/1.1 401 Unauthorized
<html><body>
<h1>Unauthorized</h1>
<p>This server could not verify that you are authorized to access the document requested.</p>
</body></html>
In this case, resetting the password for the user used in the request resolves the authorization error.
Expected Output (Success Case)
# HELP apt_autoremove_pending Apt packages pending autoremoval.
# TYPE apt_autoremove_pending gauge
apt_autoremove_pending 0
# HELP apt_upgrades_pending Apt packages pending updates by origin.
...
This confirms that metric retrieval works when authentication and network communication are functional.
2. Inspecting Firewall Configuration
On the UCS Backup system, the following command shows missing INPUT and DOCKER chains in iptables:
iptables -vnL
Example output (incomplete):
root@ucs5backup:~# iptables -vnL
Chain INPUT (policy ACCEPT 294K packets, 73M bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1500
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1581
Chain FORWARD (policy DROP 3319K packets, 199M bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 243K packets, 49M bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (0 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-1 (0 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-2 (0 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-USER (0 references)
pkts bytes target prot opt in out source destination
This indicates that Docker-related firewall chains were not initialized properly.
Normally, the Univention firewall builds these chains based on templates and UCR variables.
3. Relevant Configuration Files and Templates
Firewall rules are built using the following scripts and templates:
/etc/security/packetfilter.d/10_univention-firewall_start.sh
/etc/security/packetfilter.d/20_docker.sh
/etc/security/packetfilter.d/20_rsyslog.sh
/etc/security/packetfilter.d/50_local.sh
/etc/security/packetfilter.d/80_univention-firewall_policy.sh
/etc/univention/templates/files/etc/security/packetfilter.d
/etc/univention/templates/files/etc/security/packetfilter.d/10_univention-firewall_start.sh
/etc/univention/templates/files/etc/security/packetfilter.d/20_docker.sh
/etc/univention/templates/files/etc/security/packetfilter.d/20_rsyslog.sh
/etc/univention/templates/files/etc/security/packetfilter.d/80_univention-firewall_policy.sh
To rebuild the firewall configuration manually:
ucr commit /etc/security/packetfilter.dsystemctl restart univention-firewall
4. UCR Variables Check
ucr search --brief security/packetfilter
security/packetfilter/defaultpolicy: REJECT
security/packetfilter/disabled: <empty>
security/packetfilter/docker/disabled: <empty>
security/packetfilter/package/.*: <empty>
security/packetfilter/package/univention-apache/tcp/443/all/en: HTTPS
security/packetfilter/package/univention-apache/tcp/443/all: ACCEPT
security/packetfilter/package/univention-apache/tcp/80/all/en: HTTP
security/packetfilter/package/univention-apache/tcp/80/all: ACCEPT
security/packetfilter/package/univention-base-files/tcp/22/all/en: SSH
security/packetfilter/package/univention-base-files/tcp/22/all: ACCEPT
security/packetfilter/package/univention-base-files/udp/123/all/en: ntp
security/packetfilter/package/univention-base-files/udp/123/all: ACCEPT
security/packetfilter/package/univention-bind/tcp/53/all/en: DNS proxy
security/packetfilter/package/univention-bind/tcp/53/all: ACCEPT
security/packetfilter/package/univention-bind/tcp/7777/all/en: DNS server
security/packetfilter/package/univention-bind/tcp/7777/all: ACCEPT
security/packetfilter/package/univention-bind/udp/53/all/en: DNS proxy
security/packetfilter/package/univention-bind/udp/53/all: ACCEPT
security/packetfilter/package/univention-bind/udp/7777/all/en: DNS server
security/packetfilter/package/univention-bind/udp/7777/all: ACCEPT
security/packetfilter/package/univention-heimdal-common/tcp/544/all/en: krsh
security/packetfilter/package/univention-heimdal-common/tcp/544/all: ACCEPT
security/packetfilter/package/univention-heimdal-kdc/tcp/464/all/en: kpasswd
security/packetfilter/package/univention-heimdal-kdc/tcp/464/all: ACCEPT
security/packetfilter/package/univention-heimdal-kdc/tcp/749/all/en: kadmin
security/packetfilter/package/univention-heimdal-kdc/tcp/749/all: ACCEPT
security/packetfilter/package/univention-heimdal-kdc/tcp/88/all/en: kerberos
security/packetfilter/package/univention-heimdal-kdc/tcp/88/all: ACCEPT
security/packetfilter/package/univention-heimdal-kdc/udp/464/all/en: kpasswd
security/packetfilter/package/univention-heimdal-kdc/udp/464/all: ACCEPT
security/packetfilter/package/univention-heimdal-kdc/udp/88/all/en: kerberos
security/packetfilter/package/univention-heimdal-kdc/udp/88/all: ACCEPT
security/packetfilter/package/univention-ldap-server/tcp/389/all/en: LDAP
security/packetfilter/package/univention-ldap-server/tcp/389/all: ACCEPT
security/packetfilter/package/univention-ldap-server/tcp/636/all/en: LDAPS
security/packetfilter/package/univention-ldap-server/tcp/636/all: ACCEPT
security/packetfilter/package/univention-ldap-server/tcp/7389/all/en: LDAP
security/packetfilter/package/univention-ldap-server/tcp/7389/all: ACCEPT
security/packetfilter/package/univention-ldap-server/tcp/7636/all/en: LDAPS
security/packetfilter/package/univention-ldap-server/tcp/7636/all: ACCEPT
security/packetfilter/package/univention-nagios-client/tcp/5666/all/en: Nagios NRPE
security/packetfilter/package/univention-nagios-client/tcp/5666/all: ACCEPT
security/packetfilter/package/univention-nfs/tcp/111/all/en: portmap
security/packetfilter/package/univention-nfs/tcp/111/all: ACCEPT
security/packetfilter/package/univention-nfs/tcp/2049/all/en: NFS
security/packetfilter/package/univention-nfs/tcp/2049/all: ACCEPT
security/packetfilter/package/univention-nfs/tcp/32765:32769/all/en: NFS related RPC daemons
security/packetfilter/package/univention-nfs/tcp/32765:32769/all: ACCEPT
security/packetfilter/package/univention-nfs/udp/111/all/en: portmap
security/packetfilter/package/univention-nfs/udp/111/all: ACCEPT
security/packetfilter/package/univention-nfs/udp/2049/all/en: NFS
security/packetfilter/package/univention-nfs/udp/2049/all: ACCEPT
security/packetfilter/package/univention-nfs/udp/32765:32769/all/en: NFS related RPC daemons
security/packetfilter/package/univention-nfs/udp/32765:32769/all: ACCEPT
security/packetfilter/tcp/.*: <empty>
security/packetfilter/udp/.*: <empty>
security/packetfilter/use_packages: <empty>
Output confirms standard UCS defaults but does not indicate any specific docker or firewall disabling settings.
Root Cause:
The issue was caused by a missing installation of the univention-firewall package, resulting in absent firewall initialization scripts. Consequently, Docker networking rules were not created, preventing outbound connections from the Prometheus container.
Inspection revealed that the package univention-firewall was not installed:
dpkg -l | grep firewall
rc univention-firewall 13.0.3 all UCS - firewall integration
As a result, the scripts:
10_univention-firewall_start.sh20_docker.sh80_univention-firewall_policy.sh
were not present in /etc/security/packetfilter.d/.
Without these scripts, the Univention firewall service could not correctly create iptables chains for Docker networking, preventing outbound communication from the Prometheus container.
The reason why the package was removed during the update is that it has a dependency on the packages univention-role-common and univention-bind.
By runningapt purge linux-image-5.10-amd64, it can happen that the package univention-role-common is removed as well, which in turn also removes univention-firewall.
This issue is being tracked in the following bug report:
https://forge.univention.org/bugzilla/show_bug.cgi?id=58827
Solution:
- Install the missing package:
univention-install univention-firewall
- Restart services:
systemctl restart univention-firewall.servicesystemctl restart docker.service
After restarting the services, Docker-related chains such as DOCKER, DOCKER-USER, and DOCKER-ISOLATION-STAGE-* should now appear.
- Verify iptables chains:
iptables -vnL | less
Chain INPUT (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
461K 145M ACCEPT 0 -- lo * 0.0.0.0/0 0.0.0.0/0
320K 88M ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT 1 -- * * 0.0.0.0/0 0.0.0.0/0
4434 266K ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
277 16620 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:7777
0 0 ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:7777
16 960 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:6669
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:544
54 3200 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:7389
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:7636
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5666
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:111
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2049
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:32765:32769
0 0 ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:111
0 0 ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:2049
0 0 ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 udp dpts:32765:32769
3796 228K ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5432
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1024
36 2160 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:135
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:137:139
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3268
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3269
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:389
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:445
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:464
206 8960 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:49152:65535
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:636
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:749
1 60 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:88
431 32756 ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:123
2357 345K ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 udp dpts:137:139
0 0 ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:389
0 0 ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:445
0 0 ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:464
17546 1502K ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:53
0 0 ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:88
0 0 ACCEPT 6 -- * * 172.17.0.0/16 0.0.0.0/0 tcp dpt:3306
0 0 ACCEPT 6 -- * * 172.16.0.0/16 0.0.0.0/0 tcp dpt:3306
40146 13M REJECT 0 -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
166K 144M DOCKER-USER 0 -- * * 0.0.0.0/0 0.0.0.0/0
166K 144M DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0
85003 136M ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0
81041 8375K ACCEPT 0 -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- * br-bc8c56713577 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER 0 -- * br-bc8c56713577 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- br-bc8c56713577 !br-bc8c56713577 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- br-bc8c56713577 br-bc8c56713577 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 305K packets, 150M bytes)
pkts bytes target prot opt in out source destination
461K 145M ACCEPT 0 -- * lo 0.0.0.0/0 0.0.0.0/0
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT 6 -- !br-bc8c56713577 br-bc8c56713577 0.0.0.0/0 172.16.1.2 tcp dpt:7600
0 0 ACCEPT 6 -- !br-bc8c56713577 br-bc8c56713577 0.0.0.0/0 172.16.1.2 tcp dpt:7600
0 0 ACCEPT 6 -- !br-bc8c56713577 br-bc8c56713577 0.0.0.0/0 172.16.1.2 tcp dpt:8180
0 0 ACCEPT 6 -- !br-bc8c56713577 br-bc8c56713577 0.0.0.0/0 172.16.1.2 tcp dpt:8180
0 0 ACCEPT 6 -- !docker0 docker0 0.0.0.0/0 172.17.0.1 tcp dpt:3000
0 0 ACCEPT 6 -- !docker0 docker0 0.0.0.0/0 172.17.0.2 tcp dpt:9090
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
81041 8375K DOCKER-ISOLATION-STAGE-2 0 -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-2 0 -- br-bc8c56713577 !br-bc8c56713577 0.0.0.0/0 0.0.0.0/0
166K 144M RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (2 references)
pkts bytes target prot opt in out source destination
0 0 DROP 0 -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 DROP 0 -- * br-bc8c56713577 0.0.0.0/0 0.0.0.0/0
81041 8375K RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
166K 144M RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
Verification
Once the firewall service is reinitialized, Prometheus successfully pulls metrics from all Node Exporter endpoints.
Example confirmation via Prometheus web interface:
- Targets are marked UP
- No further
context deadline exceedederrors
Additional Notes
- When operating UCS systems hosting Docker-based applications, ensure
univention-firewallremains installed and active. - Removing this package or missing scripts can disrupt communication between Docker containers and external services.
- Use
systemctl status univention-firewallandiptables -vnLto confirm the service is correctly managing firewall rules.