Prometheus Node Exporter only listening on 127.0.0.1

UCS - Univention Corporate Server
#1

Hi,

on a Member Server I installed Dashboard Client. I have a non UCS Prometheus Server on which I want to scrape the metrics from UCS Servers as well.

Problem is, that Node_exporter is listening on 127.0.0.1

I added the entry manually to /etc/default/prometheus-node-exporter:

ARGS="-collector.diskstats.ignored-devices=^(ram|loop|fd)\d+$ \
      -collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run)($|/) \
      -collector.textfile.directory=/var/lib/prometheus/node-exporter \
      -web.listen-address 172.16.1.18:9100"

systemd output after that is:

   CGroup: /system.slice/prometheus-node-exporter.service
           └─1247 /usr/bin/prometheus-node-exporter -collector.diskstats.ignored-devices=^(ram|loop|fd)d+$ -collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run)($|/) -collector.textfile.directory=/var/lib/prometheus/node-exporter -web.listen-address 172.16.1.18:9100 -web.listen-address 127.0.0.1:9100 -web.telemetry-path=/metrics-node/metrics/

Sep 16 11:15:30 srv-00-18 prometheus-node-exporter[1247]: time="2021-09-16T11:15:30+02:00" level=info msg=" - textfile" source="node_exporter.go:157"
Sep 16 11:15:30 srv-00-18 prometheus-node-exporter[1247]: time="2021-09-16T11:15:30+02:00" level=info msg=" - sockstat" source="node_exporter.go:157"
Sep 16 11:15:30 srv-00-18 prometheus-node-exporter[1247]: time="2021-09-16T11:15:30+02:00" level=info msg=" - uname" source="node_exporter.go:157"
Sep 16 11:15:30 srv-00-18 prometheus-node-exporter[1247]: time="2021-09-16T11:15:30+02:00" level=info msg=" - vmstat" source="node_exporter.go:157"
Sep 16 11:15:30 srv-00-18 prometheus-node-exporter[1247]: time="2021-09-16T11:15:30+02:00" level=info msg=" - filesystem" source="node_exporter.go:157"
Sep 16 11:15:30 srv-00-18 prometheus-node-exporter[1247]: time="2021-09-16T11:15:30+02:00" level=info msg=" - hwmon" source="node_exporter.go:157"
Sep 16 11:15:30 srv-00-18 prometheus-node-exporter[1247]: time="2021-09-16T11:15:30+02:00" level=info msg=" - mdadm" source="node_exporter.go:157"
Sep 16 11:15:30 srv-00-18 prometheus-node-exporter[1247]: time="2021-09-16T11:15:30+02:00" level=info msg=" - meminfo" source="node_exporter.go:157"
Sep 16 11:15:30 srv-00-18 prometheus-node-exporter[1247]: time="2021-09-16T11:15:30+02:00" level=info msg=" - netdev" source="node_exporter.go:157"
Sep 16 11:15:30 srv-00-18 prometheus-node-exporter[1247]: time="2021-09-16T11:15:30+02:00" level=info msg="Listening on 127.0.0.1:9100" source="node_exporter.go:176"

Why seems source=“node_exporter.go:176” overwrite that web.listen-address setting? How can I change that behaviour?

#2

I also tried to install the general node_exporter binary from the prometheus github repository.
I started the unit with listen address “static_ip4_address:9100” and netstat shows that node exporter is listening on the address and port. But when I browse to the address there is a site not found error.
What am I missing? For the non UCS systems the installation was straight forward and clear. But adapting this to UCS does not seem to be the correct way.

#3

Despite being a bit late to this threat, but me also stumbled across this behaviour when trying to integrate an UCS in an existing standard prometheus/grafana stack.

As can be seen in node exporters commandline, there is a second “web.listen-address 127.0.0.1:9100” argument which overrides the one coming from the (edited) ARGS entry in /etc/default/prometheus-node-exporter.
This is caused by an override, the UCS team placed in “/etc/systemd/system/prometheus-node-exporter.service.d”:

root@server:/etc/systemd/system/prometheus-node-exporter.service.d# cat 10-univention-path.conf 
[Service]
ExecStart=
ExecStart=/usr/bin/prometheus-node-exporter $ARGS -web.listen-address 127.0.0.1:9100 -web.telemetry-path=/metrics-node/metrics/

Placing a second file “99-custom.conf” in the same path with this content:

[Service]
ExecStart=
ExecStart=/usr/bin/prometheus-node-exporter $ARGS

will solve the problem and:
1.) let node_exporter listen to 0.0.0.0 without the need of editing /etc/default/prometheus-node-exporter
2.) let node_exporter listen to standard /metrics instead of the manipulated /metrics-node/metrics/
3.) should survive UCS updates, as the files coming from UCS are not changed.

#4

Despite being late the problem is not solved yet. So thanks for your reply. Problem is, that there is no such directory “prometheus-node-exporter.service.d” nor the file “10-univention-path.conf” which would overwrite the listen address. I have manually installed the node_exporter binary from the github repository.

edit: Update: The problem changed (I don’t know how to change the topic title). node_exporter service is correctly listening on private address port 9100. But when I open a browser and going to http://internal_ip:9100 it is returning just nothing.

netstat shows

tcp        0      0 172.21.1.2:9100         0.0.0.0:*               LISTEN      636/node_exporter
#5

Tested your suggestion with a server which has the Dashboard App installed. Now it is listening on local address but still same problem as with installed it by hand. When I want to browse to the /metrics site browser just returns nothing. So for my non UCS Servers you can browse to http://address:9100/metrics for the raw output.

Mastodon