Monitoring cron script failure

lebernd · June 30, 2022, 7:13pm

Hello together,

after upgrading to 5.0-2 I get a lot of emails - generated every 5 minutes - like:

Cron <root@ucs-01> run-parts /usr/share/univention-monitoring-client/scripts/:

Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_dns", line 46, in <module>
   DNS.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_dns.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_dns exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_i2o_raid", line 96, in <module>
   I2O_RAIDCheck.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_i2o_raid.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_i2o_raid exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_joinstatus", line 170, in <module>
   JoinStatusCheck.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_joinstatus.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_joinstatus exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_kpasswdd", line 44, in <module>
   Kpasswdd.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_kpasswdd.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_kpasswdd exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_ldap", line 54, in <module>
   LDAP.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_ldap.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_ldap exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_mdb_maxsize", line 75, in <module>
   MDBMaxsize.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_mdb_maxsize.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_mdb_maxsize exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_nfsstatus", line 93, in <module>
   FSMountCheck.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_nfsstatus.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_nfsstatus exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_nmbd", line 74, in <module>
   NMBD.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_nmbd.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_nmbd exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_nscd", line 64, in <module>
   NSCD.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_nscd.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_nscd exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_ntp", line 46, in <module>
   NTP.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_ntp.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_ntp exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_package_status", line 47, in <module>
   PackageStatus.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_package_status.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_package_status exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_ping", line 45, in <module>
   Ping.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_ping.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_ping exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_printerqueue", line 62, in <module>
   Printerqueue.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_printerqueue.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_printerqueue exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_replication", line 68, in <module>
   ReplicationCheck.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_replication.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_replication exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_s4_connector", line 75, in <module>
   S4Connector.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_s4_connector.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_s4_connector exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_samba_drs_failures", line 86, in <module>
   CheckSambaDrsRepl.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_samba_drs_failures.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_samba_drs_failures exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_smbd", line 61, in <module>
   SMBD.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_smbd.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_smbd exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_smtp", line 49, in <module>
   SMTP.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_smtp.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_smtp exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_ssl_certificate", line 62, in <module>
   SSLCertificateCheck.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_ssl_certificate.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_ssl_certificate exited with return code 1
Traceback (most recent call last):
 File "/usr/share/univention-monitoring-client/scripts//check_univention_winbind", line 51, in <module>
   Winbind.main()
 File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 73, in main
   with open(os.path.join(NODE_EXPORTER_DIR, '%s.prom' % (plugin,)), 'w') as self._fd:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/prometheus/node-exporter/check_univention_winbind.prom'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_winbind exited with return code 1

First impuls was to install the dashboard-app and see if that changes the messages.
But I’m not sure about the licence. The appcenter tells:
“Kostenlose kommerzielle Nutzung” and if I click the link below it tells I need a subscription…
is it “kostenlos bei kommerzieller Nutzung”?

Any other approach would also be appreciated.

Best,
Bernd

tpfann · July 1, 2022, 7:26am

Hi,

just updated to 5.0-2 errata342 and I am also getting this every 5 minutes now (via eMail).
Regarding the licence for the UCS Dashboard I am confused as well - can someone please clarify?

BR,
Thomas

siv · July 1, 2022, 5:05pm

The error messages from the cronjob seem to be due to the new univention-monitoring-client not creating the necessary /var/lib/prometheus/node-exporter directory.
At least running mkdir -p /var/lib/prometheus/node-exporter fixed the issue on our installation

Would be great if it could be added to debian/dirs for the next release.

tpfann · July 1, 2022, 5:40pm

After creating the missing directory ( running `mkdir -p /var/lib/prometheus/node-exporter), the error messages are still comming in every 5 minutes.

Anyway, thx for having a look and your suggestion.

krda79 · July 3, 2022, 2:35pm

Have the same here.

After installation of UCS Dashboard app the number of errors in the mail is reduced, but it keeps sending every 5 minutes:

Traceback (most recent call last):
  File "/usr/share/univention-monitoring-client/scripts//check_univention_samba_drs_failures", line 86, in <module>
    CheckSambaDrsRepl.main()
  File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 74, in main
    self.write_metrics()
  File "/usr/share/univention-monitoring-client/scripts//check_univention_samba_drs_failures", line 65, in write_metrics
    (info_type, info) = drsuapi.DsReplicaGetInfo(self.drsuapi_handle, 1, req1)
TypeError: cannot unpack non-iterable drsuapi.DsReplicaGetInfo object
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_samba_drs_failures exited with return code 1

Anybody else getting this error. Shall we create a bug report?

olaf.peter · July 4, 2022, 11:25am

Same problem after updating to 5.0-2 (only on members, not on the primary node)

Disabling checks with
ucr set monitoring/client/cron='#*/5 * * * *'
is actually my way.

digitec-san · July 4, 2022, 9:21pm

After creating /var/lib/prometheus/node-exporter as suggested by tpfann the message disapeers.

tpfann · July 5, 2022, 7:27am

Confirmed, its the same here.

After creating the directory “/var/lib/prometheus/node-exporter” the original error message is gone but now I get this error message via email every 5 minutes

Traceback (most recent call last):
  File "/usr/share/univention-monitoring-client/scripts//check_univention_samba_drs_failures", line 86, in <module>
    CheckSambaDrsRepl.main()
  File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 74, in main
    self.write_metrics()
  File "/usr/share/univention-monitoring-client/scripts//check_univention_samba_drs_failures", line 65, in write_metrics
    (info_type, info) = drsuapi.DsReplicaGetInfo(self.drsuapi_handle, 1, req1)
TypeError: cannot unpack non-iterable drsuapi.DsReplicaGetInfo object
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_samba_drs_failures exited with return code 1

Best · July 11, 2022, 9:18am

A fix for those checks has been released with Security and bugfix errata for Univention Corporate Server.

Andreas_T · July 12, 2022, 9:53pm

After updating to 5.0-2 we are getting these errors every 5 minutes. The system uses some NFS mounts.

Traceback (most recent call last):
   File "/usr/share/univention-monitoring-client/scripts//check_univention_nfsstatus", line 93, in <module>
      FSMountCheck.main()
   File "/usr/lib/python3/dist-packages/univention/monitoring/__init__.py", line 74, in main
      self.write_metrics()
   File "/usr/share/univention-monitoring-client/scripts//check_univention_nfsstatus", line 86, in write_metrics
      msg = '%s OK, %s %s - %s' % (mounted, umounted, self.errorstate, msg)
AttributeError: 'FSMountCheck' object has no attribute 'errorstate'
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_nfsstatus exited with return code 1

There exists a variable state just before self.errorstate in the script, that should probably go there.

riess82 · July 14, 2022, 12:00pm

I have a variation looking like this (UCS 5.0-2 errata352):

Traceback (most recent call last):
File “/usr/share/univention-monitoring-client/scripts//check_univention_ntp”, line 46, in
NTP.main()
File “/usr/lib/python3/dist-packages/univention/monitoring/init.py”, line 74, in main
self.write_metrics()
File “/usr/share/univention-monitoring-client/scripts//check_univention_ntp”, line 40, in write_metrics
ntp_offset = float(output.split(’|’, 1)[-1].split(’;’)[0].split(’=’)[-1].rstrip(‘s’))
ValueError: could not convert string to float:
run-parts: /usr/share/univention-monitoring-client/scripts//check_univention_ntp exited with return code 1

riess82 · July 14, 2022, 1:30pm

Another reboot later, I receice no more error emails. Too scared now to check if the error does not email or really is gone xD
UPDATE: 1 day later I receive the error email again!

olaf.peter · July 18, 2022, 7:07am

Problem solved with update und reboot - thanks for the patch.

Best · July 20, 2022, 9:29pm

I created a bug report for this error: Bug 55017 – check_univention_ntp: ValueError: could not convert string to float: