Howto Check for Listener/ Notifier Service Status
This article will describe how to check if an issue with listener or notifier service on UCS hosts exist.
Environment:
For checking the current Listener/ Notifier Service Status, we are using a nagios-plugin, deployed by us automatically. You can find it (and many others) in /usr/lib/nagios/plugins/
Hint: Even after upgrading to UCS 5 and switching from Nagios to our dashboard, these plugins are available and can still be used.
Step 1: Get the current state of replication
Simple execute the plugin as root:
root@example:~# /usr/lib/nagios/plugins/check_univention_replication
Posible Results:
-
CRITICAL: no change of listener transaction id for last 0 checks (nid=3030 lid=3018)
Repeat the check a couple of seconds later. If it still shows “CRITICAL” proceed to step 2 -
CRITICAL: failed.ldif exists
Follow troubleshooting guideline in this article. -
OK: replication complete (nid=3122 lid=3122)
The listener state on this host is fine. Repeat the steps for all hosts in your domain.
Step 2: Restart listener / notifier if needed
If your replication seems to be stuck, restart the services.
For all server roles:
root@example:~# systemctl restart univention-directory-listener
additionally for backup and master role:
root@master:~# systemctl restart univention-directory-notifier
Hint: univention-directory-notifier only exists on UCS systems with the role Master and Backup
Verify services are running fine:
For all server roles:
root@example:~# systemctl status univention-directory-listener
additionally for backup and master role:
root@example:~# systemctl status univention-directory-notifier
If not, proceed with step 3.
Step 3: Increase debug level and restart for both services.
You might see more informative messages in the log files after increasing the loglevel.
For all server roles:
root@example:~# ucr set listener/debug/level=4 # default: 2
root@example:~# systemctl restart univention-directory-listener
additionally for backup and master role:
root@example:~# ucr set notifier/debug/level=4 # default: 2
root@example:~# systemctl restart univention-directory-notifier
See checks in the next steps.
Step 4.1: notifier
Note: You should check notifier on all master and backup servers
Check logfile for details about the error.
The Notifier start is shown in the logfile with the line containing DEBUG_INIT.
In the following you see expected debug messages when everything is fine.
If it shows ERROR you should do further troubleshooting.
root@example:~# tail -f /var/log/univention/notifier.log
11.04.19 09:02:06.194 DEBUG_INIT
11.04.19 09:02:06.194 TRANSFILE ( ALL ) : LOCK from notify_transaction_get_last_notify_id
11.04.19 09:02:06.194 TRANSFILE ( ALL ) : LOCK /var/lib/univention-ldap/notify/transaction
11.04.19 09:02:06.194 TRANSFILE ( ALL ) : FCLOSE start
11.04.19 09:02:06.194 TRANSFILE ( ALL ) : FCLOSE end
11.04.19 09:02:06.194 TRANSFILE ( ALL ) : UNLOCK /var/lib/univention-ldap/notify/transaction
11.04.19 09:02:06.195 TRANSFILE ( INFO ) : Last transaction id = 3122
Step 4.2: listener
Note: You should check listener on all servers
Check logfile for details about the error.
The Listener start is shown in the logfile with the line containing DEBUG_INIT.
In the following you see expected debug messages when everything is fine.
If it shows ERROR you should do further troubleshooting.
root@example:~# tail -f /var/log/univention/listener.log
11.04.19 09:01:45.876 DEBUG_INIT
11.04.19 09:01:45.886 LISTENER ( WARN ) : Notifier/LDAP server is master.multi.ucs:7389
11.04.19 09:01:45.886 LDAP ( PROCESS ) : connecting to ldap://master.multi.ucs:7389
11.04.19 09:01:45.893 LDAP ( INFO ) : simple_bind as cn=admin,dc=multi,dc=ucs
11.04.19 09:01:45.893 LISTENER ( INFO ) : connecting to notifier master.multi.ucs:6669
11.04.19 09:01:45.894 LISTENER ( INFO ) : established connection to 10.250.200.100 port 6669
11.04.19 09:01:45.894 LISTENER ( ALL ) : >>>Version: 3
Step 5.1: listener issue
In case you see the following entries in the listener.log, refer to this article (part “Solution”) to reset the password.
27.08.19 09:38:58.663 LISTENER ( INFO ) : LDAP-Server is dc09.multi.ucs:0
27.08.19 09:38:58.663 LISTENER ( INFO ) : no server given, choosing one by myself (dc09.multi.ucs)
27.08.19 09:38:58.663 LDAP ( INFO ) : connecting to ldap://master.example.ucs:7389
27.08.19 09:38:58.829 LDAP ( INFO ) : simple_bind as cn=ucs-slave,cn=dc,cn=computers,dc=example,dc=ucs
27.08.19 09:38:58.860 LDAP ( ERROR ) : ldap_simple_bind: Invalid credentials
27.08.19 09:38:58.861 LISTENER ( WARN ) : can not connect to LDAP server master.example.ucs:7389
27.08.19 09:38:58.861 LISTENER ( WARN ) : can not connect any server, retrying in 30 seconds
Step 5.2: notifier issue
You might see error messages in notifier.log (see below). In this case increase the database size on all master/ backup/ slave server according to this article.
14.10.19 14:30:00.631 TRANSFILE ( ALL ) : SASL: id=16385 chal=Authorization Name prom=Please enter your authorization name def=
14.10.19 14:30:00.631 TRANSFILE ( ALL ) : LDIF dn: reqSession=11597335,cn=translog
14.10.19 14:30:00.631 TRANSFILE ( ALL ) : LDIF objectClass: auditObject
14.10.19 14:30:00.631 TRANSFILE ( ALL ) : LDIF reqStart: 20191014183000Z
14.10.19 14:30:00.631 TRANSFILE ( ALL ) : LDIF reqSession: 11597335
14.10.19 14:30:00.631 TRANSFILE ( ALL ) : LDIF reqDN: uid=user42,cn=users,dc=example,dc=ucs
14.10.19 14:30:00.631 TRANSFILE ( ALL ) : LDIF reqType: m
14.10.19 14:30:00.632 TRANSFILE ( ERROR ) : 11597335 ldap_add(): Other (e.g., implementation specific) error
Still failing
If either Notifier or Listener still do not start properly you might need to open a support ticket or use community support.
To check if the notifier/ listener files are corrupted, use this article.