Problem:
Your replication stuck and you already fixed the transaction file, or made other attempts to get the replication running again, but the notifier seems to do something, but is not answering specific inquiries
/usr/lib/nagios/plugins/check_univention_replication
nid=Error: [Errno 111] Connection refused lid=57972411
Or
06.03.23 11:56:53.852 LISTENER ( WARN ) : Notifier/LDAP server is master.schein.sun:7389
06.03.23 11:56:53.852 LDAP ( PROCESS ) : connecting to ldap://master.schein.sun:7389
06.03.23 11:56:53.857 LISTENER ( ERROR ) : failed to connect to any notifier
06.03.23 11:56:53.858 LISTENER ( WARN ) : can not connect any server, retrying in 30 seconds
Investigation:
Make sure notifier is running:
ps aufx |grep noti
Check for modifications in /var/lib/univention-ldap/notify/transaction. If there are no modifications, no transactions are processed, but if you check the notifier via strace
strace -p "$(pidof -f univention-directory-notifier)"
or
lsof -p "$(pidof -f univention-directory-notifier)"
you see, that the notifier is active it might be indexing.
With the univention-translog skript you can see that the index is not okay, and the warning shows, how many entries are left to proceed with.
root@ucs01: /usr/share/univention-directory-notifier/univention-translog stat
2023-03-03 10:00:42,501:WARNING:Index=57972149 < translog=57972454 entries
---
root@ucs01: /usr/share/univention-directory-notifier/univention-translog stat
2023-03-03 10:02:28,724:WARNING:Index=57972150 < translog=57972454 entries
-----
root@ucs01: /usr/share/univention-directory-notifier/univention-translog stat
2023-03-03 10:06:41,562:WARNING:Index=57972151 < translog=57972454 entries
----------
root@ucs01: /usr/share/univention-directory-notifier/univention-translog stat
2023-03-03 10:09:50,512:WARNING:Index=57972152 < translog=57972454 entries
Solution:
Option 1
In the Bugreport 54797 we have a script, which accelerate the index for UCS 4.4-9 Version.
Option 2
For UCS5.0-3 Version, we updated the univention-translog script from above to do the reindex
/usr/share/univention-directory-notifier/univention-translog -v reindex
Option 3
The root cause of the slow indexing from the notifier is the permanent re-reading of the /var/lib/univention-ldap/notify/transaction file
So an other workaround is, to truncate the translogfile as this reduces the number of lines to parse each time. BUT for that you must know the minimum univention-directory-listener transaction-ids of all listeners in domain!
All lines after this value must stay in the transaction file!
Consult Downsizing transaction file to truncate it. After that the indexing will be much faster.