Howto update Listener CacheMasterEntry

[pmhahn]

Each change to LDAP gets processed by Univention Directory Notifier (UDN) and Univention Directory Listener (UDL). Each such transaction is assigned a sequence number, which is used to keep track of transactions already processed by UDL. This is known as the Notifier ID, as UDN broadcasts the sequence number to all listening UDLs. The sequence number of the last processed transaction can be queried from UDN on the DC Master (or Backup):

/usr/share/univention-directory-listener/get_notifier_id.py

The DC Master also tracks changes to the LDAP Schema information and assigns a second sequence number for this, which is known as the Schema ID. This sequence number is also available from UDN:

/usr/share/univention-directory-listener/get_notifier_id.py -s

Each UDL tracks those two IDs.
Since UCS-4.2 UDL uses a Lightning Memory Database (LMDB) for its cache. That cache stores a copy of all LDAP entries to allow modules to work with before-after-data. The cache also contains the so-called CacheMasterEntry, which stores the said Notifier ID and Schema ID last seen by UDL. That entry is stored in the id2entry sub-database using key 0.
In UCS-4.1 and earlier they were kept in the files /var/lib/univention-directory-listener/notifier_id and /var/lib/univention-ldap/schema/id/id, but were moved into the database for consistency.

In some cases the Notifier ID and Schema ID entries get out-of-sync. To debug replication issues it helps to compare the values from the UDL cache which the values from the system, where UDL replicates from.
In some very rare cases the values must be updated to fix replication issues, for example to skip broken transactions.

python-lmdb provides a Python interface for LMDB, which can be used to get and update the values:

ucr set repository/online/unmaintained=yes
univention-install python-lmdb
python

#!/usr/bin/python2.7
import sys
import lmdb
import struct

IS_64BITS = sys.maxsize > 2**32
CME = struct.Struct('<QQ' if IS_64BITS else '<LL')
KEY = '\x00' * (8 if IS_64BITS else 4)

env = lmdb.Environment('/var/lib/univention-directory-listener/cache', subdir=True, max_dbs=3)
db = env.open_db('id2entry')
with env.begin(db=db, write=True) as txn:
    with txn.cursor() as cur:
        value = cur.get(KEY)
        nid, sid = CME.unpack(value)
        print('nid={}\nsid={}'.format(nid, sid))
        nid = raw_input('New nid?') or nid
        sid = raw_input('New sid?') or sid
        value = CME.pack(nid, sid)
        txn.put(KEY, value)

Alternative

If the Python module is unavailable the low-level tools from the Debian package lmdb-utils can be used as well:

Get current values

The following command can be used to read the current values:

$ mdb_dump -s id2entry /var/lib/univention-directory-listener/cache |
    grep -m1 -A1 -F 0000000000000000
 0000000000000000
 390e0000000000000d00000000000000

The first output line is again the key number as a 64-bit value, the second output line is a hex-dump of 2 64-bit values in little-endian notation of the Notifier ID and Schema ID.

Change values

mdb_load can be used to update that entry:

( echo '\00\00\00\00\00\00\00\00'; echo -n '\39\0e\00\00\00\00\00\00'; echo '\0d\00\00\00\00\00\00\00') |
    mdb_load -T -s id2entry /var/lib/univention-directory-listener/cache

The first line is again 0 for the CacheMasterEntry, the second line both values again hex-encoded, but also prefixed by a backslash \.

To convert a number to the 64-bit little-endian format, you can use the following Python command:

python -c 'print("".join(r"\%02x" % ord(_) for _ in __import__("struct").pack("<q", int(__import__("sys").argv[1]))))' $VALUE

Known issues

• mdb_load in UCS-4.2 has a bug and does not correctly parse the backslash-escaped string. You can copy the binary from a UCS-4.3 system or newer.Processing: udl-mce…
• UCS-5.0.[0…4] has not shipped the Python modules python-lmdb nor the python3-lmdb; it is only available again since UCS 5.0-5 erratum 836. Therefore the Python version does not work there; Here’s a small Go-lang binary for amd64, which you can download; uncompress it with gunzip udl-mce.gz. Afterward call it as ./udl-mce to show the current values; Call it again with the option -n XXX to set the notifier ID and/or -s YYY to set the LDAP Schema ID.
  udl-mce.gz (1.4 MB)