Problem: UnicodeDecodeError and AssertionError when pruning UCS Notifier transaction logs

Knowledge Base Community
, , ,
1

Problem

Administrators may encounter failures when working with the Univention Directory Notifier transaction log, particularly when running:

/usr/share/univention-directory-notifier/univention-translog -v prune <ID>

Instead of pruning older entries, the command terminates with:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x88 in position XXXX: invalid start byte

Inspection of the transaction file typically reveals lines that contain invalid UTF-8 sequences and do not match the expected format:

<transaction_id> <LDAP-DN> <operation>

Example of corrupted entries:

1951777 4865781 <88>ö¤<9c><87>^? e
1951778 4865782 <88>ö¤<9c><87>^? e
1951779 4865783 <98>ö¤<9c><87>^? e
...

After removing these corrupted entries, a second problem may appear:

AssertionError: <large number>

This occurs even though there are no longer any UTF-8 issues, and it prevents further translog operations such as pruning.

Root Cause & Solution (Part 1): Corrupted data inside the translog (transaction) file

Root Cause

The Notifier writes every LDAP change as a UTF-8 encoded line to:

/var/lib/univention-ldap/notify/transaction

Each entry must follow the clean text-based format:

<number> <dn> <operation_flag>

If the file is corrupted — due to incomplete writes, system crashes, filesystem issues, replication interruptions, or hardware problems — some lines may contain invalid byte sequences such as 0x88, 0x9c, etc.

Any malformed line causes the parser to fail with a UnicodeDecodeError, blocking the notifier and preventing log pruning. The file may continue to grow uncontrolled (several gigabytes) if the issue is not resolved.

Solution

1. Identify corrupted lines

Use the following script to locate the first malformed translog entry:

find_bad_line.py

# find_bad_line.py
import sys

fn = sys.argv[1]
with open(fn, 'rb') as f:
    offset = 0
    line_no = 1
    for line in f:
        try:
            line.decode("utf-8")
        except UnicodeDecodeError as e:
            print("Error in line:", line_no)
            print("Offset:", offset)
            print("Exception:", e)
            print("Raw line (hex):", line.hex())
            break
        offset += len(line)
        line_no += 1

Run:

python3 find_bad_line.py /var/lib/univention-ldap/notify/transaction

2. Remove the corrupted line range

Use this script to delete a specific set of lines:

delete_range.py

# delete_range.py
import sys
import shutil

src = sys.argv[1]
start = int(sys.argv[2])
end = int(sys.argv[3])

dst = src + ".clean"
bak = src + ".bak"

# Backup
shutil.copy2(src, bak)

with open(src, "r", encoding="utf-8", errors="replace") as f, \
     open(dst, "w", encoding="utf-8") as out:

    line_no = 1
    for line in f:
        if start <= line_no <= end:
            # skip corrupted line
            pass
        else:
            out.write(line)
        line_no += 1

print("Removed lines:", start, "-", end)
print("Output:", dst)
print("Backup:", bak)

Run:

python3 delete_range.py /var/lib/univention-ldap/notify/transaction <start> <end>

Replace the original file:

mv /var/lib/univention-ldap/notify/transaction.clean \
   /var/lib/univention-ldap/notify/transaction

At this point, the corrupted lines are removed — but the index file still refers to positions that no longer exist, which causes the next problem.

Root Cause & Solution (Part 2): Index mismatch after repairing the file

Root Cause

The Notifier stores byte offsets of each transaction entry in:

/var/lib/univention-ldap/notify/transaction.index

After deleting or modifying entries in the transaction file, all byte positions in the index become invalid.

When univention-translog attempts to access an entry using outdated offsets, the program triggers:

AssertionError: <byte_offset>

This is expected behavior: the index is out of sync and must be rebuilt.

Solution

1. Stop the notifier

systemctl stop univention-directory-notifier

2. Remove the outdated index

mv /var/lib/univention-ldap/notify/transaction.index /var/lib/univention-ldap/notify/transaction.index.bak

3. Restart the notifier

systemctl start univention-directory-notifier

The service will automatically regenerate a fresh, correct index.

4. Validate

/usr/share/univention-directory-notifier/univention-translog check
/usr/lib/nagios/plugins/check_univention_replication

If both commands return clean output, the environment is fully repaired.

3

This topic was automatically closed after 24 hours. New replies are no longer allowed.