Problem:
ID-connector quue is not reduced.
If the diskcache in the id-connector is inconsistent, e.g. due to power outage, items from the cache cannot be parsed. In the queues.log:
2023-08-11 15:13:20 ERROR [InQueue.distribute_loop:356] During preprocessing of file (1/51132) ‘2023-08-10-17-43-59-023977.json’: Ran out of input
Traceback (most recent call last):
File "/ucsschool-id-connector/src/ucsschool_id_connector/queues.py", line 327, in distribute_loop
new_path = await self.preprocess_file(path)
File "/ucsschool-id-connector/src/ucsschool_id_connector/queues.py", line 288, in preprocess_file
changed |= any([await coro for coro in result_coros])
File "/ucsschool-id-connector/src/ucsschool_id_connector/queues.py", line 288, in <listcomp>
changed |= any([await coro for coro in result_coros])
File "/ucsschool-id-connector/src/plugins/plugins/listener_objects.py", line 172, in preprocess_add_mod_object
obj.old_data = self.get_old_data(obj)
File "/ucsschool-id-connector/src/plugins/plugins/listener_objects.py", line 139, in get_old_data
return self.old_data_db.get(obj.id)
File "/ucsschool-id-connector/src/ucsschool_id_connector/db.py", line 82, in get
value = self._cache.get(key, default, *args, **kwargs)
File "/usr/lib/python3.8/site-packages/diskcache/core.py", line 1173, in get
value = self._disk.fetch(mode, filename, db_value, read)
File "/usr/lib/python3.8/site-packages/diskcache/core.py", line 282, in fetch
return pickle.load(reader)
EOFError: Ran out of input
Investigation:
Root cause was an filled up filesystem, which produced 0byte files in the id-connector cache database. So removeing these 0byte file in the filesystem is not enough. They also have to be removed from the cache.
Solution:
Find the 0byte files:
find /var/lib/univention-appcenter/apps/ucsschool-id-connector/data/old_data_db/ -type f -exec ls -lah {} \+ |sort -hk6r
-rw-r--r-- 1 root root 0 Feb 20 18:15 /var/lib/univention-appcenter/apps/ucsschool-id-connector/data/old_data_db/f5/8c/c4382f344b6582df0c64ff1ba9fe.val
-rw-r--r-- 1 root root 0 Feb 20 18:15 /var/lib/univention-appcenter/apps/ucsschool-id-connector/data/old_data_db/9e/ce/8c67c92daefa6d5149c83bf00c67.val
connect with the database:
cd /var/lib/univention-appcenter/apps/ucsschool-id-connector/data/old_data_db
sqlite3 cache.db
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> .headers on
sqlite> .mode column
sqlite> select * from cache limit 1;
rowid key raw store_time expire_time access_time access_count tag size mode filename value
sqlite> select * from cache where filename='f5/8c/c4382f344b6582df0c64ff1ba9fe.val';
rowid key raw store_time expire_time access_time access_count tag size mode filename value
---------- ------------------------------------ ---------- --------------- ----------- --------------- ------------ ---------- ---------- ---------- -------------------------------------- ----------
84893 9c498f68-d46a-1038-8d1b-2fcd351ecac5 1 1740071720.6289 1740071720.6289 0 177471 4 f5/8c/c4382f344b6582df0c64ff1ba9fe.val
sqlite> delete from cache where filename='f5/8c/c4382f344b6582df0c64ff1ba9fe.val';
sqlite> select * from cache where filename='f5/8c/c4382f344b6582df0c64ff1ba9fe.val';
sqlite> select * from cache where filename='9e/ce/8c67c92daefa6d5149c83bf00c67.val';
rowid key raw store_time expire_time access_time access_count tag size mode filename value
---------- ------------------------------------ ---------- ---------------- ----------- ---------------- ------------ ---------- ---------- ---------- -------------------------------------- ----------
2201 9bf660b8-d46a-1038-8ce8-2fcd351ecac5 1 1740071720.60013 1740071720.60013 0 167117 4 9e/ce/8c67c92daefa6d5149c83bf00c67.val
sqlite> delete from cache where filename='9e/ce/8c67c92daefa6d5149c83bf00c67.val';
sqlite> select * from cache where filename='9e/ce/8c67c92daefa6d5149c83bf00c67.val';
sqlite>