Problem
In some OpenDesk/Nubus installations, the UDM REST API enters a broken state and starts returning HTTP 500 for authenticated requests.
Typical symptoms include:
ums-stack-datajobs fail during initialization.- Later
ums-stack-datajobs fail in thewait-for-udm-rest-apiinit container. - Portal functions depending on the UDM REST API stop working.
- User creation via the UDM REST API is no longer possible.
The issue is intermittent. It has been observed shortly after installation as well as during normal operation.
Investigation
The first indication is usually failed ums-stack-data jobs.
List the job pods:
kubectl -n <namespace> get pods | grep ums-stack-data
Inspect the failed job:
kubectl -n <namespace> logs <ums-stack-data-pod> --all-containers --timestamps
The init container initially reports that the API is reachable:
INFO:app:Checking if UDM Rest API can be reached...
INFO:app:Success, UDM Rest API is available
The main container then starts importing Stack Data and fails while creating the first UDM object:
INFO:univention.data_loader.actions:
Ensuring udm object groups/group,
cn=groups,dc=swp-ldap,dc=internal,
Domain Service Users
The request fails with HTTP 500:
univention.admin.rest.client.ServerError:
POST http://ums-udm-rest-api:9979/univention/udm/groups/group/: 500
cannot unpack non-iterable NoneType object
Next, inspect the UDM REST API pod:
kubectl -n <namespace> get pods | grep udm-rest-api
kubectl -n <namespace> logs <udm-rest-api-pod> \
--all-containers \
--timestamps
The server-side traceback shows the exception in authenticate():
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/tornado/web.py", line 1712, in _execute
result = self.prepare()
File "/usr/lib/python3/dist-packages/univention/admin/rest/module.py", line 163, in prepare
if authorization and self.authenticate(authorization):
File "/usr/lib/python3/dist-packages/univention/admin/rest/module.py", line 172, in authenticate
auth_type, username, userdn, password = shared_memory.authenticated[authorization]
TypeError: cannot unpack non-iterable NoneType object
After the first occurrence, subsequent authenticated requests fail with the same code path, but may report:
TypeError: cannot unpack non-iterable bool object
The error is no longer limited to object creation. Even simple API requests fail with HTTP 500, for example:
GET /univention/udm/
GET /univention/udm/settings/data/...
Example:
application:
Uncaught exception GET /univention/udm/
Traceback (most recent call last):
...
File "/usr/lib/python3/dist-packages/univention/admin/rest/module.py", line 172, in authenticate
auth_type, username, userdn, password = shared_memory.authenticated[authorization]
TypeError: cannot unpack non-iterable bool object
access:
500 GET /univention/udm/
This demonstrates that the failure occurs during request authentication before the requested UDM operation is executed.
Solution
A temporary workaround is to restart the UDM REST API and rerun the failed Stack Data job.
Restart the UDM REST API deployment:
kubectl -n <namespace> rollout restart deployment ums-udm-rest-api
After the API is available again, rerun the failed Stack Data job.
The issue has been observed to reoccur later.
Related information
This issue affects the UDM REST API authentication path and is characterized by server-side exceptions originating from:
/usr/lib/python3/dist-packages/univention/admin/rest/module.py
during access to:
shared_memory.authenticated[authorization]
The observed exceptions are:
TypeError: cannot unpack non-iterable NoneType object
and
TypeError: cannot unpack non-iterable bool object
See also Bug 59702