Problem: Provisioning-Service - Prefill fails in large environments - `prefill_queue_status":"failed"`

Provisioning - Prefill fails in large environments - prefill_queue_status":"failed"

:bulb: This issue is fixed in Provisioning-Service Version 2.2.0.

Problem

In a large school environment with 100,000 or more users, the Prefill service may fail because of a timeout while querying the UDM REST API.

The issue occurs because the service attempts to load all Distinguished Names (DNs) of a UDM module in a single request.

Output

root@ucs5primary:~/univention-support# curl -k -X  GET -u "example-consumer:SoM3-reA1ly_sEc4eT_Pa33morD" "https://localhost/univention/provisioning/v1/subscriptions/example-consumer"
{"name":"example-consumer","realms_topics":[{"realm":"udm","topic":"users/user"}],"request_prefill":true,"prefill_queue_status":"failed"}

Check the Prefill service logs:

docker logs nubus-provisioning-prefill
2026-06-29 14:30:57,713 ERROR [None] [prefill_service.handle_requests_to_prefill:83] Unknown error occurred while processing the prefill request.
  + Exception Group Traceback (most recent call last):
  |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/prefill_service.py", line 68, in handle_requests_to_prefill
  |     await self.ack_manager.process_message_with_ack_wait_extension(
  |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/backends/message_queue.py", line 169, in process_message_with_ack_wait_extension
  |     async with asyncio.TaskGroup() as task_group:
  |   File "/usr/lib/python3.11/asyncio/taskgroups.py", line 133, in __aexit__
  |     raise me from None
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 539, in start
    |     message, payload = await protocol.read()  # type: ignore[union-attr]
    |                        ^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/streams.py", line 703, in read
    |     await self._waiter
    | asyncio.exceptions.CancelledError
    |
    | The above exception was the direct cause of the following exception:
    |
    | Traceback (most recent call last):
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/prefill_service.py", line 119, in handle_message
    |     await self._handle_message(validated_message)
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/prefill_service.py", line 142, in _handle_message
    |     await self.fetch_udm(message.subscription_name, realm_topic.topic)
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/prefill_service.py", line 160, in fetch_udm
    |     await self._fill_udm_topic(this_topic, subscription_name)
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/prefill_service.py", line 181, in _fill_udm_topic
    |     urls = await self.udm.list_objects(object_type)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 193, in async_wrapped
    |     return await copy(fn, *args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 112, in __call__
    |     do = await self.iter(retry_state=retry_state)
    |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 157, in iter
    |     result = await action(retry_state)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/_utils.py", line 111, in inner
    |     return call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 393, in <lambda>
    |     self._add_action_func(lambda rs: rs.outcome.result())
    |                                      ^^^^^^^^^^^^^^^^^^^
    |   File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    |     return self.__get_result()
    |            ^^^^^^^^^^^^^^^^^^^
    |   File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    |     raise self._exception
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 116, in __call__
    |     result = await fn(*args, **kwargs)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/udm_adapter.py", line 89, in list_objects
    |     async with self._session.get(f"{self.base_url}{object_type}/", params=params) as request:
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 1510, in __aenter__
    |     self._resp: _RetType = await self._coro
    |                            ^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 779, in _request
    |     resp = await handler(req)
    |            ^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 757, in _connect_and_send_request
    |     await resp.start(conn)
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 534, in start
    |     with self._timer:
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/helpers.py", line 713, in __exit__
    |     raise asyncio.TimeoutError from exc_val
    | TimeoutError
    +------------------------------------
2026-06-29 14:30:57,736 WARNING [None] [_daemonize_unix.__exit__:141] Daemonizer exited without calling daemonize.
  + Exception Group Traceback (most recent call last):
  |   File "/app/prefill/.venv/bin/prefill", line 8, in <module>
  |     sys.exit(run())
  |              ^^^^^
  |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/main.py", line 43, in run
  |     asyncio.run(main(settings))
  |   File "/usr/lib/python3.11/asyncio/runners.py", line 190, in run
  |     return runner.run(main)
  |            ^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run
  |     return self._loop.run_until_complete(task)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
  |     return future.result()
  |            ^^^^^^^^^^^^^^^
  |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/main.py", line 37, in main
  |     await service.handle_requests_to_prefill()
  |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/prefill_service.py", line 68, in handle_requests_to_prefill
  |     await self.ack_manager.process_message_with_ack_wait_extension(
  |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/backends/message_queue.py", line 169, in process_message_with_ack_wait_extension
  |     async with asyncio.TaskGroup() as task_group:
  |   File "/usr/lib/python3.11/asyncio/taskgroups.py", line 133, in __aexit__
  |     raise me from None
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 539, in start
    |     message, payload = await protocol.read()  # type: ignore[union-attr]
    |                        ^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/streams.py", line 703, in read
    |     await self._waiter
    | asyncio.exceptions.CancelledError
    |
    | The above exception was the direct cause of the following exception:
    |
    | Traceback (most recent call last):
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/prefill_service.py", line 119, in handle_message
    |     await self._handle_message(validated_message)
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/prefill_service.py", line 142, in _handle_message
    |     await self.fetch_udm(message.subscription_name, realm_topic.topic)
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/prefill_service.py", line 160, in fetch_udm
    |     await self._fill_udm_topic(this_topic, subscription_name)
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/prefill_service.py", line 181, in _fill_udm_topic
    |     urls = await self.udm.list_objects(object_type)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 193, in async_wrapped
    |     return await copy(fn, *args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 112, in __call__
    |     do = await self.iter(retry_state=retry_state)
    |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 157, in iter
    |     result = await action(retry_state)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/_utils.py", line 111, in inner
    |     return call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 393, in <lambda>
    |     self._add_action_func(lambda rs: rs.outcome.result())
    |                                      ^^^^^^^^^^^^^^^^^^^
    |   File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    |     return self.__get_result()
    |            ^^^^^^^^^^^^^^^^^^^
    |   File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    |     raise self._exception
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 116, in __call__
    |     result = await fn(*args, **kwargs)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/udm_adapter.py", line 89, in list_objects
    |     async with self._session.get(f"{self.base_url}{object_type}/", params=params) as request:
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 1510, in __aenter__
    |     self._resp: _RetType = await self._coro
    |                            ^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 779, in _request
    |     resp = await handler(req)
    |            ^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 757, in _connect_and_send_request
    |     await resp.start(conn)
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 534, in start
    |     with self._timer:
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/helpers.py", line 713, in __exit__
    |     raise asyncio.TimeoutError from exc_val
    | TimeoutError
    +------------------------------------
/entrypoint.sh: No files found in /entrypoint.d/, skipping configuration
2026-06-29 11:50:56,710 ERROR [None] [prefill_service.handle_message:108] The maximum number of retries for prefilling the subscription %s has been reached. The prefill will not be retried again and the subscription will be marked as failed. To trigger the prefill again, delete and recreate the subscription.

Root Cause

The issue occurs with the following component versions:

  • Provisioning-Service Version 2.1.0
  • Provisioning-NATS Version 2.12

During a prefill, the service loads all DNs of a module in a single operation. This can result in an empty response and a 30-second NATS timeout.

Check the current NATS maximum payload:

root@ucs5primary:~/univention-support/substring# curl -s http://127.0.0.1:8222/varz | grep max_payload
  "max_payload": 1048576,

The configured max_payload with 1MB is insufficient for larger environments containing 20,000 or 100,000 user objects because all DNs are retrieved at once during the prefill operation.


Investigation

Reproducing the issue

The issue was reproduced by creating 20,000 users for a school.

/usr/share/ucs-school-import/scripts/ucs-school-testuser-import --httpapi --students 20 --classes 2 <ou>

/usr/share/ucs-school-import/scripts/ucs-school-testuser-import --httpapi --students 20000 --teachers 100 --classes 2 Heisenberg

/usr/share/ucs-school-import/scripts/ucs-school-user-import --school 'Heisenberg' --user_role student --source_uid 'heisenberg-student' --conffile '/usr/share/ucs-school-import/configs/ucs-school-testuser-http-import.json' --infile 'test_users_2026-06-29_14:00:52.csv'

Check the NATS maximum payload:

root@ucs5primary:~/univention-support/substring# curl -s http://127.0.0.1:8222/varz | grep max_payload
  "max_payload": 1048576,

Retrieve the nubus-provisioning-api credentials

root@ucs5primary:~/univention-support/substring# docker exec nubus-provisioning-api env | grep -i admin
ADMIN_NATS_USER=NOT_SET_NOT_REQUIRED
ADMIN_NATS_PASSWORD=NOT_SET_NOT_REQUIRED
ADMIN_USERNAME=admin
ADMIN_PASSWORD=pJoASmIEFKnhdWeH6fb8S0YAJUWPd7i4

Export the administrator credentials:

root@ucs5primary:~# export ADMIN_USERNAME=$(docker exec nubus-provisioning-api printenv ADMIN_USERNAME)
root@ucs5primary:~# export ADMIN_PASSWORD=$(docker exec nubus-provisioning-api printenv ADMIN_PASSWORD)

Dry run

root@ucs5primary:~# curl -k -X GET -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" "https://localhost/univention/provisioning/v1/subscriptions"
[]root@ucs5primary:~#

Explanation

An empty array, [], indicates that no third-party system or consumer has yet been connected to the new Provisioning API.

Subscription configuration

root@ucs5primary:~/univention-support# cat provisioning-api.json
{
  "name": "example-consumer",
  "realms_topics": [
    {
      "realm": "udm",
      "topic": "users/user"
    }
  ],
  "request_prefill": true,
  "password": "SoM3-reA1ly_sEc4eT_Pa33morD"
}

Create the subscription

root@ucs5primary:~/univention-support# curl -k -X POST -u "$ADMIN_USERNAME":"$ADMIN_PASSWORD" "https://localhost/univention/provisioning/v1/subscriptions" -H "Content-Type: application/json" --data @provisioning-api.json

Check the subscription status:

root@ucs5primary:~/univention-support# curl -k -X GET -u "example-consumer:SoM3-reA1ly_sEc4eT_Pa33morD" "https://localhost/univention/provisioning/v1/subscriptions/example-consumer"
{"name":"example-consumer","realms_topics":[{"realm":"udm","topic":"users/user"}],"request_prefill":true,"prefill_queue_status":"done"}

Workaround

Increase the max_payload value in the following file:

nano /var/lib/univention-appcenter/apps/provisioning-service/conf/nats.conf

Example configuration to reproduce the issue (max_payload is manually configured to 1KB):

{
  server_name: nats
  pid_file: "/nats.pid"
  port: 4222
  http_port: 8222
  max_payload: 1000 <-- 1kb
  lame_duck_duration: 30s
  lame_duck_grace_period: 10s
  jetstream {
    max_file_store: 1Gi
    max_memory_store: 256Mi
    store_dir: "/data"
  }
<skip>

Check the prefill with a dry run

The request fails because max_payload is configured to 1 KB. The previous default in this test was 1MB. A payload of 1 KB is too small for large environments containing 20,000 or more users.

root@ucs5primary:~/univention-support# curl -k -s -o /dev/null -w "\n=== DETAILLIERTE ZEITMESSUNG ===\nDNS-Auflösung:                      %{time_namelookup}s\nTCP-Verbindungsaufbau:              %{time_connect}s\nTLS-Handshake:                      %{time_appconnect}s\nServer-Verarbeitung (LDAP + JSON):  %{time_starttransfer}s\nGesamte Download-Dauer:             %{time_total}s\n\nHTTP-Status:                        %{http_code}\nÜbertragene Datenmenge:             %{size_download} Bytes\n" \
-u "provi-11650649$:4a9f048910581b185560eb979ecb48c7e3fddf7b075eead42910c3fe5f4f1485" \
"https://ucs5primary.miro.intranet/univention/udm/users/user/?scope=sub&hidden=true&properties=NonExistantDummyProperty&page=1&limit=0&dir=ASC"

=== DETAILLIERTE ZEITMESSUNG ===
DNS-Auflösung:                      0.001280s
TCP-Verbindungsaufbau:              0.001428s
TLS-Handshake:                      0.010115s
Server-Verarbeitung (LDAP + JSON):  311.028980s
Gesamte Download-Dauer:             311.029091s

HTTP-Status:                        503
Übertragene Datenmenge:             117 Bytes

Delete the subscription

root@ucs5primary:~/univention-support# curl -k -X  DELETE -u "example-consumer:SoM3-reA1ly_sEc4eT_Pa33morD" "https://localhost/univention/provisioning/v1/subscriptions/example-consumer"
null

Check the subscription status:

root@ucs5primary:~/univention-support# curl -k -X GET -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" "https://localhost/univention/provisioning/v1/subscriptions"
[{"name":"example-consumer","realms_topics":[{"realm":"udm","topic":"users/user"}],"request_prefill":true,"prefill_queue_status":"running"}]

After some time, the prefill_queue_status changes to failed:

root@ucs5primary:~/univention-support# curl -k -X GET -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" "https://localhost/univention/provisioning/v1/subscriptions"
[{"name":"example-consumer","realms_topics":[{"realm":"udm","topic":"users/user"}],"request_prefill":true,"prefill_queue_status":"failed"}]

Check the Prefill service logs:

docker logs nubus-provisioning-prefill
    | Traceback (most recent call last):
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 539, in start
    |     message, payload = await protocol.read()  # type: ignore[union-attr]
    |                        ^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/streams.py", line 703, in read
    |     await self._waiter
    | asyncio.exceptions.CancelledError
    |
    | The above exception was the direct cause of the following exception:
    |
    | Traceback (most recent call last):
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/prefill_service.py", line 119, in handle_message
    |     await self._handle_message(validated_message)
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/prefill_service.py", line 142, in _handle_message
    |     await self.fetch_udm(message.subscription_name, realm_topic.topic)
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/prefill_service.py", line 160, in fetch_udm
    |     await self._fill_udm_topic(this_topic, subscription_name)
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/prefill_service.py", line 181, in _fill_udm_topic
    |     urls = await self.udm.list_objects(object_type)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 193, in async_wrapped
    |     return await copy(fn, *args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 112, in __call__
    |     do = await self.iter(retry_state=retry_state)
    |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 157, in iter
    |     result = await action(retry_state)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/_utils.py", line 111, in inner
    |     return call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 393, in <lambda>
    |     self._add_action_func(lambda rs: rs.outcome.result())
    |                                      ^^^^^^^^^^^^^^^^^^^
    |   File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    |     return self.__get_result()
    |            ^^^^^^^^^^^^^^^^^^^
    |   File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    |     raise self._exception
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 116, in __call__
    |     result = await fn(*args, **kwargs)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/univention/provisioning/prefill/udm_adapter.py", line 89, in list_objects
    |     async with self._session.get(f"{self.base_url}{object_type}/", params=params) as request:
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 1510, in __aenter__
    |     self._resp: _RetType = await self._coro
    |                            ^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 779, in _request
    |     resp = await handler(req)
    |            ^^^^^^^^^^^^^^^^^^
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 757, in _connect_and_send_request
    |     await resp.start(conn)
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 534, in start
    |     with self._timer:
    |   File "/app/prefill/.venv/lib/python3.11/site-packages/aiohttp/helpers.py", line 713, in __exit__
    |     raise asyncio.TimeoutError from exc_val
    | TimeoutError
    +------------------------------------
/entrypoint.sh: No files found in /entrypoint.d/, skipping configuration

Fixed in Version 2.2.0

This issue is fixed in Provisioning-Service Version 2.2.0.

Verify the deployed container versions:

root@ucs5primary:~# docker ps
CONTAINER ID   IMAGE                                                                              COMMAND                  CREATED          STATUS          PORTS                                                                                            NAMES
eaa7538a44ab   artifacts.software-univention.de/nubus/images/provisioning-dispatcher:2.2.0        "tini -- /entrypoint…"   31 seconds ago   Up 31 seconds                                                                                                    nubus-provisioning-dispatcher
e7e7a6cd67d0   artifacts.software-univention.de/nubus/images/provisioning-prefill:2.2.0           "tini -- /entrypoint…"   31 seconds ago   Up 31 seconds                                                                                                    nubus-provisioning-prefill
651a5354bb2d   artifacts.software-univention.de/nubus/images/provisioning-udm-transformer:2.2.0   "tini -- /entrypoint…"   32 seconds ago   Up 31 seconds                                                                                                    nubus-provisioning-udm-transformer
75f9011b906a   artifacts.software-univention.de/nubus/images/provisioning-api:2.2.0               "tini -- /entrypoint…"   32 seconds ago   Up 31 seconds   0.0.0.0:40000->7777/tcp, :::40000->7777/tcp                                                      nubus-provisioning-api
3cd99161c4c4   artifacts.software-univention.de/library/nats:2.14.3                               "/nats-server -c /ru…"   33 seconds ago   Up 32 seconds   0.0.0.0:4222->4222/tcp, :::4222->4222/tcp, 0.0.0.0:8222->8222/tcp, :::8222->8222/tcp, 6222/tcp   provisioning-service_nats_1

Check the updated NATS maximum payload to 16MB:

root@ucs5primary:~# curl -s http://127.0.0.1:8222/varz | grep max_payload
  "max_payload": 16777216,

Check the UDM connection configuration of the Prefill container:

root@ucs5primary:~# docker exec nubus-provisioning-prefill env | grep -i udm
UDM_HOST=ucs5primary.univention.intranet
UDM_PORT=443
UDM_URL_PATH_PREFIX=/univention
UDM_PROTOCOL=https
UDM_USERNAME=ucs5primary$
UDM_PASSWORD=4aIH07nfJCvd6EPyX5SK

Verify the UDM REST API request:

curl -k -s -o /dev/null -w "\n=== DETAILLIERTE ZEITMESSUNG ===\nDNS-Auflösung:                      %{time_namelookup}s\nTCP-Verbindungsaufbau:              %{time_connect}s\nTLS-Handshake:                      %{time_appconnect}s\nServer-Verarbeitung (LDAP + JSON):  %{time_starttransfer}s\nGesamte Download-Dauer:             %{time_total}s\n\nHTTP-Status:                        %{http_code}\nÜbertragene Datenmenge:             %{size_download} Bytes\n" \
-H "Accept: application/json" \
-u "ucs5primary\$:4aIH07nfJCvd6EPyX5SK" \
"https://ucs5primary.univention.intranet/univention/udm/users/user/?scope=sub&hidden=true&properties=NonExistantDummyProperty&page=1&limit=0&dir=ASC"

=== DETAILLIERTE ZEITMESSUNG ===
DNS-Auflösung:                      0.000444s
TCP-Verbindungsaufbau:              0.000531s
TLS-Handshake:                      0.004914s
Server-Verarbeitung (LDAP + JSON):  27.346697s
Gesamte Download-Dauer:             27.401551s

HTTP-Status:                        200
Übertragene Datenmenge:             19756964 Bytes

Check the subscription configuration:

root@ucs5primary:~# cat provisioning-api.json
{
  "name": "example-consumer",
  "realms_topics": [
    {
      "realm": "udm",
      "topic": "users/user"
    }
  ],
  "request_prefill": true,
  "password": "SoM3-reA1ly_sEc4eT_Pa33morD"
}

Retrieve the nubus-provisioning-api credentials

root@ucs5primary:~/univention-support/substring# docker exec nubus-provisioning-api env | grep -i admin
ADMIN_NATS_USER=NOT_SET_NOT_REQUIRED
ADMIN_NATS_PASSWORD=NOT_SET_NOT_REQUIRED
ADMIN_USERNAME=admin
ADMIN_PASSWORD=pJoASmIEFKnhdWeH6fb8S0YAJUWPd7i4

Export the administrator credentials:

root@ucs5primary:~# export ADMIN_USERNAME=$(docker exec nubus-provisioning-api printenv ADMIN_USERNAME)
root@ucs5primary:~# export ADMIN_PASSWORD=$(docker exec nubus-provisioning-api printenv ADMIN_PASSWORD)

Delete the existing subscription:

root@ucs5primary:~# curl -k -X  DELETE -u "example-consumer:SoM3-reA1ly_sEc4eT_Pa33morD" "https://localhost/univention/provisioning/v1/subscriptions/example-consumer"

Recreate the subscription:

root@ucs5primary:~# curl -k -X POST -u "$ADMIN_USERNAME":"$ADMIN_PASSWORD" "https://localhost/univention/provisioning/v1/subscriptions" -H "Content-Type: application/json" --data @provisioning-api.json
null

The subscription initially has the status running:

root@ucs5primarycurl -k -X GET -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" "https://localhost/univention/provisioning/v1/subscriptions"ons"
[{"name":"example-consumer","realms_topics":[{"realm":"udm","topic":"users/user"}],"request_prefill":true,"prefill_queue_status":"running"}]

After the prefill completes, the status changes to done:

root@ucs5primary:~# curl -k -X GET -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" "https://localhost/univention/provisioning/v1/subscriptions"
[{"name":"example-consumer","realms_topics":[{"realm":"udm","topic":"users/user"}],"request_prefill":true,"prefill_queue_status":"done"}]

Solution

The issue is fixed in Provisioning-Service Version 2.2.0. Upgrade the Provisioning Service to the latest available version:

univention-app update
univention-app upgrade provisioning-service

References

https://docs.software-univention.de/manual/5.2/en/domain-ldap/nubus-provisioning-service.html#provisioning-service