Problem: UDM REST API 500 / 503 Gateway Errors Caused by Failed ums-stack-data Initialization

Knowledge Base Community
, , ,
1

UDM REST API 500 / 503 Gateway Errors Caused by Failed ums-stack-data Initialization

Problem

The UDM REST API produces 500 Internal Server Error and 503 Service Unavailable / Gateway errors. Consequently, all related down-stream functions including the Univention Portal, User Creation via API, Provisioning API, Self-Service, and connected components (such as the Open-Xchange Connector) become inaccessible.

Gateway Access Logs

The UDM REST API frontend proxy logs indicate HTTP 500 and 503 errors when trying to process requests:

{"time":"2026-06-08T23:21:38.365219466Z","stream":"stdout","log":"08.06.26 23:21:38.365 NETWORK ( ERROR ) : access: [ee598206-a] [GATEWAY] 500 PUT /univention/udm/settings/extended_attribute/cn%3D<ATTR_NAME>%2Ccn%3D<APP_NAME>%2Ccn%3Dcustom%20attributes%2Ccn%3Dunivention%2C<LDAP_BASE_DN> (<CLIENT_IP>) 564.28ms","kubernetes":{"pod_name":"ums-udm-rest-api-<HASH>-<ID>","namespace_name":"<NAMESPACE>","pod_id":"<POD_UUID>","labels":{"app.kubernetes.io/component":"server","app.kubernetes.io/instance":"ums","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"udm-rest-api","app.kubernetes.io/version":"12.0.7","helm.sh/chart":"nubusUdmRestApi-0.42.6","pod-template-hash":"<HASH>"},"annotations":{"checksum/configmap":"<CM_CHECKSUM_1>","cni.projectcalico.org/containerID":"<CONTAINER_ID>","cni.projectcalico.org/podIP":"<POD_IP>/32","cni.projectcalico.org/podIPs":"<POD_IP>/32","intents.otterize.com/service-name":"ums-udm-rest-api"},"host":"<NODE_NAME>","pod_ip":"<POD_IP>","container_name":"main","docker_id":"<DOCKER_ID>","container_hash":"<IMAGE_HASH>","container_image":"<IMAGE_REF>"}}

{"time":"2026-06-08T23:22:03.811923598Z","stream":"stdout","log":"08.06.26 23:22:03.811 NETWORK ( ERROR ) : access: [40e0ee7e-9] [GATEWAY] 503 GET /univention/udm/settings/data/cn=stack-data-ums.done,cn=data,cn=univention,<LDAP_BASE_DN> (<CLIENT_IP>) 0.91ms","kubernetes":{"pod_name":"ums-udm-rest-api-<HASH>-<ID>","namespace_name":"<NAMESPACE>","pod_id":"<POD_UUID>","labels":{"app.kubernetes.io/component":"server","app.kubernetes.io/instance":"ums","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"udm-rest-api","app.kubernetes.io/version":"12.0.7","helm.sh/chart":"nubusUdmRestApi-0.42.6","pod-template-hash":"<HASH>"},"annotations":{"checksum/configmap":"<CM_CHECKSUM_1>","cni.projectcalico.org/containerID":"<CONTAINER_ID>","cni.projectcalico.org/podIP":"<POD_IP>/32","cni.projectcalico.org/podIPs":"<POD_IP>/32","intents.otterize.com/service-name":"ums-udm-rest-api"},"host":"<NODE_NAME>","pod_ip":"<POD_IP>","container_name":"main","docker_id":"<DOCKER_ID>","container_hash":"<IMAGE_HASH>","container_image":"<IMAGE_REF>"}}

Investigation

The UDM REST API pod (e.g., ums-udm-rest-api-<HASH>-<ID>) operates in a degraded state. The pod’s frontend proxy accepts incoming requests but cannot forward them to its internal backend process on port 9979, triggering HTTP 599 internal connection failures and resulting in HTTP 503 Gateway responses.

This pattern typically indicates one of the following:

  • The UDM backend process has not started or has crashed while the frontend layer remains active.
  • The underlying LDAP service is unreachable, preventing backend initialization.
  • The stack-data initialization job has not completed successfully, as evidenced by failed lookups for cn=stack-data-ums.done.
  • Configuration inconsistencies: A ConfigMap checksum discrepancy between environments (e.g., testing vs. stable namespaces) may point to rollout issues.

Diagnostic Step-by-Step Commands

To gather data during investigation, execute the following commands:

  1. Current pod status:
kubectl -n <NAMESPACE> get pods -l app.kubernetes.io/name=udm-rest-api -o wide
  1. Full pod description:
kubectl -n <NAMESPACE> describe pod <UDM_REST_API_POD_NAME>
  1. Container logs (current and previous run):
kubectl -n <NAMESPACE> logs <UDM_REST_API_POD_NAME>
kubectl -n <NAMESPACE> logs <UDM_REST_API_POD_NAME> --previous
  1. Status of initialization and stack-data jobs:
kubectl -n <NAMESPACE> get jobs
kubectl -n <NAMESPACE> get pods | grep -E 'stack-data|init'
  1. Recent namespace events:
kubectl -n <NAMESPACE> get events --sort-by='.lastTimestamp' | tail -30

Collected Diagnostic Data Analysis

1. Pod Status

$ kubectl -n <NAMESPACE> get pods -l app.kubernetes.io/name=udm-rest-api -o wide
NAME                                                            READY   STATUS      RESTARTS   AGE   IP              NODE        NOMINATED NODE   READINESS GATES
ums-udm-rest-api-1-update-univention-object-identifier-<HASH>   0/1     Completed   0          9h    <POD_IP_1>      <NODE_1>    <none>           <none>
ums-udm-rest-api-<HASH>-<ID>                                    1/1     Running     0          9h    <POD_IP_2>      <NODE_2>    <none>           <none>
ums-udm-rest-api-blocklist-cleanup-<ID>-<HASH_1>               0/1     Error       0          19m   <POD_IP_3>      <NODE_3>    <none>           <none>
ums-udm-rest-api-blocklist-cleanup-<ID>-<HASH_2>               0/1     Error       0          19m   <POD_IP_4>      <NODE_3>    <none>           <none>
ums-udm-rest-api-blocklist-cleanup-<ID>-<HASH_3>               0/1     Error       0          18m   <POD_IP_5>      <NODE_3>    <none>           <none>
ums-udm-rest-api-blocklist-cleanup-<ID>-<HASH_4>               0/1     Error       0          20m   <POD_IP_6>      <NODE_3>    <none>           <none>
ums-udm-rest-api-blocklist-cleanup-<ID>-<HASH_5>               0/1     Error       0          8m5s  <POD_IP_7>      <NODE_3>    <none>           <none>
ums-udm-rest-api-blocklist-cleanup-<ID>-<HASH_6>               0/1     Error       0          16m   <POD_IP_8>      <NODE_3>    <none>           <none>
ums-udm-rest-api-blocklist-cleanup-<ID>-<HASH_7>               0/1     Error       0          13m   <POD_IP_9>      <NODE_3>    <none>           <none>

2. Status of Initialization and Stack-Data Jobs

$ kubectl -n <NAMESPACE> get jobs
NAME                                                     STATUS      COMPLETIONS   DURATION   AGE
impress-backend-init                                     Complete    1/1           36s        10h
opendesk-nextcloud-aio-cron-<ID>                         Complete    1/1           4s         82s
openproject-seeder-1                                     Complete    1/1           106s       10h
ums-keycloak-bootstrap-bootstrap-1                       Complete    1/1           2m57s      10h
ums-provisioning-register-consumers-1                    Running     0/1           10h        10h
ums-stack-data-ums-1                                     Failed      0/1           10h        10h
ums-udm-rest-api-1-update-univention-object-identifier   Complete    1/1           3m21s      10h
ums-udm-rest-api-blocklist-cleanup-<ID>                  Failed      0/1           46m        46m

$ kubectl -n <NAMESPACE> get pods | grep -E 'stack-data|init'
impress-backend-init-<HASH>       0/1     Completed   0          10h
ums-stack-data-ums-1-<HASH_1>     0/1     Init:Error  0          7h3m
ums-stack-data-ums-1-<HASH_2>     0/1     Init:Error  0          8h
ums-stack-data-ums-1-<HASH_3>     0/1     Init:Error  0          9h
ums-stack-data-ums-1-<HASH_4>     0/1     Error       0          10h
ums-stack-data-ums-1-<HASH_5>     0/1     Init:Error  0          7h34m
ums-stack-data-ums-1-<HASH_6>     0/1     Init:Error  0          8h
ums-stack-data-ums-1-<HASH_7>     0/1     Init:Error  0          9h

Root Cause Analysis

The UDM REST API pod itself (ums-udm-rest-api-1-update-univention-object-identifier) completed successfully (1/1 Complete). The failure originates upstream in the ums-stack-data-ums-1 job.

Function of ums-stack-data-ums

In Nubus, the ums-stack-data-ums job writes core “Stack Data” into LDAP and ConfigMaps. This includes shared configuration settings such as UCR values, portal entries, service URLs, Keycloak realm configurations, and provisioning parameters. Downstream services (Provisioning API, Portal, OX-Connector, Self-Service) cannot start until this job completes successfully.

Key Evidence

  1. Backoff Limit Exhaustion: The job ums-stack-data-ums-1 remained in a Failed state after 8 pod attempt retries over a 10-hour period.
  2. Init Container Failures: Multiple pods report Init:Error, indicating missing prerequisites before the main workload container starts.
  3. First Pod Execution Error: The initial pod attempt (<HASH_4>) shows status Error (not Init:Error), meaning the main container started but subsequently failed.
  4. Stuck Downstream Jobs: Dependent tasks awaiting Provisioning API activation are stuck in CrashLoopBackOff or BackOff.

Required Resources for Stack Data

The UDM REST API requires specific ConfigMaps and Secrets:

  • ums-stack-data-ums-ucr (ConfigMap containing base.conf / base-defaults.conf)
  • ums-ldap-server-admin (Secret containing the LDAP admin password)
  • ums-udm-rest-api-ldap-conf

Certificate / PKI Misconfiguration

The events log displays the following critical warning:

34s Warning ErrGetKeyPair issuer/selfsigned-issuer Error getting keypair for CA issuer: secrets "<ROOT_CERT_SECRET>" not found

Nubus 1.19.1 uses cert-manager with a self-signed issuer that retrieves the CA key pair from the designated root certificate secret (e.g., opendesk-root-cert-secret). Without this CA secret, TLS certificates cannot be issued for internal services (including LDAP-TLS, Provisioning API, internal Keycloak, UMC, and Portal Backend).

Because ums-stack-data-ums-1 writes data to LDAP via TLS, missing TLS certificates prevent connection establishment and cause the initialization process to fail.

Deep-Dive Analysis

To inspect the certificate and job status in detail, execute:

  1. Check Certificates and Issuers:
kubectl -n <NAMESPACE> get certificate,issuer,clusterissuer
kubectl -n <NAMESPACE> describe issuer
  1. Inspect ums-stack-data Pod and Container Logs:
kubectl -n <NAMESPACE> describe pod ums-stack-data-ums
kubectl -n <NAMESPACE> logs ums-stack-data-ums --all-containers --previous

(For further details on PKI integration, refer to the openDesk Enhanced External PKI Documentation).

Solution

Ensure the correct CA certificate keypair secret exists in the cluster to enable TLS communication across LDAP and internal services.

  1. Provision CA Credentials: Verify that the required root certificate secret (e.g., opendesk-root-cert-secret) is properly deployed in the target namespace.
  2. Re-trigger the Job: After verifying the PKI setup, remove the failed job to force a re-execution:
kubectl -n <NAMESPACE> delete job ums-stack-data-ums-<HASH>
  1. Verify Resolution: Confirm that ums-stack-data-ums-1 finishes with status Completed and that the UDM REST API successfully connects to the backend process on port 9979.