Problem: Kelvin REST API Upgrade Fails Due to Docker Digest Mismatch

Knowledge Base Community
, , , ,
1

Problem: Kelvin REST API Upgrade Fails Due to Docker Digest Mismatch

Summary

On a UCS 5.2 system, upgrading the ucsschool Kelvin REST API application from version 1.10.3 to newer releases fails. The upgrade process aborts with Docker image verification errors, preventing both the intermediate upgrade to version 2.0.0 and the direct upgrade to the latest available version (3.1.1).

Environment

  • UCS version: 5.2-3 errata283

  • Installed apps/packages:

    • ucsschool 5.2v4
    • ucsschool-kelvin-rest-api 1.10.3

  • Upgradeable app:

    • ucsschool-kelvin-rest-api

Problem Description

When attempting to upgrade the Kelvin REST API, Docker fails to pull the required image due to a digest verification error.

Error during manual Docker pull

docker pull artifacts.software-univention.de/ucsschool/ucsschool-kelvin-rest-api:2.0.0

error pulling image configuration: image config verification failed for digest
sha256:d36d89a2367c6b01c99a9c755c2c25c06a9891001a44bd9c20188552125771c5

Observation in local Docker metadata

While investigating the issue, it was discovered that the locally stored Docker metadata contains a different digest for the same image tag. The file below shows a mismatch between the expected and stored digests:

/var/lib/docker/image/overlay2/repositories.json

The digest associated with ucsschool-kelvin-rest-api:2.0.0 does not match the digest Docker attempts to verify during the pull operation.

Investigation

  • The correct digest for the Kelvin REST API image is the one ending with …1c5.
  • It is unclear why an incorrect digest is registered locally for version 2.0.0.
  • The Univention App Center upgrade path first attempts an upgrade to 2.0.0, instead of upgrading directly from 1.10.3 to the latest release (3.1.1).

Even when running the recommended commands:

univention-app update
univention-app upgrade ucsschool-kelvin-rest-api

the upgrade still fails with a similar Docker error:

filesystem layer verification failed for digest sha256:a2ea00fef38db5495570c19d5927d953c49e3f025ec09b62c199f4df9cecf6b4
Downloading Docker image artifacts.software-univention.de/ucsschool/ucsschool-kelvin-rest-api:release3.1.1 failed

Manually modifying Docker digest information is not recommended, as digests are validated during a proper docker pull and must remain consistent.

Root Cause

A mismatch between locally cached Docker image metadata and the actual image digests caused Docker’s image verification to fail. As a result, Docker was unable to correctly associate the downloaded image layers with the expected digest, which prevented the Kelvin REST API container from being pulled and blocked the upgrade process.

Solution

Remove the affected Kelvin REST API Docker images (including their associated digests) from the local Docker cache and then pull the image again.

Step 1: Remove existing Docker images

docker rmi artifacts.software-univention.de/ucsschool/ucsschool-kelvin-rest-api:2.0.0
docker rmi artifacts.software-univention.de/ucsschool/ucsschool-kelvin-rest-api:release3.1.1

This ensures that all cached layers and incorrect digest references are removed.

Step 2: Pull the latest image and upgrade the app

docker pull artifacts.software-univention.de/ucsschool/ucsschool-kelvin-rest-api:release3.1.1
univention-app upgrade ucsschool-kelvin-rest-api

After removing the stale Docker metadata, the image can be downloaded with the correct digest and the upgrade completes successfully.

Additional Notes

  • Do not edit Docker metadata files such as repositories.json manually.
  • If similar issues occur with other App Center applications, clearing the affected Docker images is a safe first troubleshooting step.
  • Ensure sufficient disk space and a stable network connection when re-pulling large Docker images.