The upgrade of a docker app has failed. What are the next steps?
All app upgrades are extensively tested in virtual machines. Unfortunately, this does not completely prevent problems during the upgrade in the “real world”. When an app upgrade fails, the App Center leaves the app in that status for further investigation. Here are a few hints on how to deal with this situation.
Prerequisites:
- User account with Domain Admin rights, e.g. Administrator
- Remote login to UCS via SSH
1. Check App Center logfile
The App Center logfile can be found on the UCS system at /var/log/univention/appcenter.log. It contains all the information about the installation, upgrade and de-installation of apps. To find upgrade logs about the desired app, search for Going to upgrade [NAME OF APP], e.g. My App:
10463 actions.upgrade 19-06-27 17:10:12 [ INFO]: Going to upgrade My App (3.2.0)
10463 actions.register 19-06-27 17:10:12 [ DEBUG]: /usr/sbin/update-rc.d returned with 0
10463 actions.register 19-06-27 17:10:13 [ DEBUG]: Calling /bin/systemctl daemon-reload
10463 actions.upgrade 19-06-27 17:10:13 [ DEBUG]: Calling prescript (preinst)
10463 actions.register 19-06-27 17:10:13 [ DEBUG]: /bin/systemctl returned with 0
10463 actions.register 19-06-27 17:10:13 [ INFO]: Adding localhost to LDAP object
10463 actions.register 19-06-27 17:10:13 [ INFO]: Setting overview variables
10463 actions.upgrade 19-06-27 17:10:13 [ DEBUG]: Calling /var/cache/univention-appcenter/appcenter-test.software-univention.de/4.3/myapp_20190607165648.preinst --binddn uid=Administrator,cn=users,dc=w2k12,dc=test --locale en --old-version 10.0.10-2019-02-18 --error-file /tmp/tmpLjGKqe --version 10.2.0 --bindpwdfile /tmp/tmpF69cS6
10463 actions.upgrade 19-06-27 17:10:13 [ DEBUG]: /var/cache/univention-appcenter/appcenter-test.software-univention.de/4.3/myapp_20190606165648.preinst returned with 1
10463 actions.upgrade 19-06-27 17:10:13 [CRITICAL]: Unable to upgrade myapp. Aborting...
10463 actions.upgrade.progress 19-06-27 17:10:13 [ DEBUG]: 100
The App Center logfile is the first place to look for hints on what went wrong. This information can be helpful to identify the problem or to include in a forum post.
The example above says that the app’s preinst script returned with exit code 1 and thus allows further investigation starting with this script.
2. Check Join logfile
Most apps have a Univention join script for making changes to the central UCS domain, for example creating system accounts etc. If there is evidence of a failed join or joinscript in the appcenter.log, please consult the join logfile at /var/log/univention/join.log on the UCS host and look for the appid, e.g. myapp:
RUNNING 50myapp.inst
2019-06-28 15:12:27.144564097+02:00 (in joinscript_init)
Object exists: cn=services,cn=univention,dc=w2k12,dc=test
Object modified: cn=master,cn=dc,cn=computers,dc=w2k12,dc=test
some ERROR
3. App status
During the upgrade the old app version is removed and the latest app version is going to be installed. To check the status (which version is installed) the following command can be used:
$ univention-app info
UCS: 4.4-0 errata168
Installed: mailserver=12.0 appxyz1=4.44 appxyz2=10.0.10-2019-02-18
Upgradable:
If the old app version is still installed, retry the upgrade. If the new version is installed, the upgrade problem has to be identified, fixed and the upgrade manually completed. In some cases this means to remove and re-install the app.
4. Docker logs
Sometime the app’s docker container is not able to start properly. Container logs can be investigated with the command univention-app logs $APPID, which is a shortcut to docker logs CONTAINER_ID.
The container ID is normally stored in an UCR variable:
$ ucr get appcenter/apps/appid/container
9257852d2f1dea59ba96ed55d53cfc70ebbb3c6e419a3415c5447b7c4b157544
Or can be found in the output of the command docker ps. This is especially useful when the app is a so called multi container app, which uses Docker compose.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9257852d2f1d docker.software-univention.de/myapp-app:2.4 "node /build/main.js" About a minute ago Up About a minute 0.0.0.0:40002->8080/tcp myapp-app
13893a7443ad docker.software-univention.de/myapp-db:2.4 "docker-entrypoint..." About a minute ago Up About a minute 27017/tcp myapp-db
Here the container ID for the myapp App is 9257852d2f1d. Now we can see the container logs with:
$ docker logs d9f9339afe11
Note: you are using a pure-JavaScript implementation of bcrypt.
While this implementation will work correctly, it is known to be
approximately three times slower than the native implementation.
In order to use the native implementation instead, run
meteor npm install --save bcrypt
in the root directory of your application.
Presence started serverId=xwP2r5HgNgmmz82XL
Kadira: completed instrumenting the app
5. Backup all data and configs of the app
All the app’s data and configuration are normally stored in the directory /var/lib/univention-appcenter/apps/myapp. Backup this directory before starting to make changes to the UCS system.
Additionally backup the MySQL or PostgreSQL databases on the UCS system, if installed and used by the app. Some apps use the host’s database to store configuration and user data.
6. Try to uninstall the app
It may be necessary to remove the current app to have a clean setup before installing the new app version for a new installation instead of an app upgrade. This can be done on the command line with:
$ univention-app remove myappid
7. Try to install the app again
Apps can be installed on the command line with:
$ univention-app install myappid
8. Consult Univention Help
Do not hesitate to ask here on Univention Help.