UCS5 & Nextcloud is a gamble

Hi@all,

I have already done several installation runs of a UCS5 environment. Apps that are installed as Docker such as Nextcloud behave predictably. Only the basic configuration done. Suddenly the URL is no longer accessible → 404

Short look at the UCS. The container is gone.

A look at the Appcenter first gives the impression:

Nextcloud = Installed

Let’s go to “Manage application” → Start → Does not work → Apply which brings me back to the app overview. Now there is:

Nextcloud = Not installed

This is a test environment. So no stress :slight_smile: Sure, I do a rollback in Proxmox to the state before.

If any logfiles or similar are helpful for Univention just let me know which ones.

with best
sven

Hi @pixel ,
thanks for sharing :slight_smile:
Would you mind to tell me if this was a new installation or an upgrade and post me your:

univention-app info 
docker images  |grep --color nextcloud
docker ps  |grep --color nextcloud

Thanks ahead

All the UCS5 test environments are completely reinstalled.

root@cloud01:~# univention-app info
UCS: 5.0-1 errata218
Installed: 
Upgradable:
root@cloud01:~# docker images  |grep --color nextcloud
docker.software-univention.de/nextcloud   21.0.7-0            1c36271a798a        2 months ago        1.18GB
docker.software-univention.de/nextcloud   20.0.9-0            1e6e540ebbd7        10 months ago       1.11GB
root@cloud01:~# docker ps  |grep --color nextcloud
root@cloud01:~#

Thanks

Please use this article to remove :

docker.software-univention.de/nextcloud   20.0.9-0            1e6e540ebbd7        10 months ago       1.11GB

and try again.

I hope this helps

We run into the same problem after installing updates around 5.0-1-errata193. After some time UCS reports that nextcloud isn’t installed anymore. Maybe it’s related to 5.0-1-errata196?

Even after removing all Nextcloud containers and images, reboot and reinstall, the error comes back after some time. This is the third try for us.

After installation univention-app info reports correctly:

UCS: 5.0-1 errata218
Installed: dhcp-server=12.0 mailserver=12.0 open-xchange-guard=2.10.5-ucs1 open-xchange-text=7.10.5-ucs1 oxseforucs=7.10.5-ucs4 samba4=4.13 4.4/nextcloud=21.0.7-0
Upgradable:

But after server is running for about 30 minutes it reports like this and in the app center you should install Nextcloud again:

UCS: 5.0-1 errata218
Installed: dhcp-server=12.0 mailserver=12.0 open-xchange-guard=2.10.5-ucs1 open-xchange-text=7.10.5-ucs1 oxseforucs=7.10.5-ucs4 samba4=4.13
Upgradable:

The Nextcloud container can then still be up, running and accessible via web. But when it isn’t accessible anymore, the output of journalctl -xb | grep docker then shows something like this:

Feb 12 16:53:45 testsrv named[1408]: listening on IPv4 interface docker0, 172.17.43.1#53
Feb 12 16:53:45 testsrv kernel: IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
Feb 12 16:53:46 testsrv dhcpd[1803]: No subnet declaration for docker0 (172.17.43.1).
Feb 12 16:53:46 testsrv dhcpd[1803]: ** Ignoring requests on docker0.  If this is not what
Feb 12 16:53:46 testsrv dhcpd[1803]:    to which interface docker0 is attached. **
Feb 12 16:53:47 testsrv dockerd[1487]: time="2022-02-12T16:53:47.678965599+01:00" level=info msg="Removing stale sandbox ID1 (ID2)"
Feb 12 16:53:47 testsrv dockerd[1487]: time="2022-02-12T16:53:47.910935113+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint ID1 ID2], retrying...."
Feb 12 16:53:47 testsrv systemd[1]: run-docker-netns-123456789012.mount: Succeeded.
-- The unit run-docker-netns-123456789012.mount has successfully entered the 'dead' state.

So the container stalls and get stopped, but until now we didn’t know why and only the Nextcloud container is affected. In our search for the error we also discovered that the output of univention-app is very inconstant.

Most times it’s correct:

nextcloud
  Name: Nextcloud Hub
  Latest version: 21.0.7-0
  Installations: testsrv

But after some time, it shows this:

nextcloud
  Name: Nextcloud Hub
  Latest version: 21.0.3-0
  Installations:

or that:

nextcloud
  Name: Nextcloud Hub
  Latest version: 21.0.5-0
  Installations:

Removing cached data for 4.x and 5.0 like stated in this post doesn’t solve the problem. When running univention-app remove nextcloud it want to remove the version it thinks is the actual one, but this is not always the installed one.
After running univention-app update mostly points to the real current version, but even when it reports the server under installations, it isn’t listed in univention-app info after it is gone there.

@pixel
The container shouldn’t be “gone”. It should be just stopped or stalled.

Try docker container ls --all to show up also stopped container.

Hi @pixel @HardCoreTec ,
adding information helped me to see a pattern in the behavior of Nextcloud in which I would like to invest a little deeper before I’ll get back to you. As you’ve told that the error comes back after some time.. I am yet not able to say what to do. Please remain patient.
Thanks ahead.

Yes, no problem for me at least. The “snapshot button” in Proxmox is my friend :slight_smile:

Without wanting to anthropomorphize “my” UCS5 I try to interpret “its” behavior.

It seems that the behavior always occurs in combination with another Docker app. For me it was Collabra-Dev. Either NextCloud remained installed or Collabra. And somehow I have the feeling that it only happens on the Node. Several test runs where all apps were installed on one master did not show this behavior.

At the end of the day I managed to install both. But not via the web browser → App-Center but via univention-app-install.

This only worked when the “old” installation was completely removed:

  • Uninstall app
  • Remove UCR varibles
  • Clean database
  • Remove application folder

I hope that helps

with best
sven

Hellp @pixel @HardCoreTec,

thanks for waiting. The error could be cause be a now known bug. Please wait till we publish an errata or check if the workaround in this article can solve your problem.

Please let me know.

Thanks for the info. I would like to add that during my tests not only the app in the app center disappeared.

The Docker container was also gone.

In the case of NextCloud and Collabora-Dev, I removed both (including DB, App directory and UCR variables). First installed NextCloud and the offered update.

Subsequently installed Collabora-Dev. Since then, the problems have not occurred.

Sometimes an installed app disappears in the App Center. I have also noticed this with Jitsi. However, it only disappears in the display. The Docker container remains there.

I am not sure but I think that in the previous attempt I installed Collabora-Dev first and then NextCloud.

Maybe the containers also disappeared when both are installed and only then the update for NC is installed.

I will test it further.

Thanks for the feedback. As far as I know the behaviors is not logical. Your pattern might differ to mine while the root is the same. So I would suggest to apply the workaround and check the behavior. Hope this helps.

Thanks for getting into it so fast.

We applied the workaround yesterday, fully removed (images, container) and reinstalled Nextcloud, but as described after ~30 minutes, it was gone from univention-app info and ~1h later, the URL was not accessible anymore. :roll_eyes:

We then reinstalled Nextcloud again, until now it’s detected and running. But this was also the case on 5.0-x when it was running for 1-2 weeks before the problems started.

There seems to be definitely more than one cache problem on UCS 5.0-1:
In the appcenter logfile the cache is complaining every hour [or after running univention-app update manually] about an inaccessible .apps.en.json file even when it was created or modified by itself.

Shorted output of /var/log/univention/appcenter.log:

 31908 actions.update                   22-02-14 15:07:03 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/app-categories.ini"...
 31908 actions.update                   22-02-14 15:07:03 [   DEBUG]:   ... Not Modified
 31908 actions.update                   22-02-14 15:07:03 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/rating.ini"...
 31908 actions.update                   22-02-14 15:07:03 [   DEBUG]:   ... Not Modified
 31908 actions.update                   22-02-14 15:07:03 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/license_types.ini"...
 31908 actions.update                   22-02-14 15:07:03 [   DEBUG]:   ... Not Modified
 31908 actions.update                   22-02-14 15:07:03 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/ucs.ini"...
 31908 actions.update                   22-02-14 15:07:03 [   DEBUG]:   ... Not Modified
 31908 actions.update                   22-02-14 15:07:03 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/suggestions.json"...
 31908 actions.update                   22-02-14 15:07:03 [   DEBUG]:   ... Not Modified
 31908 actions.update                   22-02-14 15:07:03 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/5.0/all.tar.gpg"...
 31908 actions.update                   22-02-14 15:07:03 [    INFO]: Downloading "http://appcenter.software-univention.de/meta-inf/5.0/all.tar.zsync"...
 31908 actions.update                   22-02-14 15:07:03 [   DEBUG]: Calling zsync http://appcenter.software-univention.de/meta-inf/5.0/all.tar.zsync -q -o /var/cache/univention-appcenter/appcenter.software-univention.de/5.0/.all.tar
 31908 actions.update                   22-02-14 15:07:05 [   DEBUG]: Extracting archive in /var/cache/univention-appcenter/appcenter.software-univention.de/5.0
 31908 actions.update                   22-02-14 15:07:05 [   DEBUG]: Removing old files...
 31908 actions.update                   22-02-14 15:07:05 [   DEBUG]: Unpacking /var/cache/univention-appcenter/appcenter.software-univention.de/5.0/.all.tar...
 31908 actions.update                   22-02-14 15:07:05 [   DEBUG]: Calling tar -C /var/cache/univention-appcenter/appcenter.software-univention.de/5.0 -xf /var/cache/univention-appcenter/appcenter.software-univention.de/5.0/.all.tar
 31908 actions.update                   22-02-14 15:07:06 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/4.4/all.tar.gpg"...
 31908 actions.update                   22-02-14 15:07:06 [    INFO]: Downloading "http://appcenter.software-univention.de/meta-inf/4.4/all.tar.zsync"...
 31908 actions.update                   22-02-14 15:07:06 [   DEBUG]: Calling zsync http://appcenter.software-univention.de/meta-inf/4.4/all.tar.zsync -q -o /var/cache/univention-appcenter/appcenter.software-univention.de/4.4/.all.tar
 31908 actions.update                   22-02-14 15:07:07 [   DEBUG]: Extracting archive in /var/cache/univention-appcenter/appcenter.software-univention.de/4.4
 31908 actions.update                   22-02-14 15:07:07 [   DEBUG]: Removing old files...
 31908 actions.update                   22-02-14 15:07:07 [   DEBUG]: Unpacking /var/cache/univention-appcenter/appcenter.software-univention.de/4.4/.all.tar...
 31908 actions.update                   22-02-14 15:07:07 [   DEBUG]: Calling tar -C /var/cache/univention-appcenter/appcenter.software-univention.de/4.4 -xf /var/cache/univention-appcenter/appcenter.software-univention.de/4.4/.all.tar
 31908 actions.update                   22-02-14 15:07:08 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/4.3/all.tar.gpg"...
 31908 actions.update                   22-02-14 15:07:08 [    INFO]: Downloading "http://appcenter.software-univention.de/meta-inf/4.3/all.tar.zsync"...
 31908 actions.update                   22-02-14 15:07:08 [   DEBUG]: Calling zsync http://appcenter.software-univention.de/meta-inf/4.3/all.tar.zsync -q -o /var/cache/univention-appcenter/appcenter.software-univention.de/4.3/.all.tar
 31908 actions.update                   22-02-14 15:07:09 [   DEBUG]: Extracting archive in /var/cache/univention-appcenter/appcenter.software-univention.de/4.3
 31908 actions.update                   22-02-14 15:07:09 [   DEBUG]: Removing old files...
 31908 actions.update                   22-02-14 15:07:09 [   DEBUG]: Unpacking /var/cache/univention-appcenter/appcenter.software-univention.de/4.3/.all.tar...
 31908 actions.update                   22-02-14 15:07:09 [   DEBUG]: Calling tar -C /var/cache/univention-appcenter/appcenter.software-univention.de/4.3 -xf /var/cache/univention-appcenter/appcenter.software-univention.de/4.3/.all.tar
 31908 cache                            22-02-14 15:07:09 [   DEBUG]: Cache outdated. Need to rebuild
 31908 cache                            22-02-14 15:07:09 [   DEBUG]: Unable to get mtime for cache: [Errno 2] No such file or directory: '/var/cache/univention-appcenter/appcenter.software-univention.de/5.0/.apps.en.json'
 31908 cache                            22-02-14 15:07:09 [   DEBUG]: Cannot load cache: mtimes of cache files do not match: None < 1644847626.1846244
 31908 cache                            22-02-14 15:07:09 [   DEBUG]: Saved 42 apps into cache
 31908 cache                            22-02-14 15:07:09 [   DEBUG]: Cache outdated. Need to rebuild
 31908 cache                            22-02-14 15:07:09 [   DEBUG]: Unable to get mtime for cache: [Errno 2] No such file or directory: '/var/cache/univention-appcenter/appcenter.software-univention.de/4.4/.apps.en.json'
 31908 cache                            22-02-14 15:07:09 [   DEBUG]: Cannot load cache: mtimes of cache files do not match: None < 1644847627.9886723
 31908 cache                            22-02-14 15:07:11 [   DEBUG]: Saved 425 apps into cache
 31908 cache                            22-02-14 15:07:11 [   DEBUG]: Cache outdated. Need to rebuild
 31908 cache                            22-02-14 15:07:11 [   DEBUG]: Unable to get mtime for cache: [Errno 2] No such file or directory: '/var/cache/univention-appcenter/appcenter.software-univention.de/4.3/.apps.en.json'
 31908 cache                            22-02-14 15:07:11 [   DEBUG]: Cannot load cache: mtimes of cache files do not match: None < 1644847629.452711
 31908 cache                            22-02-14 15:07:12 [   DEBUG]: Saved 255 apps into cache
 31908 actions.update                   22-02-14 15:07:12 [   DEBUG]: Updating app files...
 31908 actions.update                   22-02-14 15:07:12 [    INFO]: File: /usr/share/univention-management-console/modules/apps.xml
 31908 actions.update                   22-02-14 15:07:12 [    INFO]: 
 31908 actions.update                   22-02-14 15:07:12 [    INFO]: Multifile: /etc/apache2/sites-available/000-default.conf
 31908 actions.update                   22-02-14 15:07:12 [    INFO]: 
 31908 actions.update                   22-02-14 15:07:13 [    INFO]: Multifile: /etc/apache2/sites-available/default-ssl.conf
 31908 actions.update                   22-02-14 15:07:13 [    INFO]: 
 31908 actions.update                   22-02-14 15:07:13 [    INFO]: File: /usr/share/univention-management-console/i18n/de/apps.mo
 31908 actions.update                   22-02-14 15:07:13 [    INFO]: 
 31908 actions.update                   22-02-14 15:07:15 [   DEBUG]: Updating app icon files in UMC directory...

...

  3459 actions.update                   22-02-14 16:13:21 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/app-categories.ini"...
  3459 actions.update                   22-02-14 16:13:21 [   DEBUG]:   ... Not Modified
  3459 actions.update                   22-02-14 16:13:21 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/rating.ini"...
  3459 actions.update                   22-02-14 16:13:21 [   DEBUG]:   ... Not Modified
  3459 actions.update                   22-02-14 16:13:21 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/license_types.ini"...
  3459 actions.update                   22-02-14 16:13:21 [   DEBUG]:   ... Not Modified
  3459 actions.update                   22-02-14 16:13:21 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/ucs.ini"...
  3459 actions.update                   22-02-14 16:13:21 [   DEBUG]:   ... Not Modified
  3459 actions.update                   22-02-14 16:13:21 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/suggestions.json"...
  3459 actions.update                   22-02-14 16:13:21 [   DEBUG]:   ... Not Modified
  3459 actions.update                   22-02-14 16:13:21 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/5.0/all.tar.gpg"...
  3459 actions.update                   22-02-14 16:13:21 [   DEBUG]:   ... Not Modified
  3459 actions.update                   22-02-14 16:13:21 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/4.4/all.tar.gpg"...
  3459 actions.update                   22-02-14 16:13:22 [   DEBUG]:   ... Not Modified
  3459 actions.update                   22-02-14 16:13:22 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/4.3/all.tar.gpg"...
  3459 actions.update                   22-02-14 16:13:22 [   DEBUG]:   ... Not Modified
  3459 actions.update.progress          22-02-14 16:13:22 [   DEBUG]: 100
  3459 cache                            22-02-14 16:13:22 [   DEBUG]: Cache outdated. Need to rebuild
  3459 cache                            22-02-14 16:13:22 [   DEBUG]: Unable to get mtime for cache: [Errno 2] No such file or directory: '/var/cache/univention-appcenter/appcenter.software-univention.de/5.0/.apps.en.json'
  3459 cache                            22-02-14 16:13:22 [   DEBUG]: Cannot load cache: mtimes of cache files do not match: None < 1644847626.1846244
  3459 cache                            22-02-14 16:13:22 [   DEBUG]: Saved 42 apps into cache
  3459 cache                            22-02-14 16:13:22 [   DEBUG]: Cache outdated. Need to rebuild
  3459 cache                            22-02-14 16:13:22 [   DEBUG]: Unable to get mtime for cache: [Errno 2] No such file or directory: '/var/cache/univention-appcenter/appcenter.software-univention.de/4.4/.apps.en.json'
  3459 cache                            22-02-14 16:13:22 [   DEBUG]: Cannot load cache: mtimes of cache files do not match: None < 1644847627.9886723
  3459 cache                            22-02-14 16:13:23 [   DEBUG]: Saved 425 apps into cache
  3459 cache                            22-02-14 16:13:23 [   DEBUG]: Cache outdated. Need to rebuild
  3459 cache                            22-02-14 16:13:23 [   DEBUG]: Unable to get mtime for cache: [Errno 2] No such file or directory: '/var/cache/univention-appcenter/appcenter.software-univention.de/4.3/.apps.en.json'
  3459 cache                            22-02-14 16:13:23 [   DEBUG]: Cannot load cache: mtimes of cache files do not match: None < 1644847629.452711
  3459 cache                            22-02-14 16:13:24 [   DEBUG]: Saved 255 apps into cache

This behavior also occurs after running rm /var/cache/univention-appcenter/appcenter.software-univention.de/[4|5|.*/.apps.*json.

e.g. output of ls -la /var/cache/univention-appcenter/appcenter.software-univention.de/5.0/.apps.en.json:

-rw-r--r-- 1 root root 864406 Feb 14 16:13 /var/cache/univention-appcenter/appcenter.software-univention.de/5.0/.apps.en.json

So univention-app update [re]creates the file correctly, but can’t access it afterwards.

Ok, thanks a lot.
I keep looking into it and keep you informed. Don’t let your good friend “snapshot button” leave the house till then…:wink:
best regards

This is more or less how it happened for me too.

I think you already have data in the Nextcloud and a complete removal is out of the question?

I think so too and wanted to express this with post no.8.

Hi,
thanks for the update. Please check the progress here too.

Hello everybody @HardCoreTec @pixel , we have published an errata which should fix this problem. Please try again and let me know.

I have already installed a test environment (1 x master, 2 x node) with the current version.

All the apps I had problems with before (suddenly disappeared) could now be installed without any problems.

I will continue to test it and report back.

Thanks, the errate was installed ?

Yes. Systems were newly installed → Update → Apps

Nice :slight_smile: Let me know please.
Thanks ahead.

As this error doesn’t happen immediately to us, we are need some days to confirm that the error is really gone.

Right now, we can confirm that it fixes the at least the bug were it can’t access the cache files. After the bugfix the 'Unable to get mtime for cache' error is gone and /var/log/univention/appcenter.log looks like this:

 12836 actions.update                   22-02-16 18:13:00 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/app-categories.ini"...
 12836 actions.update                   22-02-16 18:13:01 [   DEBUG]:   ... Not Modified
 12836 actions.update                   22-02-16 18:13:01 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/rating.ini"...
 12836 actions.update                   22-02-16 18:13:01 [   DEBUG]:   ... Not Modified
 12836 actions.update                   22-02-16 18:13:01 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/license_types.ini"...
 12836 actions.update                   22-02-16 18:13:01 [   DEBUG]:   ... Not Modified
 12836 actions.update                   22-02-16 18:13:01 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/ucs.ini"...
 12836 actions.update                   22-02-16 18:13:01 [   DEBUG]:   ... Not Modified
 12836 actions.update                   22-02-16 18:13:01 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/suggestions.json"...
 12836 actions.update                   22-02-16 18:13:01 [   DEBUG]:   ... Not Modified
 12836 actions.update                   22-02-16 18:13:01 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/5.0/all.tar.gpg"...
 12836 actions.update                   22-02-16 18:13:01 [   DEBUG]:   ... Not Modified
 12836 actions.update                   22-02-16 18:13:01 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/4.4/all.tar.gpg"...
 12836 actions.update                   22-02-16 18:13:01 [   DEBUG]:   ... Not Modified
 12836 actions.update                   22-02-16 18:13:01 [    INFO]: Downloading "https://appcenter.software-univention.de/meta-inf/4.3/all.tar.gpg"...
 12836 actions.update                   22-02-16 18:13:01 [   DEBUG]:   ... Not Modified
 12836 actions.update.progress          22-02-16 18:13:01 [   DEBUG]: 100
 12836 cache                            22-02-16 18:13:01 [   DEBUG]: Cache outdated. Need to rebuild
 12836 cache                            22-02-16 18:13:01 [   DEBUG]: Loaded 42 apps from cache
 12836 cache                            22-02-16 18:13:01 [   DEBUG]: Cache outdated. Need to rebuild
 12836 cache                            22-02-16 18:13:01 [   DEBUG]: Loaded 425 apps from cache
 12836 cache                            22-02-16 18:13:01 [   DEBUG]: Cache outdated. Need to rebuild
 12836 cache                            22-02-16 18:13:02 [   DEBUG]: Loaded 255 apps from cache

Now it does not rebuild the cache and load the apps from it. But univention-app update still complaining about outdated cache files, which seems to be strange when nothing is updated/modified.