OwnCloud on UCS fails to start after blackout / hard power cycle with 502 bad gateway

UCS - Univention Corporate Server
#1

I’m using the VMware image instance of UCS 4.4-with-owncloud.

After a blackout last night, I noticed my ownCloud is not working.

Going to https://ipaddr/univention/portal/

  • Can log in
  • Can see the main menu

Going to https://ipaddr/owncloud

  • Error 502
  • Displayed in the browser - Bad Gateway The proxy server received an invalid response from an upstream server. Apache/2.4.25 (Univention) Server at 172.16.30.13 Port 443

I logged into the console, and ran docker logs owncloud_owncloud_1 > ~/oc.log

This seems to be the first sign of trouble after the reboot:

[29.LDAP.sh] no previous installation found
Upgrading server database…
An unhandled exception has been thrown:
Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occurred in driver: SQLSTATE[HY000] [2002] Connection refused in /var/www/owncloud/lib/private/DB/Connection.php:62
Stack trace:
#0 /var/www/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(474): OC\DB\Connection->connect()
#1 /var/www/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(436): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /var/www/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(378): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /var/www/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(843): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /var/www/owncloud/lib/private/DB/Connection.php(148): Doctrine\DBAL\Connection->setTransactionIsolation()
#5 /var/www/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(260): OC\DB\Connection->__construct()
#6 /var/www/owncloud/lib/private/DB/ConnectionFactory.php(139): Doctrine\DBAL\DriverManager::getConnection()
#7 /var/www/owncloud/lib/private/Server.php(552): OC\DB\ConnectionFactory->getConnection()
#8 /var/www/owncloud/lib/composer/pimple/pimple/src/Pimple/Container.php(122): OC\Server->OC{closure}()
#9 /var/www/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(108): Pimple\Container->offsetGet()
#10 /var/www/owncloud/lib/private/ServerContainer.php(86): OC\AppFramework\Utility\SimpleContainer->query()
#11 /var/www/owncloud/lib/private/Server.php(1260): OC\ServerContainer->query()
#12 /var/www/owncloud/lib/private/Server.php(424): OC\Server->getDatabaseConnection()
#13 /var/www/owncloud/lib/composer/pimple/pimple/src/Pimple/Container.php(122): OC\Server->OC{closure}()
#14 /var/www/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(108): Pimple\Container->offsetGet()
#15 /var/www/owncloud/lib/private/ServerContainer.php(86): OC\AppFramework\Utility\SimpleContainer->query()
#16 /var/www/owncloud/lib/private/Server.php(1190): OC\ServerContainer->query()
#17 /var/www/owncloud/lib/private/Server.php(607): OC\Server->getAppConfig()
#18 /var/www/owncloud/lib/composer/pimple/pimple/src/Pimple/Container.php(122): OC\Server->OC{closure}()
#19 /var/www/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(108): Pimple\Container->offsetGet()
#20 /var/www/owncloud/lib/private/ServerContainer.php(86): OC\AppFramework\Utility\SimpleContainer->query()
#21 /var/www/owncloud/lib/private/Server.php(1458): OC\ServerContainer->query()
#22 /var/www/owncloud/lib/private/legacy/app.php(344): OC\Server->getAppManager()
#23 /var/www/owncloud/lib/private/legacy/app.php(108): OC_App::getEnabledApps()
#24 /var/www/owncloud/lib/base.php(590): OC_App::loadApps()
#25 /var/www/owncloud/lib/base.php(1070): OC::init()
#26 /var/www/owncloud/console.php(74): require_once(’/var/www/ownclo…’)
#27 /var/www/owncloud/occ(11): require_once(’/var/www/ownclo…’)
#28 {main}Disabling OpenID Connect app…

Is it possible that the DB was damaged by the hard power cycle? How can I fix this?

This issue has happened to me twice, and I “solved” it with a full reinstall… but don’t want to go through that again!

Please provide specific commands, because I’m new to docker and don’t really know how to troubleshoot this myself.

Thanks in advance!

#2

I am adding my debugging notes here, in case it’s of use to someone else.

I found that mariadb runs on the “host”, not the container.

Confirmed that mariadb won’t start:

root@owncloud:~# service mysql start
Job for mariadb.service failed because a fatal signal was delivered to the control process.
See “systemctl status mariadb.service” and “journalctl -xe” for details.

root@owncloud:~# systemctl status mariadb.service
● mariadb.service - MariaDB 10.1.48 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/mariadb.service.d
└─univention.conf
Active: activating (start) since Sat 2023-04-22 08:58:53 AEST; 595ms ago
Docs: man:mysqld(8)
systemd - MariaDB Knowledge Base
Process: 55511 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 55509 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
Process: 70382 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=cd /usr/bin/..; /usr/bin/galera_recovery; [ $? -eq 0 ] &&
Process: 70377 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 70376 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 70494 (mysqld)
Status: “Starting Innodb crash recovery”
Tasks: 2 (limit: 19660)
Memory: 3.3M
CPU: 222ms
CGroup: /system.slice/mariadb.service
‣ 70494 [mysqld]

Apr 22 08:58:53 owncloud systemd[1]: Stopped MariaDB 10.1.48 database server.
Apr 22 08:58:53 owncloud systemd[1]: Starting MariaDB 10.1.48 database server…
Apr 22 08:58:53 owncloud mysqld[70494]: 2023-04-22 8:58:53 140027127754112 [Note] /usr/sbin/mysqld (mysqld 10.1.48-MariaDB-0+deb9u2) starting as process
Apr 22 08:58:53 owncloud systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
Apr 22 08:58:53 owncloud systemd[1]: Failed to start MariaDB 10.1.48 database server.
Apr 22 08:58:53 owncloud systemd[1]: mariadb.service: Unit entered failed state.
Apr 22 08:58:53 owncloud systemd[1]: mariadb.service: Failed with result ‘signal’.

Now to figure out what’s going on from here…

#3

After quite a lot of time spent researching this, I concluded that the innodb tables were corrupted, and all documentation I found to repair it relied on having a backup. Well, I didn’t know if UCS configured any backups… and instead of going further down the rabbit hole, I ran out of time.

In the end, my fastest solution is to reinstall from scratch, and sync data back onto OwnCloud from one of my clients.

I learnt my lesson - never subject OwnCloud to a hard power cycle. So I’m now running the VM on an old laptop, with effectively a 5 hour UPS :slight_smile:

#4

Hi I have exact same issue, when the UCS crashes owncloud cannot be recovered , whole appliance has to be restored from the backup …I cannot keep doing that all the time. I have done that at least 2 times.

this is very annoying … any fix for this yet ?

#5

I agree it’s annoying. The weird thing is that my server has very little activity, and the crash happened overnight, when there was no activity for hours! I can understand a DB corruption if the power goes out during the middle of transactions, but a fatal corruption during a quiescent time seems like a flaw in MariaDB. It’s hard to believe, as databases with transaction log recovery have existed for decades…

Mastodon