After normal Errata Updates, nextcloud got an internal Error - database not found

Hello all,

this day i’ve done an normal errata update from 4.4-4 errata617 to 4.4-4 errata648. Everything look likes normal. Nextlcoud Version is 18.0.6-0. After reboot of the Server my Nextlcoud was not reachabel and i got this error in the browser:

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

The are not output in the log: /var/lib/univention-appcenter/apps/nextcloud/data/nextcloud-data/nextcloud.log

Older outputs were this:

2020-07-02T21:30:13+00:00","remoteAddr":"","user":"--","app":"core","method":"","url":"--","message":{"Exception":"OC\\ServerNotAvailableException","Message":"Lost connection to LDA
P server.","Code":0,"Trace":[{"file":"/var/www/html/apps/user_ldap/lib/LDAP.php","line":414,"function":"processLDAPError","class":"OCA\\User_LDAP\\LDAP","type":"->","args":["*** sensitive parameter replaced ***"]},{"file":"/var/www
/html/apps/user_ldap/lib/LDAP.php","line":337,"function":"postFunctionCall","class":"OCA\\User_LDAP\\LDAP","type":"->","args":[]},{"file":"/var/www/html/apps/user_ldap/lib/LDAP.php","line":274,"function":"invokeLDAPMethod","class":
"OCA\\User_LDAP\\LDAP","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/apps/user_ldap/lib/Connection.php","line":626,"function":"startTls","class":"OCA\\User_LDAP\\LDAP","type":"->","args":["***
sensitive parameter replaced ***"]},{"file":"/var/www/html/apps/user_ldap/lib/Connection.php","line":575,"function":"doConnect","class":"OCA\\User_LDAP\\Connection","type":"->","args":["darkdevil.osit.cc","7389"]},{"file":"/var/ww
w/html/apps/user_ldap/lib/Connection.php","line":195,"function":"establishConnection","class":"OCA\\User_LDAP\\Connection","type":"->","args":[]},{"file":"/var/www/html/apps/user_ldap/lib/Connection.php","line":203,"function":"init
","class":"OCA\\User_LDAP\\Connection","type":"->","args":[]},{"file":"/var/www/html/apps/user_ldap/lib/Access.php","line":1267,"function":"getConnectionResource","class":"OCA\\User_LDAP\\Connection","type":"->","args":[]},{"file":
"/var/www/html/apps/user_ldap/lib/Access.php","line":1039,"function":"count","class":"OCA\\User_LDAP\\Access","type":"->","args":["(&(&(objectclass=nextcloudUser)(nextcloudEnabled=1))(displayname=*))",["dc=osit,dc=cc"],["dn"],null,
null]},{"file":"/var/www/html/apps/user_ldap/lib/User_LDAP.php","line":564,"function":"countUsers","class":"OCA\\User_LDAP\\Access","type":"->","args":["(&(&(objectclass=nextcloudUser)(nextcloudEnabled=1))(displayname=*))"]},{"file
":"/var/www/html/apps/user_ldap/lib/User_Proxy.php","line":335,"function":"countUsers","class":"OCA\\User_LDAP\\User_LDAP","type":"->","args":[]},{"file":"/var/www/html/apps/support/lib/Service/SubscriptionService.php","line":128,"
function":"countUsers","class":"OCA\\User_LDAP\\User_Proxy","type":"->","args":[]},{"file":"/var/www/html/apps/support/lib/Service/SubscriptionService.php","line":270,"function":"getUserCount","class":"OCA\\Support\\Service\\Subscr
iptionService","type":"->","args":[]},{"file":"/var/www/html/apps/support/lib/Service/SubscriptionService.php","line":333,"function":"getSubscriptionInfo","class":"OCA\\Support\\Service\\SubscriptionService","type":"->","args":[]},
{"file":"/var/www/html/apps/support/lib/BackgroundJobs/CheckSubscription.php","line":47,"function":"checkSubscription","class":"OCA\\Support\\Service\\SubscriptionService","type":"->","args":[]},{"file":"/var/www/html/lib/private/B
ackgroundJob/Job.php","line":61,"function":"run","class":"OCA\\Support\\BackgroundJobs\\CheckSubscription","type":"->","args":[null]},{"file":"/var/www/html/lib/private/BackgroundJob/TimedJob.php","line":55,"function":"execute","cl
ass":"OC\\BackgroundJob\\Job","type":"->","args":[{"__class__":"OC\\BackgroundJob\\JobList"},{"__class__":"OC\\Log"}]},{"file":"/var/www/html/cron.php","line":125,"function":"execute","class":"OC\\BackgroundJob\\TimedJob","type":"-
>","args":[{"__class__":"OC\\BackgroundJob\\JobList"},{"__class__":"OC\\Log"}]}],"File":"/var/www/html/apps/user_ldap/lib/LDAP.php","Line":375,"CustomMessage":"Error while running background job (class: OCA\\Support\\BackgroundJobs
\\CheckSubscription, arguments: )"},"userAgent":"--","version":"18.0.6.0"}

Apache, Postgres… are running, but in uns monitoring i have seen this:
PostgreSQL DB nextcloud Statistics and PostgreSQL DB postgres Statistics unknown - database not found.
So what is going on, where is are the databases?

And yes have a daily backup of the vm, so i can roll back, but i think after do the update again, i have this error again. Gitlab Enterprise is also on this VM installed from Appcenter. This is working fine.

Best Regards :slight_smile:

Ok, i’ve revert the backup and start the Update again. Same. Nextcloud broken. Update Logs attached. Maybe this helps. update-verbose-log.txt (35,1 KB) update-normal-log.txt (1,8 KB)

I do not know, whether I can really help.
Did you check, whether PostgreSQL was running after update failed? The second is, that some services start with a delay after boot.

Yes the PostgreSQL Server is running. No errors too with “systemctl --failed”. I’ve also restarted the service.

Please check
/var/lib/univention-appcenter/apps/nextcloud/data/nextcloud-data/update.log
whether there is recommended to add indices.

In this case you should do a

univention-app shell nextcloud sudo -u www-data /var/www/html/occ db:add-missing-indices

What does the apache log tell you after doing a “service apache2 restart” in /var/log/apsache2/error_log?

Hello @Mornsgrans, now we’re getting closer. :slight_smile:

The Update.log did not exist on my system. I can remember the command. I did that successfully after an update. So Nextcloudstatus was green.

In the Apache Errorlog i found (after Upate) this infos:

[Sun Jul 05 11:26:40.071869 2020] [mpm_prefork:notice] [pid 1352] AH00169: caught SIGTERM, shutting down
[Sun Jul 05 11:26:42.003306 2020] [mpm_prefork:notice] [pid 14806] AH00163: Apache/2.4.25 (Univention) OpenSSL/1.0.2u configured -- resuming normal operations
[Sun Jul 05 11:26:42.003441 2020] [core:notice] [pid 14806] AH00094: Command line: '/usr/sbin/apache2'

I have also executed only for test the Missing-Indices Kommando. But, really, there are a lot of errors. Maybe it is normal because i have done this in the past.

An unhandled exception has been thrown:
Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occurred in driver: SQLSTATE[08006] [7] FATAL:  Datenbank-Locale ist inkompatibel mit Betriebssystem
DETAIL:  Die Datenbank wurde mit LC_COLLATE »de_AT.UTF-8« initialisiert, was von setlocale() nicht erkannt wird.
HINT:  Erzeugen Sie die Datenbank neu mit einer anderen Locale oder installieren Sie die fehlende Locale. in /var/www/html/lib/private/DB/Connection.php:68
Stack trace:
#0 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(420): OC\DB\Connection->connect()
#1 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(380): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(324): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(683): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /var/www/html/lib/private/DB/Connection.php(155): Doctrine\DBAL\Connection->setTransactionIsolation(2)
#5 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(195): OC\DB\Connection->__construct(Array, Object(Doctrine\DBAL\Driver\PDOPgSql\Driver), Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\EventM
anager))
#6 /var/www/html/lib/private/DB/ConnectionFactory.php(159): Doctrine\DBAL\DriverManager::getConnection(Array, Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\EventManager))
#7 /var/www/html/lib/private/Server.php(749): OC\DB\ConnectionFactory->getConnection('pgsql', Array)
#8 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\Server->OC\{closure}(Object(OC\Server))
#9 /var/www/html/lib/private/ServerContainer.php(125): Pimple\Container->offsetGet('OCP\\IDBConnecti...')
#10 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(162): OC\ServerContainer->query('OCP\\IDBConnecti...')
#11 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server))
#12 /var/www/html/lib/private/ServerContainer.php(125): Pimple\Container->offsetGet('DatabaseConnect...')
#13 /var/www/html/lib/private/Server.php(1666): OC\ServerContainer->query('DatabaseConnect...')
#14 /var/www/html/lib/private/Server.php(422): OC\Server->getDatabaseConnection()
#15 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\Server->OC\{closure}(Object(OC\Server))
#16 /var/www/html/lib/private/ServerContainer.php(125): Pimple\Container->offsetGet('OC\\Authenticati...')
#17 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(70): OC\ServerContainer->query('OC\\Authenticati...', true)
#18 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(100): OC\AppFramework\Utility\SimpleContainer->buildClass(Object(ReflectionClass))
#19 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(115): OC\AppFramework\Utility\SimpleContainer->resolve('OC\\Authenticati...')
#20 /var/www/html/lib/private/ServerContainer.php(150): OC\AppFramework\Utility\SimpleContainer->query('OC\\Authenticati...', true)
#21 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(70): OC\ServerContainer->query('OC\\Authenticati...', true)
#22 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(100): OC\AppFramework\Utility\SimpleContainer->buildClass(Object(ReflectionClass))
#23 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(115): OC\AppFramework\Utility\SimpleContainer->resolve('OC\\Authenticati...')
#24 /var/www/html/lib/private/ServerContainer.php(150): OC\AppFramework\Utility\SimpleContainer->query('OC\\Authenticati...', true)
#25 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(162): OC\ServerContainer->query('OC\\Authenticati...')
#26 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server))
#27 /var/www/html/lib/private/ServerContainer.php(125): Pimple\Container->offsetGet('OC\\Authenticati...')
#28 /var/www/html/lib/private/Server.php(434): OC\ServerContainer->query('OC\\Authenticati...')
#29 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\Server->OC\{closure}(Object(OC\Server))
#30 /var/www/html/lib/private/ServerContainer.php(125): Pimple\Container->offsetGet('OC\\User\\Session')
#31 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(162): OC\ServerContainer->query('OC\\User\\Session')
#32 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server))
#33 /var/www/html/lib/private/ServerContainer.php(125): Pimple\Container->offsetGet('UserSession')
#34 /var/www/html/lib/private/Server.php(1542): OC\ServerContainer->query('UserSession')
#35 /var/www/html/lib/private/Server.php(802): OC\Server->getUserSession()
#36 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\Server->OC\{closure}(Object(OC\Server))
#37 /var/www/html/lib/private/ServerContainer.php(125): Pimple\Container->offsetGet('OC\\App\\AppManag...')
#38 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(162): OC\ServerContainer->query('OC\\App\\AppManag...')
#39 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server))
#40 /var/www/html/lib/private/ServerContainer.php(125): Pimple\Container->offsetGet('AppManager')
#41 /var/www/html/lib/private/Server.php(1837): OC\ServerContainer->query('AppManager')
#42 /var/www/html/lib/private/legacy/app.php(347): OC\Server->getAppManager()
#43 /var/www/html/lib/private/legacy/app.php(114): OC_App::getEnabledApps()
#44 /var/www/html/lib/base.php(645): OC_App::loadApps(Array)
#45 /var/www/html/lib/base.php(1089): OC::init()
#46 /var/www/html/console.php(48): require_once('/var/www/html/l...')
#47 /var/www/html/occ(11): require_once('/var/www/html/c...')

And i can’t connect to postgresql after reboot.

sudo -u postgres psql       
could not change directory to "/root": Keine Berechtigung
psql: FATAL:  Datenbank-Locale ist inkompatibel mit Betriebssystem
DETAIL:  Die Datenbank wurde mit LC_COLLATE »de_AT.UTF-8« initialisiert, was von setlocale() nicht erkannt wird.
TIP:  Erzeugen Sie die Datenbank neu mit einer anderen Locale oder installieren Sie die fehlende Locale.

Normaly it look like this:

sudo -u postgres psql 
could not change directory to "/root": Keine Berechtigung
psql (9.6.17)
Type "help" for help.

postgres=# \l
                                   List of databases
   Name    |   Owner   | Encoding |   Collate   |    Ctype    |    Access privileges    
-----------+-----------+----------+-------------+-------------+-------------------------
 nextcloud | nextcloud | UTF8     | de_AT.UTF-8 | de_AT.UTF-8 | nextcloud=CTc/nextcloud
 postgres  | postgres  | UTF8     | de_AT.UTF-8 | de_AT.UTF-8 | 
 template0 | postgres  | UTF8     | de_AT.UTF-8 | de_AT.UTF-8 | =c/postgres            +
           |           |          |             |             | postgres=CTc/postgres
 template1 | postgres  | UTF8     | de_AT.UTF-8 | de_AT.UTF-8 | =c/postgres            +
           |           |          |             |             | postgres=CTc/postgres
(4 rows)

Ok, i see the problem. On this server all databases are created with de_AT.UTF-8, on all other servers they have de_DE.UTF-8. The locale de_AT.UTF-8 did not exist on the system. I’ve generated the locale. But if i enter “locales”, i did not see the the generated one.

dpkg-reconfigure locales
Generating locales (this might take a while)...
  de_DE.UTF-8... done
  en_US.UTF-8... done
  de_AT.UTF-8... done
Generation complete.

Edit: Solved, after another reboot, after locale generation pgsql is working again.

Great!

It seems, that the errata did change someting with the “locale”

Yes, it looks like. Here is my Bugreport: https://forge.univention.org/bugzilla/show_bug.cgi?id=51614

Is the de_AT.UTF-8 locale configured in UCR in any way? Please check the output of ucr search locale

locale/default: de_DE.UTF-8:UTF-8
Localisation properties such as time/date formats and the preferred translation are specified in locales. The ins
talled locales can be selected using the UMC module 'System settings'. This variable selects the default locale fo
r the system.

locale: en_US.UTF-8:UTF-8 de_DE.UTF-8:UTF-8
Localisation properties such as time/date formats and the preferred translation are specified in locales. The ins
talled locales can be selected using the UMC module 'System settings'. This variable contains the installed locale
s.

Not really.

Okay, therefor UCS will not configure the correct locales once they are (re-)generated during any update. Add your locale to the UCR variable locale and the problem should not reappear.

ucr set locale="en_US.UTF-8:UTF-8 de_DE.UTF-8:UTF-8 de_AT.UTF-8:UTF-8"

1 Like

Very thank for the fast Fix :slight_smile: :slight_smile: :slight_smile: