Kopano MySQL startup failure

[goudduif]

In My ESXi server there was a backup disk, available, this one died last night
Everything on this Host was so slow, that I can’t read my mail most off the time.
I was able to initiate a univention server shutdown.

Removed the backup Disk, and restart the server.
But now the MariaDB failed to start

root@nlaio01:~# service mysql start
Job for mariadb.service failed because a fatal signal was delivered to the control process.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
root@nlaio01:~# systemctl status mariadb.service
● mariadb.service - MariaDB database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: signal) since Thu 2018-09-13 23:07:17 CEST; 4s ago
  Process: 16964 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=killed, signal=ABRT)
  Process: 16848 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, statu
  Process: 16843 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 16842 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
 Main PID: 16964 (code=killed, signal=ABRT)
   Status: "InnoDB: Fatal: Trying to access page number 3757731199 in space 0 space name ./ibdata1, which is outside the tablespace bounds. Byte offset 0, len 16384 i/o type 10.Please check that the configuration matches
    Tasks: 0 (limit: 4915)
   Memory: 820.0K
      CPU: 320ms
   CGroup: /system.slice/mariadb.service

Sep 13 23:07:17 nlaio01 systemd[1]: Failed to start MariaDB database server.
Sep 13 23:07:17 nlaio01 systemd[1]: mariadb.service: Unit entered failed state.
Sep 13 23:07:17 nlaio01 systemd[1]: mariadb.service: Failed with result 'signal'.
Sep 13 23:07:22 nlaio01 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
Sep 13 23:07:22 nlaio01 systemd[1]: Stopped MariaDB database server.
Sep 13 23:07:22 nlaio01 systemd[1]: Starting MariaDB database server...

root@nlaio01:~# journalctl -xe
Sep 13 23:07:39 nlaio01 PAM-univentionsambadomain[17426]: continuing as user mypc$
Sep 13 23:07:39 nlaio01 smbd[17426]: pam_unix(samba:session): session opened for user mypc$ by (uid=0)
Sep 13 23:07:39 nlaio01 kopano-search[581]: [error  ] gsoap connect: ()
Sep 13 23:07:39 nlaio01 kopano-search[581]: [error  ] HrLogon server "default:" user "SYSTEM": network error
Sep 13 23:07:39 nlaio01 kopano-search[581]: 2018-09-13 23:07:39,672 - search - WARNING - could not connect to server at 'default:', retrying in 5 sec
Sep 13 23:07:40 nlaio01 kopano-spooler[594]: Thu Sep 13 23:07:40 2018: [error  ] [  594] gsoap connect: ()
Sep 13 23:07:40 nlaio01 kopano-spooler[594]: Thu Sep 13 23:07:40 2018: [error  ] [  594] HrLogon server "default:" user "SYSTEM": network error
Sep 13 23:07:40 nlaio01 kopano-spooler[594]: Thu Sep 13 23:07:40 2018: [error  ] [  594] Unable to open admin session: network error (80040115)
Sep 13 23:07:40 nlaio01 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
Sep 13 23:07:40 nlaio01 systemd[1]: Stopped MariaDB database server.
-- Subject: Unit mariadb.service has finished shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has finished shutting down.
Sep 13 23:07:40 nlaio01 systemd[1]: Starting MariaDB database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has begun starting up.
Sep 13 23:07:41 nlaio01 mysqld[17553]: 2018-09-13 23:07:41 139672489804352 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB-0+deb9u1) starting as process 17553 ...
Sep 13 23:07:41 nlaio01 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
Sep 13 23:07:41 nlaio01 systemd[1]: Failed to start MariaDB database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Sep 13 23:07:41 nlaio01 systemd[1]: mariadb.service: Unit entered failed state.
Sep 13 23:07:41 nlaio01 systemd[1]: mariadb.service: Failed with result 'signal'.
Sep 13 23:07:43 nlaio01 kopano-spooler[594]: Thu Sep 13 23:07:43 2018: [error  ] [  594] gsoap connect: ()
Sep 13 23:07:43 nlaio01 kopano-spooler[594]: Thu Sep 13 23:07:43 2018: [error  ] [  594] HrLogon server "default:" user "SYSTEM": network error
Sep 13 23:07:43 nlaio01 kopano-spooler[594]: Thu Sep 13 23:07:43 2018: [error  ] [  594] Unable to open admin session: network error (80040115)
Sep 13 23:07:43 nlaio01 check_nrpe[17575]: Remote 192.168.0.5 accepted a Version 3 Packet
Sep 13 23:07:44 nlaio01 kopano-search[581]: [error  ] gsoap connect: ()
Sep 13 23:07:44 nlaio01 kopano-search[581]: [error  ] HrLogon server "default:" user "SYSTEM": network error
Sep 13 23:07:44 nlaio01 kopano-search[581]: 2018-09-13 23:07:44,678 - search - WARNING - could not connect to server at 'default:', retrying in 5 sec
Sep 13 23:07:46 nlaio01 kopano-spooler[594]: Thu Sep 13 23:07:46 2018: [error  ] [  594] gsoap connect: ()
Sep 13 23:07:46 nlaio01 kopano-spooler[594]: Thu Sep 13 23:07:46 2018: [error  ] [  594] HrLogon server "default:" user "SYSTEM": network error
Sep 13 23:07:46 nlaio01 kopano-spooler[594]: Thu Sep 13 23:07:46 2018: [error  ] [  594] Unable to open admin session: network error (80040115)
Sep 13 23:07:46 nlaio01 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
Sep 13 23:07:46 nlaio01 systemd[1]: Stopped MariaDB database server.
-- Subject: Unit mariadb.service has finished shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has finished shutting down.
Sep 13 23:07:46 nlaio01 systemd[1]: Starting MariaDB database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has begun starting up.
Sep 13 23:07:47 nlaio01 postfix/smtpd[3005]: connect from 220-134-195-68.HINET-IP.hinet.net[220.134.195.68]
Sep 13 23:07:47 nlaio01 mysqld[17711]: 2018-09-13 23:07:47 140407022449216 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB-0+deb9u1) starting as process 17711 ...

Even When I restore a snapshot from yesterday evening, I receive still the same error.
Hopefully one off you can provide me some info, how I can solve this issue.
When I try this:

How to recover MySQL/MariaDB after InnoDB storage corruption

I receive the same error message as soon I try to start the service.

[fbartels]

When I try this:

That may not be what you want to hear, but if you still do not get MySQL up with the information from the link and therefore are not able to get a mysqldump then there is not much hope for your data.

I’d recommend to go through old backups to see which one of these you can still restore.

[Moritz_Bunkus]

What Felix said.

In addition to that a couple of "do"s and "don’t"s:

1. Snapshots aren’t backups.
2. Snapshots of running databases are worthless unless the database hat committed everything to disk and was in read-only mode (or wasn’t running at all) prior to the snapshot being taken. If you’re using snapshots as a fallback when implementing important changes (e.g. running updates), take them when machines are turned off. This isn’t important only for databases, but also for Samba (AD DCs use bi-directional synchronisation and must be snapshotted in consistent state, hence: when they’re turned off).
3. Backups of database files (the stuff in /var/lib/mysql) are worthless unless the database wasn’t running when the backup was made. Use mysqldump … --single-transaction … for creating textual representations of the current database structure & content. You can back up those files just fine.
4. Use specialized tools for backup if available. For Kopano there’s kopano-backup which dumps mailboxes and public folders into local files and which can run incremental backups very well and fast. You can then back up the files created by kopano-backup with your regular backup mechanism.
5. Create backups regularly. At least daily. For really important stuff think about ways to keep even more copies than just the daily ones (e.g. for mail you might forward all incoming and outgoing mail to a secondary server where it all ends up in a single mailbox and where all mail older than e.g. seven days is deleted automatically; or use the kopano-backup mechanism hourly; or think of something else).
6. Store some backups offsite, if possible, for disaster recovery purpose. For speed purposes you can keep two sets/run two backups: a daily one on-site and one off-site daily or less frequently.

m.

[fbartels]

Just a small addition to the above

Use specialized tools for backup if available.

While I do agree that kopano-backup should be used, having a mysql dump (as well as a backup of the attachments) never hurts.

[goudduif]

That may not be what you want to hear, but if you still do not get MySQL up with the information from the link and therefore are not able to get a mysqldump then there is not much hope for your data.

I’m not sure if I do the right steps, when I add

[mysql]
port = 3306
innodb_force_recovery=3
innodb_purge_threads=0

in /etc/mysql/my.cnf,

But when I start the mysql service, I receive exact the same error as when I start them normally.
So my question is, did I apply the right settings to start mysql in the debugging mode?

Use specialized tools for backup if available.

This is the first task as soon everything is back available!

[fbartels]

So my question is, did I apply the right settings to start mysql in the debugging mode?

Yes, I would think so.

https://stackoverflow.com/questions/38245974/mysql-server-crashes-innodb-outside-the-tablespace-bounds also mentions that you should increase innodb_force_recovery until you are able to get a dump. But be aware that the higher value here, the bigger the chance that your dump will not be complete.