MariaDB won't start after server crash. No error logs

Question

My CentOS 7 server recently crashed by having a hard drive unplugged while it was running and now MariaDB won't start and I'm not getting any error logs.

Everything else on the server seems to be running fine and running xfs_repair (similar to fsck for xfs) on the disk didn't help. Here's specifically what errors I'm getting:

$ sudo systemctl start mariadb
Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.

$ sudo systemctl -l status mariadb
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2018-07-29 19:34:03 MDT; 1min 22s ago
  Process: 21579 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
  Process: 21577 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
  Process: 27150 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=1/FAILURE)
 Main PID: 21577 (code=exited, status=0/SUCCESS)

Jul 29 19:34:03 giivaserver systemd[1]: Starting MariaDB database server...
Jul 29 19:34:03 giivaserver mariadb-prepare-db-dir[27150]: Database MariaDB is not initialized, but the directory /var/lib/mysql is not empty, so initialization cannot be done.
Jul 29 19:34:03 giivaserver mariadb-prepare-db-dir[27150]: Make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.
Jul 29 19:34:03 giivaserver systemd[1]: mariadb.service: control process exited, code=exited status=1
Jul 29 19:34:03 giivaserver systemd[1]: Failed to start MariaDB database server.
Jul 29 19:34:03 giivaserver systemd[1]: Unit mariadb.service entered failed state.
Jul 29 19:34:03 giivaserver systemd[1]: mariadb.service failed.

As far as I can tell, I'm getting no information back about what is happening or why it can't start. There are also no error log files anywhere that I could find.

Here's my my.cnf:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

There are no files at all in my /var/log/mariadb/ or /var/run/mariadb/ folders. I also don't have a host-name.err (giivaserver.err in my case) in either /var/log/ or /var/log/mysql as described on the official article What to Do if MariaDB Doesn't Start.

I tried putting innodb_force_recovery = 1 incrementing from 1 to 5 under [mysqld] in my my.cnf and it didn't affect anything.

I also tried backing up and renaming my /var/lib/mysql folder and rerunning mysql_install_db as some online said deleting it could help, but that made no difference.

I have tried everything I can think of to this point and can't find any feedback from my system as to why MariaDB won't start.

Edit:

The contents of my /var/lib/mysql folder after renaming the original:

drwx------.  6 mysql mysql   122 Jul 29 19:03 .
drwxr-xr-x. 61 root  root   4096 Jul 29 19:03 ..
-rw-rw----.  1 mysql mysql 16384 Jul 29 19:02 aria_log.00000001
-rw-rw----.  1 mysql mysql    52 Jul 29 19:02 aria_log_control
drwx------.  2 mysql mysql  4096 Jul 29 19:02 mysql
drwxr-xr-x.  8 mysql mysql   232 Jul 29 18:58 mysqlBAK
drwx------.  2 mysql mysql  4096 Jul 29 19:02 performance_schema
drwx------.  2 mysql mysql     6 Jul 29 19:02 test

Answer 1

After several days I was able to fix it without any data loss. As I suspected, there was no data loss from the outage. I did many things to try to fix the issue but here is what I believe fixed it:

I renamed my /var/lib/mysql folder to make sure I had a backup, then I deleted my /etc/my.cnf file. I reinstalled mariadb and mariadb-server then started it up. From there I copied my backed-up /var/lib/mysql folder back in and in my /etc/my.cnf added the following lines (gotten from here):

port = 8881
innodb_force_recovery=3
innodb_purge_threads=0

(I think I actually used recovery mode 4 but 3 would have worked for me too)

From there I was able to start up MariaDB with all of my databases. I then ran mysqlcheck --all-databases -p to find that none of my databases had been corrupted. From here, I was unable to get MariaDB to successfully stop, and then I had issues starting it up again, so I repeated the process of deleting the /var/lib/mysql/ folder and running the database installation so that I could copy my backup back in.

From there I started MariaDB with InnoDB forced recovery and once it was running I dumped everything using the command mysqldump -u root -p --all-databases > alldb.sql. I then force stopped MariaDB as it wouldn't stop properly, then rebooted. I uninstalled MariaDB, deleted my /var/lib/mysql/, and renamed my /etc/my.cnf, then reinstalled MariaDB. Once I got it running fresh, I imported my dumped databases using mysql -u root -p < alldb.sql and flushed privileges in the MySQL shell using FLUSH PRIVILEGES; and everything is working great now with no data loss.

As I am running CentOS, I had to do some SELinux stuff along the way. I mostly just followed what the pop-up diagnostics dialogue said to do, then reran the SELinux commands I did originally when installing my Nextcloud server in the first place to get all of the permissions right.

Answer 2

Looks like the database storage has become corrupted. Restore from backup, and put a big sign on your hard drives, "do not unplug whilst running".

Answer 3

In my case moving aside the /var/lib/mysql file to /var/lib/mysql-save and restarting also worked great, as mysql starts up just fine.

Then looking at the two directories I had in the mysql:

ls /var/lib/mysql
total 110628
-rw-rw---- 1 mysql mysql    16384 Dec  5 12:32 aria_log.00000001
-rw-rw---- 1 mysql mysql       52 Dec  5 12:32 aria_log_control
-rw-rw---- 1 mysql mysql      976 Dec  5 12:32 ib_buffer_pool
-rw-rw---- 1 mysql mysql 12582912 Dec  5 12:32 ibdata1
-rw-rw---- 1 mysql mysql 50331648 Dec  5 12:32 ib_logfile0
-rw-rw---- 1 mysql mysql 50331648 Dec  5 12:29 ib_logfile1
-rw-rw---- 1 mysql mysql        0 Dec  5 12:30 multi-master.info
drwx------ 2 mysql mysql     4096 Dec  5 12:29 mysql
drwx------ 2 mysql mysql     4096 Dec  5 12:29 performance_schema
drwx------ 2 mysql mysql     4096 Dec  5 12:29 test

and in my saved directory:

ls -l /var/lib/mysql-saved
-rw-rw---- 1 mysql mysql    16384 Dec  5 12:23 aria_log.00000001
-rw-rw---- 1 mysql mysql       52 Dec  5 12:23 aria_log_control
-rw-r--r-- 1 root  root         0 Dec  4 21:43 debian-10.3.flag
-rw-rw---- 1 mysql mysql     2372 Dec  5 12:23 ib_buffer_pool
-rw-rw---- 1 mysql mysql 79691776 Dec  5 12:23 ibdata1
-rw-rw---- 1 mysql mysql 50331648 Dec  5 12:23 ib_logfile0
-rw-rw---- 1 mysql mysql 50331648 Dec  4 20:23 ib_logfile1
-rw-rw---- 1 mysql mysql        0 Sep  1  2020 multi-master.info
drwx------ 2 mysql mysql     4096 May 11  2021 mysql
drwx------ 2 mysql mysql     4096 Jun 21 09:30 owncloud
drwx------ 2 mysql mysql     4096 May 11  2021 performance_schema
-rw-rw---- 1 mysql mysql        0 Dec  4 18:09 tc.log

Owncloud is my data base, but what is this tc.log file? Turns out it is a transaction log that mysql uses to record transactions in process that may need to be restarted after a crash.

Well, apparently mysql doesn't handle a zero length tc.log file gracefully; it simply exits. So removing that file, and everything starts up just fine.

What version you ask?:

mysql --version
mysql  Ver 15.1 Distrib 10.3.31-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2