We are running a LAMP stack on CentOS 7. We have SCL enabled but only PHP and Apache are using it. We were not able to get MariaDB to successfully cutover on a test machine so we never attempted a cutover in production.
We've been experiencing more severe database crashes lately. The crashes and corruptions are a chronic problem but this time it took our wiki offline. We repaired the wiki tables and the MariaDB database and thought we cleared the issue (until the next time). However, I noticed the following in the logfile after a reboot:
InnoDB: The log sequence number in ibdata files does not match InnoDB: the log sequence number in the ib_logfiles! InnoDB: Database was not shut down normally!
The log sequence number in ibdata files does not match is a pretty common problem and it has been discussed ad nauseam. But we should not experience it on a clean shutdown/reboot. (corrections please).
As an experiment I performed a
sudo su - followed by
shutdown -r now. The logs then revealed:
# tail /var/log/mariadb/mariadb.log 180901 23:01:26 Percona XtraDB (http://www.percona.com) 5.5.59-MariaDB-38.11 started; log sequence number 1478830786 180901 23:01:26 [Note] Plugin 'FEEDBACK' is disabled. 180901 23:01:26 [Note] Server socket created on IP: '0.0.0.0'. 180901 23:01:26 [Note] Event Scheduler: Loaded 0 events 180901 23:01:26 [Note] /usr/libexec/mysqld: ready for connections. Version: '5.5.60-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server 180901 23:01:30 [ERROR] mysqld: Table './my_wiki/wikicryptopp_objectcache' is marked as crashed and should be repaired 180901 23:01:30 [Warning] Checking table: './my_wiki/wikicryptopp_objectcache' 180901 23:05:01 [ERROR] mysqld: Table './my_wiki/wikicryptopp_module_deps' is marked as crashed and should be repaired 180901 23:05:01 [Warning] Checking table: './my_wiki/wikicryptopp_module_deps'
It appears systemd is not shutting down the database properly. (corrections please).
I searched the CentOS and Systemd bug trackers but did not find a related issue. I am hoping someone else has experienced the issues and can provide the steps to clear it.
How do I fix the MariaDB shutdown problem?
This is as far as I have gotten when figuring out what is going on during shutdown. I don't know if this is correct (maybe not) because I am not a systemd guy.
[root@ftpit ~]# find /etc -name mariadb.service /etc/systemd/system/multi-user.target.wants/mariadb.service [root@ftpit ~]# find /etc -name mysql.service [root@ftpit ~]# find /opt/rh -name mariadb.service [root@ftpit ~]# find /opt/rh -name mysql.service
# cat /etc/systemd/system/multi-user.target.wants/mariadb.service # It's not recommended to modify this file in-place, because it will be # overwritten during package upgrades. If you want to customize, the # best way is to create a file "/etc/systemd/system/mariadb.service", # containing # .include /lib/systemd/system/mariadb.service # ...make your changes here... # or create a file "/etc/systemd/system/mariadb.service.d/foo.conf", # which doesn't need to include ".include" call and which will be parsed # after the file mariadb.service itself is parsed. # # For more info about custom unit files, see systemd.unit(5) or # http://fedoraproject.org/wiki/Systemd#How_do_I_customize_a_unit_file.2F_add_a_custom_unit_file.3F # For example, if you want to increase mariadb's open-files-limit to 10000, # you need to increase systemd's LimitNOFILE setting, so create a file named # "/etc/systemd/system/mariadb.service.d/limits.conf" containing: # [Service] # LimitNOFILE=10000 # Note: /usr/lib/... is recommended in the .include line though /lib/... # still works. # Don't forget to reload systemd daemon after you change unit configuration: # root> systemctl --system daemon-reload [Unit] Description=MariaDB database server After=syslog.target After=network.target [Service] Type=simple User=mysql Group=mysql ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n # Note: we set --basedir to prevent probes that might trigger SELinux alarms, # per bug #547485 ExecStart=/usr/bin/mysqld_safe --basedir=/usr ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID # Give a reasonable amount of time for the server to start up/shut down TimeoutSec=300 # Place temp files in a secure directory, not /tmp PrivateTmp=true [Install] WantedBy=multi-user.target