1

We are running a LAMP stack on CentOS 7. We have SCL enabled but only PHP and Apache are using it. We were not able to get MariaDB to successfully cutover on a test machine so we never attempted a cutover in production.

We've been experiencing more severe database crashes lately. The crashes and corruptions are a chronic problem but this time it took our wiki offline. We repaired the wiki tables and the MariaDB database and thought we cleared the issue (until the next time). However, I noticed the following in the logfile after a reboot:

InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
InnoDB: Database was not shut down normally! 

The log sequence number in ibdata files does not match is a pretty common problem and it has been discussed ad nauseam. But we should not experience it on a clean shutdown/reboot. (corrections please).

As an experiment I performed a sudo su - followed by shutdown -r now. The logs then revealed:

# tail /var/log/mariadb/mariadb.log
180901 23:01:26 Percona XtraDB (http://www.percona.com) 5.5.59-MariaDB-38.11 started; log sequence number 1478830786
180901 23:01:26 [Note] Plugin 'FEEDBACK' is disabled.
180901 23:01:26 [Note] Server socket created on IP: '0.0.0.0'.
180901 23:01:26 [Note] Event Scheduler: Loaded 0 events
180901 23:01:26 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.60-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
180901 23:01:30 [ERROR] mysqld: Table './my_wiki/wikicryptopp_objectcache' is marked as crashed and should be repaired
180901 23:01:30 [Warning] Checking table:   './my_wiki/wikicryptopp_objectcache'
180901 23:05:01 [ERROR] mysqld: Table './my_wiki/wikicryptopp_module_deps' is marked as crashed and should be repaired
180901 23:05:01 [Warning] Checking table:   './my_wiki/wikicryptopp_module_deps'

It appears systemd is not shutting down the database properly. (corrections please).

I searched the CentOS and Systemd bug trackers but did not find a related issue. I am hoping someone else has experienced the issues and can provide the steps to clear it.

How do I fix the MariaDB shutdown problem?


This is as far as I have gotten when figuring out what is going on during shutdown. I don't know if this is correct (maybe not) because I am not a systemd guy.

[root@ftpit ~]# find /etc -name mariadb.service
/etc/systemd/system/multi-user.target.wants/mariadb.service
[root@ftpit ~]# find /etc -name mysql.service
[root@ftpit ~]# find /opt/rh -name mariadb.service
[root@ftpit ~]# find /opt/rh -name mysql.service

And then:

# cat /etc/systemd/system/multi-user.target.wants/mariadb.service
# It's not recommended to modify this file in-place, because it will be
# overwritten during package upgrades.  If you want to customize, the
# best way is to create a file "/etc/systemd/system/mariadb.service",
# containing
#       .include /lib/systemd/system/mariadb.service
#       ...make your changes here...
# or create a file "/etc/systemd/system/mariadb.service.d/foo.conf",
# which doesn't need to include ".include" call and which will be parsed
# after the file mariadb.service itself is parsed.
#
# For more info about custom unit files, see systemd.unit(5) or
# http://fedoraproject.org/wiki/Systemd#How_do_I_customize_a_unit_file.2F_add_a_custom_unit_file.3F

# For example, if you want to increase mariadb's open-files-limit to 10000,
# you need to increase systemd's LimitNOFILE setting, so create a file named
# "/etc/systemd/system/mariadb.service.d/limits.conf" containing:
#       [Service]
#       LimitNOFILE=10000

# Note: /usr/lib/... is recommended in the .include line though /lib/...
# still works.
# Don't forget to reload systemd daemon after you change unit configuration:
# root> systemctl --system daemon-reload

[Unit]
Description=MariaDB database server
After=syslog.target
After=network.target

[Service]
Type=simple
User=mysql
Group=mysql

ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n
# Note: we set --basedir to prevent probes that might trigger SELinux alarms,
# per bug #547485
ExecStart=/usr/bin/mysqld_safe --basedir=/usr
ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID

# Give a reasonable amount of time for the server to start up/shut down
TimeoutSec=300

# Place temp files in a secure directory, not /tmp
PrivateTmp=true

[Install]
WantedBy=multi-user.target

1 Answer 1

0

"The log sequence number in ibdata files does not match" shouldn't occur commonly. This is unlikely to be a systemd problem. I'd take a full mysqldump and restore this into a clean instance and let that clear up the innodb discrepancies.

note: systemctl show mariadb.service is a better way of showing the systemd config.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy