0

Long story short, I discovered expire_log_days = 3 a little too late. My /var/log directory is completely filled with old mysql-bin files, and per google/SE search, it is strongly recommended not to manually delete them.

The trouble is, I can't get into a MySQL shell to PURGE them, nor am I able to start the service with the new config setting expire_log_days.

I removed some older non-MySQL logs (about 500mb worth), but am still unable to get mysql to start.

Is there a less painful way out of this?

[root@box mariadb]# systemctl restart mariadb.service
Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.
[root@box mariadb]# journalctl -xe
-- Subject: Unit session-203.scope has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-203.scope has begun starting up.
Dec 07 14:01:35 my.sql.box systemd[1]: Starting MariaDB database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mariadb.service has begun starting up.
Dec 07 14:01:36 my.sql.box mysqld_safe[23884]: 161207 14:01:36 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Dec 07 14:01:36 my.sql.box mysqld_safe[23884]: 161207 14:01:36 mysqld_safe Starting mysqld daemon with databases from /data/db/mysql
Dec 07 14:01:36 my.sql.box mysqld_safe[23884]: 161207 14:01:36 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
Dec 07 14:01:37 my.sql.box systemd[1]: mariadb.service: control process exited, code=exited status=1
Dec 07 14:01:37 my.sql.box systemd[1]: Failed to start MariaDB database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mariadb.service has failed.
-- 
-- The result is failed.
Dec 07 14:01:37 my.sql.box systemd[1]: Unit mariadb.service entered failed state.
Dec 07 14:01:37 my.sql.box systemd[1]: mariadb.service failed.
Dec 07 14:01:37 my.sql.box audispd[4973]: node=my.sql.box type=SERVICE_START msg=audit(1481137297.301:1657): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=
Dec 07 14:01:37 my.sql.box polkitd[4981]: Unregistered Authentication Agent for unix-process:23850:300226 (system bus name :1.460, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_U

[root@box log]# systemctl status mariadb.service
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2016-12-07 14:09:17 EST; 7s ago
  Process: 24840 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
  Process: 24838 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
  Process: 24810 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
 Main PID: 24838 (code=exited, status=0/SUCCESS)

Dec 07 14:09:12 my.sql.box systemd[1]: Starting MariaDB database server...
Dec 07 14:09:13 my.sql.box mysqld_safe[24838]: 161207 14:09:13 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Dec 07 14:09:13 my.sql.box mysqld_safe[24838]: 161207 14:09:13 mysqld_safe Starting mysqld daemon with databases from /data/db/mysql
Dec 07 14:09:16 my.sql.box mysqld_safe[24838]: 161207 14:09:16 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
Dec 07 14:09:17 my.sql.box systemd[1]: mariadb.service: control process exited, code=exited status=1
Dec 07 14:09:17 my.sql.box systemd[1]: Failed to start MariaDB database server.
Dec 07 14:09:17 my.sql.box systemd[1]: Unit mariadb.service entered failed state.
Dec 07 14:09:17 my.sql.box systemd[1]: mariadb.service failed.
[root@box log]# 
  • You'll probably end up needing to manually remove the oldest binary logs, until you can get mysql to start. – AKHolland Dec 7 '16 at 19:29
  • 500 MB of free space should be sufficient to start mysql. Can you show the output of df -h and tail -f /var/log/mysqld.log when trying to start mysql ? – shodanshok Dec 7 '16 at 20:13
  • If Linux and extN, you could possibly adjust the reserve value for your filesystem? – Zoredache Dec 7 '16 at 20:32
  • Depending on your distro, you can access mysql with "mysql --defaults-file=/etc/mysql/debian.cnf". If you have a debian based distro – Orphans Dec 8 '16 at 6:52
1

Assuming it is your binary logs that are occupying most of your disk space. All you need to do is move at least one binary log to a separate partition that has extra disk space. Only move do not delete. Then execute PURGE BINARY LOGS TO 'binlog.file'; in mysql console and choose a binary log older than the log file you moved in the previous step. That should free up more disk space. Then move back the binary log file to it's proper location and if needed purge more binary logs.

0

You should disable temporarily those logs by commenting all the expire_log_days or log_bin at the my.cnf file in order to turn off server logging and then manually move/delete the /var/log binaries.

Then start the server, enable again the commented flags and restart MariaDB.

  • Is that entirely safe, being it is a replication client? – a coder Dec 7 '16 at 21:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.