4

I have a relatively large database (> 500 GB), and have run a yum update last night, that upgraded mysql-community-server.

Since then, the MySQL server has been in "Server shutdown in progress" status for 16 hours now:

# service mysqld status
Redirecting to /bin/systemctl status mysqld.service
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mysqld.service.d
           └─override.conf
   Active: deactivating (stop-sigterm) since Mon 2020-07-20 01:18:28 CEST; 14h ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
 Main PID: 8318 (mysqld)
   Status: "Server shutdown in progress"
   CGroup: /system.slice/mysqld.service
           └─8318 /usr/sbin/mysqld

Plus, mysqld is now constantly using 300% CPU.

Is there a problem, or is this just a consequence of the database being large? Is there any way I can get some idea of the time left, or see what the server is actually doing?

And is there anything I can do to speed up the process?

For information, the server is a 6-core Xeon with 128GB RAM and NVMe drives, so rather fast.

There is plenty of available disk space and free RAM.

# free -m
              total        used        free      shared  buff/cache   available
Mem:         128889       88292       11189        2225       29407       37145
Swap:         16381        5168       11213

Update: it's now been 72 hours. Nothing has changed.

12
  • 1
    Could you please add to the question the output of: mysql -e "SHOW ENGINE INNODB STATUS \G" and mysql -e "SHOW FULL PROCESSLIST"|grep -vi sleep Jul 20, 2020 at 14:18
  • 1
    Make sure you have enough disk space: df -h Jul 20, 2020 at 14:19
  • Check server usage: top vmstat -w 5 3 iostat -x 5 3 Jul 20, 2020 at 14:20
  • @MirceaVutcovici The server is not started, so the mysql command fails with "Can't connect to local MySQL server". There is 300 GB free disk space. There is 38 GB RAM available. iostat shows some read/write, but not much (~5 /s).
    – BenMorel
    Jul 20, 2020 at 19:46
  • 2
    I'm assuming at this point you don't want to pull the plug and see what happens... What are the last modification times for your database files? Can you get a stack trace of the process(es) that are still running, using gstack/pstack? What do the running process(es) show if you attach to them with strace -p PID? Jul 22, 2020 at 23:53

1 Answer 1

4
+500

Seems it's known bug and the reason why it happened because you haven't stopped database before the upgrade and some query was running.

You can find more details on Percona article

Try to use command line utility mytop to see what's going on. But you've mentioned you can't reach the database, try under root user. If it's possible make a backup of the database, if you can't in worse case you try to repair the DB using commands in MySQL documentation

In this answer is suggestion it's may related to the time settings on your server.

To obtain information about the stuck process you may use lsof -c mysqld should give you hint.

Instead of killing the mysqld process rather try to restart the server from command systemctl reboot or shutdown -r now, don't pull it out of the plug.

2
  • I was a bit reluctant a first to restart the server, but... it just worked! The MySQL server is now up and running. I ran mysqlcheck and all tables are OK. I'm surprised by your statement, though: "it happened because you haven't stopped database before the upgrade and some query was running". I have a lot of queries running all the time, and never stopped the database server before the upgrade. Shouldn't yum take care of stopping the MySQL server before the upgrade?
    – BenMorel
    Jul 24, 2020 at 20:37
  • Well, when there isn't any activity on DB service you can upgrade it and nothing will happen. But usually you don't want any connections to database during the update/upgrade, as it can kill the query in the middle so the transaction will hang. Correct is to stop the DB server before the upgrade or kill the connections to it, one of the ways is temporarily change the password in web app so it can't connect, or stop the httpd/apache/nginx. Simply when something is writing to it, it can go wrong.Better safe than sorry :) Jul 24, 2020 at 23:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .