Server shutdown in progress for hours

Question

I have a relatively large database (> 500 GB), and have run a yum update last night, that upgraded mysql-community-server.

Since then, the MySQL server has been in "Server shutdown in progress" status for 16 hours now:

# service mysqld status
Redirecting to /bin/systemctl status mysqld.service
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mysqld.service.d
           └─override.conf
   Active: deactivating (stop-sigterm) since Mon 2020-07-20 01:18:28 CEST; 14h ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
 Main PID: 8318 (mysqld)
   Status: "Server shutdown in progress"
   CGroup: /system.slice/mysqld.service
           └─8318 /usr/sbin/mysqld

Plus, mysqld is now constantly using 300% CPU.

Is there a problem, or is this just a consequence of the database being large? Is there any way I can get some idea of the time left, or see what the server is actually doing?

And is there anything I can do to speed up the process?

For information, the server is a 6-core Xeon with 128GB RAM and NVMe drives, so rather fast.

There is plenty of available disk space and free RAM.

# free -m
              total        used        free      shared  buff/cache   available
Mem:         128889       88292       11189        2225       29407       37145
Swap:         16381        5168       11213

---

Update: it's now been 72 hours. Nothing has changed.

Answer 1

Seems it's known bug and the reason why it happened because you haven't stopped database before the upgrade and some query was running.

You can find more details on Percona article

Try to use command line utility mytop to see what's going on. But you've mentioned you can't reach the database, try under root user. If it's possible make a backup of the database, if you can't in worse case you try to repair the DB using commands in MySQL documentation

In this answer is suggestion it's may related to the time settings on your server.

To obtain information about the stuck process you may use lsof -c mysqld should give you hint.

Instead of killing the mysqld process rather try to restart the server from command systemctl reboot or shutdown -r now, don't pull it out of the plug.