I'm running MySQL on an Ubuntu 10.04 server, with the MySQL $datadir located on an iSCSI volume. The filesystem is tagged with "_netdev" in /etc/fstab. On system shutdown, upstart sends MySQL a SIGTERM, then moves on to shutting down the network interfaces, without waiting for MySQL to exit or for the iSCSI disk to be unmounted or disconnected. This, of course, results in XFS not unmounting cleanly and MySQL needing to spend several minutes replaying InnoDB logs on startup.

I'm not clear on the right place to begin poking to fix this. From what I've read, upstart should already be smart enough to wait for iSCSI filesystems to unmount and disconnect before it shuts down the network interfaces, although this isn't happening. And I have no idea how to tell it that it also needs to wait for a clean shutdown of MySQL.

This is what the console shows during the shutdown process:

Broadcast message from root@mysql-a1
    (unknown) at 11:19 ...

The system is going down for halt NOW!
Power button pressed
 * Stopping Bacula File daemon...                                        [ OK ]
Stopping file integrity checker: samhain.
 * Running nssldap-update-ignoreusers...                                 [ OK ]
 * Stopping multipath daemon multipathd                                  [ OK ]
 * Stopping nagios-nrpe nagios-nrpe                                      [ OK ]
 * Stopping Name Service Cache Daemon nscd                               [ OK ]
 * Stopping Postfix Mail Transport Agent postfix                         [ OK ]
 * Stopping SSH throttling throttle-ssh                                  [ OK ]
 * Stopping puppet agent                                                 [ OK ]
 * Stopping system logging syslog-ng                                     [ OK ]
Stopping statistics collection and monitoring daemon: collectdcollectd[1210]: Exiting normally.
collectd[1210]: collectd: Stopping 15 read threads.
collectdmon[1209]: Info: collectd terminated with exit status 0
collectdmon[1209]: Info: shutting down collectdmon
 * Asking all remaining processes to terminate...                        [ OK ]
 * All processes ended within 1 seconds....                              [ OK ]
 * Deconfiguring network interfaces...                                          [ 2884.248199] end_request: I/O error, dev dm-0, sector 5216
[ 2884.249807] end_request: I/O error, dev dm-0, sector 4192
[ 2884.817855] end_request: I/O error, dev dm-0, sector 0
[ 2884.819347] XFS (dm-0): Device dm-0: metadata write error block 0x0
[ 2884.821281] XFS (dm-0): I/O Error Detected. Shutting down filesystem
[ 2884.823393] XFS (dm-0): Please umount the filesystem and rectify the problem(s)
[ 2885.367423] end_request: I/O error, dev dm-0, sector 0
init: mysql main process (4168) terminated with status 1
                                                                         [ OK ]
 * Deactivating swap...                                                  [ OK ]
 * Unmounting weak filesystems...                                        [ OK ]
 * Unmounting local filesystems...                                       [ OK ]
 * Disconnecting iSCSI targets                                           [ OK ]
 * Stopping iSCSI initiator service                                      [ OK ]
 * Will now halt
[ 2886.802324] Power down.
Connection to bottom.cw closed.
bottom:~ insyte$

This is the fstab entry:

/dev/mapper/21db3d79bf30ef4846c9ce90069680087 /srv/mysql xfs  _netdev,noatime,nodev,noexec  0 0

You have a couple options:

  1. Upgrade to precise. The issue is fixed there.
  2. Change the stop on condition (in /etc/init/mysql.conf) to stop on starting rc RUNLEVEL=[016]
  3. Go into whatever init script (I am pretty sure it is sysv) brings down the network interfaces, and add the command initctl emit deconfiguring-networking just before the interfaces are brought down. Then, change the stop on condition of mysql to stop on deconfiguring-networking or runlevel [016]

The middle option is probably simplest.

  • I've verified that the second option does work, but I don't understand why. Can you please clarify what it's doing?
    – Insyte
    Sep 17 '14 at 16:15
  • Yeah. In a dirty hack, it prevents the sysvinit compatibility job from running, which means that no sysvinit scripts are stopped until mysql stops. Since the scripts that bring down the network and unmount your filesystem are run from sysvinit, you are guaranteed to still have them when mysql is shutting down. Sep 19 '14 at 0:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.