1

I'm running a DigitalOcean droplet with Apache, PHP and MySQL (8.1.6). MySQL restarted unexpectedly this morning, twice in a row, under minimal load.

How can I determine what might have caused this? The previous error logs are all gzipped and empty (is that normal?). I wasn't SSH'd into the server when this happened, so it's not something I did.

Here's the entirety of /var/log/mysql/error.log showing the two shutdowns.

022-07-30T13:24:00.952736Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.29-0ubuntu0.20.04.3).             
2022-07-30T13:24:08.863447Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.29-0ubuntu0.20.04.3)  (Ubuntu).
2022-07-30T13:24:13.726982Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.20.04.2) starting as process 2365053
2022-07-30T13:24:13.753328Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-07-30T13:24:14.472300Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-07-30T13:24:20.421561Z 4 [System] [MY-013381] [Server] Server upgrade from '80029' to '80030' started.
2022-07-30T13:24:32.733700Z 4 [System] [MY-013381] [Server] Server upgrade from '80029' to '80030' completed.
2022-07-30T13:24:32.998287Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-07-30T13:24:32.998440Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-07-30T13:24:33.001764Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/tmp' in the path is accessible to all OS users. Consider choosing a different directory.
2022-07-30T13:24:33.022249Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/run/mysqld/mysqlx.sock
2022-07-30T13:24:33.022457Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.20.04.2'  socket: '/tmp/tmp.vbQx6rn6ge/mysqld.sock'  port: 0  (Ubuntu).
2022-07-30T13:24:33.067042Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.20.04.2).
2022-07-30T13:24:35.232251Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.20.04.2)  (Ubuntu).
2022-07-30T13:24:37.572513Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.20.04.2) starting as process 2365514
2022-07-30T13:24:37.582988Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-07-30T13:24:38.082993Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-07-30T13:24:38.322802Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-07-30T13:24:38.322844Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-07-30T13:24:38.356500Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/run/mysqld/mysqlx.sock
2022-07-30T13:24:38.356615Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.20.04.2'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  (Ubuntu).

Also, the output of mysqltuner.py:

 >>  MySQLTuner 2.0.1
         * Jean-Marie Renouard <[email protected]>
         * Major Hayden <[email protected]>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.pl/
 >>  Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
[OK] Logged in using credentials from Debian maintenance account.
[OK] Currently running supported MySQL version 8.0.30-0ubuntu0.20.04.2
[OK] Operating on 64-bit architecture
 
-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/log/mysql/error.log exists
[--] Log file: /var/log/mysql/error.log (2K)
[OK] Log file /var/log/mysql/error.log is not empty
[OK] Log file /var/log/mysql/error.log is smaller than 32 Mb
[OK] Log file /var/log/mysql/error.log is readable.
[!!] /var/log/mysql/error.log contains 3 warning(s).
[OK] /var/log/mysql/error.log doesn't contain any error.
[--] 4 start(s) detected in /var/log/mysql/error.log
[--] 1) 2022-07-30T13:24:38.356615Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.20.04.2'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  (Ubuntu).
[--] 2) 2022-07-30T13:24:38.356500Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/run/mysqld/mysqlx.sock
[--] 3) 2022-07-30T13:24:33.022457Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.20.04.2'  socket: '/tmp/tmp.vbQx6rn6ge/mysqld.sock'  port: 0  (Ubuntu).
[--] 4) 2022-07-30T13:24:33.022249Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/run/mysqld/mysqlx.sock
[--] 2 shutdown(s) detected in /var/log/mysql/error.log
[--] 1) 2022-07-30T13:24:35.232251Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.20.04.2)  (Ubuntu).
[--] 2) 2022-07-30T13:24:08.863447Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.29-0ubuntu0.20.04.3)  (Ubuntu).
 
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA 
[--] Data in InnoDB tables: 1.5G (Tables: 48)
[--] Data in MyISAM tables: 6.1M (Tables: 6)
[OK] Total fragmented tables: 0
 
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.
 
-------- Views Metrics -----------------------------------------------------------------------------
 
-------- Triggers Metrics --------------------------------------------------------------------------
 
-------- Routines Metrics --------------------------------------------------------------------------
 
-------- Security Recommendations ------------------------------------------------------------------
[--] Skipped due to unsupported feature for MySQL 8
 
-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined
 
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 2h 45m 29s (1M q [103.514 qps], 116K conn, TX: 4G, RX: 214M)
[--] Reads / Writes: 49% / 51%
[--] Binary logging is enabled (GTID MODE: OFF)
[--] Physical Memory     : 7.8G
[--] Max MySQL memory    : 11.8G
[--] Other process memory: 0B
[--] Total buffers: 2.0G global + 65.9M per thread (151 max threads)
[--] P_S Max memory usage: 72B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 2.4G (31.31% of installed RAM)
[!!] Maximum possible memory usage: 11.8G (151.35% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (50/1M)
[OK] Highest usage of available connections: 3% (6/151)
[OK] Aborted connections: 0.00%  (0/116975)
[--] Skipped name resolution test due to skip_networking=ON in system variables.
[--] Query cache have been removed in MySQL 8
[!!] Sorts requiring temporary tables: 20% (597 temp sorts / 2K sorts)
[OK] No joins without indexes
[OK] Temporary tables created on disk: 0% (0 on disk / 162 total)
[OK] Thread cache hit rate: 99% (6 created / 116K connections)
[OK] Table cache hit rate: 99% (952K hits / 953K requests)
[OK] table_definition_cache (2000) is greater than number of tables (394)
[OK] Open file limit used: 0% (28/10K)
[OK] Table locks acquired immediately: 99% (1K immediate / 1K locks)
[OK] Binlog cache memory access: 100.00% (456184 Memory / 456184 Total)
 
-------- Performance schema ------------------------------------------------------------------------
[--] Performance_schema is activated.
[--] Memory used by P_S: 72B
[--] Sys schema is installed.
 
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.
 
-------- MyISAM Metrics ----------------------------------------------------------------------------
[--] MyISAM Metrics are disabled on last MySQL versions.
 
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 2.0G/1.5G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (4.6875%): 48.0M * 2/2.0G should be equal to 25%
[!!] InnoDB buffer pool instances: 8
[--] Number of InnoDB Buffer Pool Chunk: 16 for 8 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 99.95% (135494768 hits/ 135565036 total)
[!!] InnoDB Write Log efficiency: 72.11% (3489529 hits/ 4839323 total)
[OK] InnoDB log waits: 0.00% (0 waits / 1349794 writes)
 
-------- Aria Metrics ------------------------------------------------------------------------------
[--] Aria Storage Engine not available.
 
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
 
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
 
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
 
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: ROW
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server
 
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Check warning line(s) in /var/log/mysql/error.log file
    MySQL was started within the last 24 hours - recommendations may be inaccurate
    Reduce your overall MySQL memory footprint for system stability
    Dedicate this server to your database for highest performance.
    Buffer Key MyISAM set to 0, no MyISAM table detected
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: https://bit.ly/2TcGgtU
Variables to adjust:
  *** MySQL's maximum memory usage is dangerously high ***
  *** Add RAM before increasing MySQL buffer variables ***
    sort_buffer_size (> 256K)
    read_rnd_buffer_size (> 256K)
    key_buffer_size=0
    innodb_log_file_size should be (=256M) if possible, so InnoDB total log files size equals 25% of buffer pool size.
    innodb_buffer_pool_instances(=2)

I know it warns that the max theoretical memory usage is too high, but there's no way it would have gotten to that point -- and my DigitalOcean graphs show no RAM spikes anyway.

Any troubleshooting help would be appreciated!

3
  • Since the two shutdowns occurred within 35 seconds of each other (and I assume you're not sending signals to the process), I'm guessing your startup script is flaky. Jul 30, 2022 at 17:04
  • I'm using whatever the default is -- where would I find that? And why did it restart the first time? (No, I'm not sending signals to the process.)
    – mgiuffrida
    Jul 30, 2022 at 17:19
  • Well, that's pretty much all I know, but use system tools like top (or atop) to see if anything bad happens when you start MySQL. You're probably not running out of system resources, but that would help check. Jul 30, 2022 at 17:24

1 Answer 1

5

Observation in the posted error log,

auto update from 8.0.29 
to
8.0.30

Confirmed by MySQLTuner report of 2h-45m.

Reasonable conclusion?

3
  • I've confirmed that's what happened -- according to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log, myslq-client and mysql-server were automatically upgraded. I guess I need to figure out how to disable that and update them manually during more convenient hours.
    – mgiuffrida
    Jul 31, 2022 at 1:48
  • Found details here, including instructions at the end for changing the time that automatic upgrades happen: wiki.debian.org/UnattendedUpgrades
    – mgiuffrida
    Jul 31, 2022 at 2:02
  • @mgiuffrida Good work on confirming it was the autoupdate. Jan 31 at 13:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .