0

Quick overview of the company product I'm working on before I ask my question:

We have a server running a pretty much stock install of Ubuntu 18.04 and a vision app. The main purpose of the server is to run the vision app. The vision app logs data to a MySQL database (the MySQL install is pretty much stock as well). We also have a website that is essentially a front end for the database. If the mysql systemd service stops running and does not automatically restart then the vision app will crash and not be able to restart until MySQL is started again manually, which is absolutely unacceptable (the vision app has to run 24 hours a day with not more than an occasional few seconds of downtime).

To protect out client's networks, our servers are on isolated local networks that do NOT have internet access.

What has been happening lately is the mysql systemd service has been crashing and not restarting. I made this post:

mysql.service location on Ubuntu 18.04 server?

which resulted in doing the following:

cd /etc/systemd/system
sudo mkdir mysql.service.d
cd mysql.service.d
sudo nano override.conf

Then I entered the following for override.conf:

[Service]
Restart=always

This did not solve the problem, that is to say MySQL crashes and does not restart after anywhere from 5 to 20 hours of running. I've check the RAM and hard disk usage during the MySQL crash and both are substantially below 75% usage so that does not seem to be the problem either.

Some Googling and browsing /var/log led me to /var/log/unattended-upgrades. The most recent crash occurred at about 6:28 am on January 3rd, 2019.

cd /var/log/unattended-upgrades
cat unattended-upgrades.log

output:

(some stuff omitted)
2019-01-02 15:43:33,230 INFO Starting unattended upgrades script
2019-01-02 15:43:33,230 INFO Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
2019-01-03 06:28:00,499 INFO Initial blacklisted packages: 
2019-01-03 06:28:00,534 INFO Initial whitelisted packages: 
2019-01-03 06:28:00,534 INFO Starting unattended upgrades script
2019-01-03 06:28:00,534 INFO Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
2019-01-03 06:28:02,248 INFO Removing unused kernel packages: linux-modules-extra-4.15.0-29-generic linux-image-4.15.0-29-generic linux-headers-4.15.0-29 linux-headers-4.15.0-29-generic linux-modules-4.15.0-29-generic
2019-01-03 06:28:36,159 ERROR Auto-removing the packages failed!
2019-01-03 06:28:36,159 ERROR Error message: installArchives() failed
2019-01-03 06:28:36,159 ERROR dpkg returned an error! See /var/log/unattended-upgrades/unattended-upgrades-dpkg.log for details
2019-01-03 06:28:36,561 INFO No packages found that can be upgraded unattended and no pending auto-removals
2019-01-03 07:13:21,059 INFO Initial blacklisted packages: 
2019-01-03 07:13:21,137 INFO Initial whitelisted packages: 
(some stuff omitted)

next command:

cat unattended-upgrades-shutdown.log

output:

(some stuff omitted)
Log started: 2019-01-03  06:28:02
(Reading database ... 232457 files and directories currently installed.)
Removing linux-modules-extra-4.15.0-29-generic (4.15.0-29.31) ...
Setting up mysql-server-5.7 (5.7.24-0ubuntu0.18.04.1) ...
Checking if update is needed.
Checking server version.
Error occurred: The mysql.session exists but is not correctly configured. The mysql.session needs SELECT privileges in the performance_schema database and the mysql.db table and also SUPER privileges.
mysql_upgrade failed with exit status 5
dpkg: error processing package mysql-server-5.7 (--configure):
 installed mysql-server-5.7 package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 mysql-server-5.7
Log ended: 2019-01-03  06:28:35

I'm a vision person, not a server/networking expert, so I would not profess to understand these logs entirely, but it seems that unattended-upgrades is attempting to update MySQL and failing due to not having root permissions, which makes absolutely no sense since unattended-upgrades would need root permissions to do almost everything it does. Anyhow, after more Googling I found this:

cd /etc/apt/apt.conf.d
cat 20auto-upgrades

output:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

So at this point here are my questions:

1) Based on the logs above, does seems likely that unattended-upgrades is causing the MySQL service to crash? If so, why is this the case?

2) Being that our server does not have internet connectivity, would it be a good general practice to disable automatic updates anyway?

3) Is there anything else I should check that could cause the MySQL service to crash and not auto-restart?

4) Why did adding override.conf with Restart=always not make MySQL automatically restart? When I type sudo systemctl start mysql.service MySQL does successfully restart. Isn't that what the Restart=always line is supposed to do automatically? Is there another way I can enter this line so it will actually work?

5) Has anybody else experienced unattended-upgrades causing MySQL to crash? Is this a known problem? I couldn't find anybody else mentioning this upon Googling which seems odd.

6) I'm under the impression that the next step is to disable auto-updates. If this is the case, in 20auto-upgrades should I set Update-Package-Lists to 0, Unattended-Upgrade to 0, or both? I don't see any reason to leave updating package lists on if unattended upgrade is turned off so my inclination is to set both to 0 unless somebody suggests otherwise.

  • As a DBA, I consider it unacceptable to allow package management to even install -- much less upgrade -- production MySQL/MariaDB. But, to address your question, it sounds like you may have an issue with an entry in the mysql.user table. See askubuntu.com/q/1069377/94176. – Michael - sqlbot Jan 4 at 0:58
  • Thanks for the suggestion Michael. So far I have set Update-Package-Lists and Unattended-Upgrade both to "0" in 20auto-upgrades and that seems to have resolved the concern. – cdahms Jan 6 at 2:07
0

After unattended upgrade of mysql server I had no option but remove and reinstall mysql-server and restore databases.

The samples of log files:

2019-01-24 06:19:16,643 INFO Initial whitelisted packages:
2019-01-24 06:19:16,644 INFO Starting unattended upgrades script
2019-01-24 06:19:16,644 INFO Allowed origins are: o=Ubuntu,a=cosmic, o=Ubuntu,a=cosmic-security, o=UbuntuESM,a=cosmic
2019-01-24 06:19:22,702 INFO Packages that will be upgraded: apt apt-utils libapt-inst2.0 libapt-pkg5.0 mysql-client-5.7 mysql-client-core-5.7 mysql-server mysql-server-5.7 mysql-server-core-5.7
2019-01-24 06:19:22,710 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2019-01-24 06:22:03,228 ERROR Installing the upgrades failed!
2019-01-24 06:22:03,242 ERROR error message: installArchives() failed
2019-01-24 06:22:03,256 ERROR dpkg returned a error! See /var/log/unattended-upgrades/unattended-upgrades-dpkg.log for details
Log started: 2019-01-24  06:21:05
Preconfiguring packages ...
(Reading database ... 91129 files and directories currently installed.)
Preparing to unpack .../mysql-server-5.7_5.7.25-0ubuntu0.18.10.2_amd64.deb ...
Unpacking mysql-server-5.7 (5.7.25-0ubuntu0.18.10.2) over (5.7.24-0ubuntu0.18.10.1) ...
Preparing to unpack .../mysql-server-core-5.7_5.7.25-0ubuntu0.18.10.2_amd64.deb ...
Unpacking mysql-server-core-5.7 (5.7.25-0ubuntu0.18.10.2) over (5.7.24-0ubuntu0.18.10.1) ...
Setting up mysql-server-core-5.7 (5.7.25-0ubuntu0.18.10.2) ...
Processing triggers for systemd (239-7ubuntu10.6) ...
Processing triggers for man-db (2.8.4-2) ...
Setting up mysql-server-5.7 (5.7.25-0ubuntu0.18.10.2) ...
Checking if update is needed.
Checking server version.
Running queries to upgrade MySQL server.
mysql_upgrade: (non fatal) [ERROR] 1728: Cannot load from mysql.proc. The table is probably corrupted
mysql_upgrade: (non fatal) [ERROR] 1545: Failed to open mysql.event
mysql_upgrade: [ERROR] 1136: Column count doesn't match value count at row 1
mysql_upgrade failed with exit status 5
dpkg: error processing package mysql-server-5.7 (--configure):
 installed mysql-server-5.7 package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 mysql-server-5.7
Log ended: 2019-01-24  06:22:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.