Quick overview of the company product I'm working on before I ask my question:
We have a server running a pretty much stock install of Ubuntu 18.04 and a vision app. The main purpose of the server is to run the vision app. The vision app logs data to a MySQL database (the MySQL install is pretty much stock as well). We also have a website that is essentially a front end for the database. If the mysql systemd service stops running and does not automatically restart then the vision app will crash and not be able to restart until MySQL is started again manually, which is absolutely unacceptable (the vision app has to run 24 hours a day with not more than an occasional few seconds of downtime).
To protect out client's networks, our servers are on isolated local networks that do NOT have internet access.
What has been happening lately is the mysql systemd service has been crashing and not restarting. I made this post:
which resulted in doing the following:
cd /etc/systemd/system sudo mkdir mysql.service.d cd mysql.service.d sudo nano override.conf
Then I entered the following for override.conf:
This did not solve the problem, that is to say MySQL crashes and does not restart after anywhere from 5 to 20 hours of running. I've check the RAM and hard disk usage during the MySQL crash and both are substantially below 75% usage so that does not seem to be the problem either.
Some Googling and browsing
/var/log led me to
/var/log/unattended-upgrades. The most recent crash occurred at about 6:28 am on January 3rd, 2019.
cd /var/log/unattended-upgrades cat unattended-upgrades.log
(some stuff omitted) 2019-01-02 15:43:33,230 INFO Starting unattended upgrades script 2019-01-02 15:43:33,230 INFO Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic 2019-01-03 06:28:00,499 INFO Initial blacklisted packages: 2019-01-03 06:28:00,534 INFO Initial whitelisted packages: 2019-01-03 06:28:00,534 INFO Starting unattended upgrades script 2019-01-03 06:28:00,534 INFO Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic 2019-01-03 06:28:02,248 INFO Removing unused kernel packages: linux-modules-extra-4.15.0-29-generic linux-image-4.15.0-29-generic linux-headers-4.15.0-29 linux-headers-4.15.0-29-generic linux-modules-4.15.0-29-generic 2019-01-03 06:28:36,159 ERROR Auto-removing the packages failed! 2019-01-03 06:28:36,159 ERROR Error message: installArchives() failed 2019-01-03 06:28:36,159 ERROR dpkg returned an error! See /var/log/unattended-upgrades/unattended-upgrades-dpkg.log for details 2019-01-03 06:28:36,561 INFO No packages found that can be upgraded unattended and no pending auto-removals 2019-01-03 07:13:21,059 INFO Initial blacklisted packages: 2019-01-03 07:13:21,137 INFO Initial whitelisted packages: (some stuff omitted)
(some stuff omitted) Log started: 2019-01-03 06:28:02 (Reading database ... 232457 files and directories currently installed.) Removing linux-modules-extra-4.15.0-29-generic (4.15.0-29.31) ... Setting up mysql-server-5.7 (5.7.24-0ubuntu0.18.04.1) ... Checking if update is needed. Checking server version. Error occurred: The mysql.session exists but is not correctly configured. The mysql.session needs SELECT privileges in the performance_schema database and the mysql.db table and also SUPER privileges. mysql_upgrade failed with exit status 5 dpkg: error processing package mysql-server-5.7 (--configure): installed mysql-server-5.7 package post-installation script subprocess returned error exit status 1 Errors were encountered while processing: mysql-server-5.7 Log ended: 2019-01-03 06:28:35
I'm a vision person, not a server/networking expert, so I would not profess to understand these logs entirely, but it seems that
unattended-upgrades is attempting to update MySQL and failing due to not having root permissions, which makes absolutely no sense since
unattended-upgrades would need root permissions to do almost everything it does. Anyhow, after more Googling I found this:
cd /etc/apt/apt.conf.d cat 20auto-upgrades
APT::Periodic::Update-Package-Lists "1"; APT::Periodic::Unattended-Upgrade "1";
So at this point here are my questions:
1) Based on the logs above, does seems likely that
unattended-upgrades is causing the MySQL service to crash? If so, why is this the case?
2) Being that our server does not have internet connectivity, would it be a good general practice to disable automatic updates anyway?
3) Is there anything else I should check that could cause the MySQL service to crash and not auto-restart?
4) Why did adding
Restart=always not make MySQL automatically restart? When I type
sudo systemctl start mysql.service MySQL does successfully restart. Isn't that what the
Restart=always line is supposed to do automatically? Is there another way I can enter this line so it will actually work?
5) Has anybody else experienced
unattended-upgrades causing MySQL to crash? Is this a known problem? I couldn't find anybody else mentioning this upon Googling which seems odd.
6) I'm under the impression that the next step is to disable auto-updates. If this is the case, in
20auto-upgrades should I set
0, or both? I don't see any reason to leave updating package lists on if unattended upgrade is turned off so my inclination is to set both to 0 unless somebody suggests otherwise.