I have been having some problems with crashes on my KVM host (Lubuntu 20.04), and when troubleshooting, I noticed some time-related errors.
Upon further investigation, to my horror, I saw that time was not being synced. I am sure it was set up before, I have no clue how it became un-setup.
admin@virtland:~$ sudo timedatectl
[sudo] password for admin:
Local time: Fri 2020-07-10 09:14:14 EDT
Universal time: Fri 2020-07-10 13:14:14 UTC
RTC time: Fri 2020-07-10 13:14:14
Time zone: America/New_York (EDT, -0400)
System clock synchronized: no
NTP service: n/a
RTC in local TZ: no
admin@virtland:~$
I found this thread and tried the top answer, but no no avail. https://askubuntu.com/questions/929805/timedatectl-ntp-sync-cannot-set-to-yes
admin@virtland:~$ sudo systemctl stop ntp
admin@virtland:~$ sudo ntpd -gq
10 Jul 09:17:57 ntpd[34358]: ntpd [email protected] (1): Starting
10 Jul 09:17:57 ntpd[34358]: Command line: ntpd -gq
10 Jul 09:17:57 ntpd[34358]: proto: precision = 0.070 usec (-24)
10 Jul 09:17:57 ntpd[34358]: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): good hash signature
10 Jul 09:17:57 ntpd[34358]: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): loaded, expire=2020-12-28T00:00:00Z last=2017-01-01T00:00:00Z ofs=37
10 Jul 09:17:57 ntpd[34358]: Listen and drop on 0 v6wildcard [::]:123
10 Jul 09:17:57 ntpd[34358]: Listen and drop on 1 v4wildcard 0.0.0.0:123
10 Jul 09:17:57 ntpd[34358]: Listen normally on 2 lo 127.0.0.1:123
10 Jul 09:17:57 ntpd[34358]: Listen normally on 3 enp6s0 10.0.0.18:123
10 Jul 09:17:57 ntpd[34358]: Listen normally on 4 lo [::1]:123
10 Jul 09:17:57 ntpd[34358]: Listen normally on 5 enp6s0 [fe80::7285:c2ff:fe65:9f19%3]:123
10 Jul 09:17:57 ntpd[34358]: Listening on routing socket on fd #22 for interface updates
10 Jul 09:17:58 ntpd[34358]: Soliciting pool server 209.50.63.74
10 Jul 09:17:59 ntpd[34358]: Soliciting pool server 4.53.160.75
10 Jul 09:18:00 ntpd[34358]: Soliciting pool server 69.89.207.199
10 Jul 09:18:00 ntpd[34358]: Soliciting pool server 72.30.35.88
10 Jul 09:18:01 ntpd[34358]: Soliciting pool server 173.0.48.220
10 Jul 09:18:01 ntpd[34358]: Soliciting pool server 162.159.200.1
10 Jul 09:18:01 ntpd[34358]: Soliciting pool server 108.61.73.243
10 Jul 09:18:02 ntpd[34358]: Soliciting pool server 208.79.89.249
10 Jul 09:18:02 ntpd[34358]: Soliciting pool server 208.67.75.242
10 Jul 09:18:02 ntpd[34358]: Soliciting pool server 91.189.94.4
10 Jul 09:18:03 ntpd[34358]: Soliciting pool server 91.189.89.198
10 Jul 09:18:03 ntpd[34358]: Soliciting pool server 67.217.112.181
10 Jul 09:18:04 ntpd[34358]: Soliciting pool server 91.189.89.199
10 Jul 09:18:04 ntpd[34358]: Soliciting pool server 64.225.34.103
10 Jul 09:18:05 ntpd[34358]: Soliciting pool server 91.189.91.157
10 Jul 09:18:06 ntpd[34358]: Soliciting pool server 2001:67c:1560:8003::c8
10 Jul 09:18:06 ntpd[34358]: ntpd: time slew +0.001834 s
ntpd: time slew +0.001834s
admin@virtland:~$ sudo service ntp start
admin@virtland:~$ sudo timedatectl
Local time: Fri 2020-07-10 09:18:21 EDT
Universal time: Fri 2020-07-10 13:18:21 UTC
RTC time: Fri 2020-07-10 13:18:21
Time zone: America/New_York (EDT, -0400)
System clock synchronized: no
NTP service: n/a
RTC in local TZ: no
admin@virtland:~$
I thought maybe I needed to use some more up-to-date instructions, so I tried this: https://linuxconfig.org/how-to-sync-time-on-ubuntu-20-04-focal-fossa-linux
admin@virtland:~$ timedatectl set-ntp off
Failed to set ntp: NTP not supported
admin@virtland:~$ timedatectl set-ntp on
Failed to set ntp: NTP not supported
Then I tried this, from a different thread:
admin@virtland:~$ sudo systemctl status systemd-timesyncd.service
[sudo] password for admin:
● systemd-timesyncd.service
Loaded: masked (Reason: Unit systemd-timesyncd.service is masked.)
Active: inactive (dead)
I have never touched timesyncd.conf, but it is entirely commented out anyway:
cat /etc/systemd/timesyncd.conf
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.
[Time]
#NTP=
#FallbackNTP=ntp.ubuntu.com
#RootDistanceMaxSec=5
#PollIntervalMinSec=32
#PollIntervalMaxSec=2048
I checked timedatectl again, and now it is on, but still not using NTP. I understand that NTP is more precise, and that can be important in some situations. Not sure if virtualization with pci passthrough needs extremely precise time or not.
From other stuff I was reading, I thought maybe NTP was conflicting with timesyncd. So remove ntp for the time being:
sudo systemctl stop ntp
sudo apt-get purge ntp
But after purging ntp, NTP showed as active!
admin@virtland:~$ timedatectl
Local time: Fri 2020-07-10 09:34:52 EDT
Universal time: Fri 2020-07-10 13:34:52 UTC
RTC time: Fri 2020-07-10 13:34:52
Time zone: America/New_York (EDT, -0400)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
Am I going crazy? Is NTP still here somehow? Nope.
admin@virtland:~$ sudo systemctl start ntp
Failed to start ntp.service: Unit ntp.service not found.
Apologies for not asking a more focused question, but what the heck is going on here?
I am well and truly lost. Also, I will edit this post later and make a not as to whether removing NTP (and thus activating it?!) fixed the stability problems that led me down this rabbit hole.
Edit: The next thing I did was disable ntp on timesyncd and (re)install NTP as described here. https://www.digitalocean.com/community/tutorials/how-to-set-up-time-synchronization-on-ubuntu-18-04
That resulted in:
admin@virtland:~$ ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
0.us.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
1.us.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
2.us.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
3.us.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
0.ubuntu.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.000
1.ubuntu.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.000
2.ubuntu.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.000
3.ubuntu.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.000
ntp.ubuntu.com .POOL. 16 p - 64 0 0.000 0.000 0.000
admin@virtland:~$ timedatectl
Local time: Fri 2020-07-10 10:35:39 EDT
Universal time: Fri 2020-07-10 14:35:39 UTC
RTC time: Fri 2020-07-10 14:35:40
Time zone: America/New_York (EDT, -0400)
System clock synchronized: no
NTP service: n/a
RTC in local TZ: no
admin@virtland:~$ systemctl status systemd-timesyncd
● systemd-timesyncd.service
Loaded: masked (Reason: Unit systemd-timesyncd.service is masked.)
Active: inactive (dead)
admin@virtland:~$ nano /etc/ntp.conf
admin@virtland:~$ systemctl status systemd-timesyncd
● systemd-timesyncd.service
Loaded: masked (Reason: Unit systemd-timesyncd.service is masked.)
Active: inactive (dead)
admin@virtland:~$ ntpstat
unsynchronised
polling server every 8 s
I reversed those changes as recommended my Michael Hampton: Does this mean it's working?
boss@virtland:~$ sudo systemctl stop ntp
Failed to stop ntp.service: Unit ntp.service not loaded.
boss@virtland:~$ sudo timedatectl set-ntp yes
boss@virtland:~$ sudo timedatectl set-ntp on
boss@virtland:~$ ntpq -p
bash: /usr/bin/ntpq: No such file or directory
boss@virtland:~$ systemctl status systemd-timesyncd
● systemd-timesyncd.service - Network Time Synchronization
Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; >
Active: active (running) since Fri 2020-07-10 10:49:18 EDT; 50s ago
Docs: man:systemd-timesyncd.service(8)
Main PID: 108365 (systemd-timesyn)
Status: "Initial synchronization to time server 91.189.94.4:123 (ntp.ubu>
Tasks: 2 (limit: 154317)
Memory: 1.8M
CGroup: /system.slice/systemd-timesyncd.service
└─108365 /lib/systemd/systemd-timesyncd
Jul 10 10:49:17 virtland systemd[1]: Starting Network Time Synchronization...
Jul 10 10:49:18 virtland systemd[1]: Started Network Time Synchronization.
Jul 10 10:49:18 virtland systemd-timesyncd[108365]: Initial synchronization t>
lines 1-14/14 (END)
timedatectl
Local time: Fri 2020-07-10 10:52:56 EDT
Universal time: Fri 2020-07-10 14:52:56 UTC
RTC time: Fri 2020-07-10 14:52:56
Time zone: America/New_York (EDT, -0400)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
So I guess it is working. Since the crashes that took me down this path are still happening, I guess the time wasn't the issue.