I've had a DigitalOcean VM running Jenkins without any problems for the past year-ish. This week suddenly problems started happening.
It started with that one day I noticed some
apt related process (
apt-check I think) had hung at 100% CPU usage. I rebooted the machine and things went back to normal.
However, now the machine has basically become unusable. SSH is slow and I can barely access the Jenkins web interface. So, I started to dig around and see what was the causing the issue.
Checking IO usage
The most notable thing I noticed was that while one of my Jenkins jobs was running, IO utilization was peaking at 100% - constantly.
Ok, that's kind of weird. I went ahead an installed iotop on the machine and had a look at what was using so much IO. Just running
iotop didn't show much. However, when I ran
iotop --accumulated I noticed that
jbd2 was causing high IO wait times, without actually doing much work (pretty much no read/write activity)...
So then I went ahead and ran some tests (after shutting down Jenkins) to measure the actual read/write performance on the machine. Reading was fine, however, writing was strange.
root@jenkins:~# dd if=/dev/zero of=/tmp/output bs=1G count=1 1073741824 bytes (1.1 GB) copied, 271.903 s, 3.9 MB/s
That seems very slow for a DigitalOcean VM using SSD storage. But, just to make sure I ran the same test an a different VM with the same specs (1G 30GB AMS3):
root@gitlab:~# dd if=/dev/zero of=/tmp/output bs=1G count=1 1073741824 bytes (1.1 GB) copied, 36.9623 s, 29.0 MB/s
Clearly something is weird with the jenkins machine. To me it feels like a hardware issue (some very light Googling regarding jbd2 io wait times mentioned RAID issues).
The machine has been idle over night without jenkins running. IO utilization is constantly fluctuating (10-50%) and I'm seeing high CPU usage by apt processes. Not sure if that is relevant.
New VM (from snapshot)
As a final test, to test for hardware problems, I have now also created a new VM from a snapshot of my jenkins machine. At the moment it seems that the new VM is working fine (jbd2 isn't causing high IO wait times - when idle or when a jenkins job is running).
Could this indeed be a hardware issue with the old jenkins VM? I'll contact DigitalOcean as well and see what they say. This all seems a bit worrying to me.
DigitalOcean have confirmed that the host node is experiencing high I/O load. It worries me that (I'm assuming) a different customer can cause my droplet to become unusable. DO have however migrated my droplet to a different host and now things are back to normal.