1

I've had a DigitalOcean VM running Jenkins without any problems for the past year-ish. This week suddenly problems started happening.

It started with that one day I noticed some apt related process (apt-check I think) had hung at 100% CPU usage. I rebooted the machine and things went back to normal.

However, now the machine has basically become unusable. SSH is slow and I can barely access the Jenkins web interface. So, I started to dig around and see what was the causing the issue.

Checking IO usage

The most notable thing I noticed was that while one of my Jenkins jobs was running, IO utilization was peaking at 100% - constantly.

enter image description here

Ok, that's kind of weird. I went ahead an installed iotop on the machine and had a look at what was using so much IO. Just running iotop didn't show much. However, when I ran iotop --accumulated I noticed that jbd2 was causing high IO wait times, without actually doing much work (pretty much no read/write activity)...

enter image description here

Write Tests

So then I went ahead and ran some tests (after shutting down Jenkins) to measure the actual read/write performance on the machine. Reading was fine, however, writing was strange.

root@jenkins:~# dd if=/dev/zero of=/tmp/output bs=1G count=1
1073741824 bytes (1.1 GB) copied, 271.903 s, 3.9 MB/s

That seems very slow for a DigitalOcean VM using SSD storage. But, just to make sure I ran the same test an a different VM with the same specs (1G 30GB AMS3):

root@gitlab:~# dd if=/dev/zero of=/tmp/output bs=1G count=1
1073741824 bytes (1.1 GB) copied, 36.9623 s, 29.0 MB/s

Clearly something is weird with the jenkins machine. To me it feels like a hardware issue (some very light Googling regarding jbd2 io wait times mentioned RAID issues).

Idle Overnight

The machine has been idle over night without jenkins running. IO utilization is constantly fluctuating (10-50%) and I'm seeing high CPU usage by apt processes. Not sure if that is relevant.

enter image description hereenter image description here

New VM (from snapshot)

As a final test, to test for hardware problems, I have now also created a new VM from a snapshot of my jenkins machine. At the moment it seems that the new VM is working fine (jbd2 isn't causing high IO wait times - when idle or when a jenkins job is running).

enter image description here

Conclusion

Could this indeed be a hardware issue with the old jenkins VM? I'll contact DigitalOcean as well and see what they say. This all seems a bit worrying to me.

Update

DigitalOcean have confirmed that the host node is experiencing high I/O load. It worries me that (I'm assuming) a different customer can cause my droplet to become unusable. DO have however migrated my droplet to a different host and now things are back to normal.

  • Adjacency is a problem for other major cloud providers as well. I've seen similar problems in AWS with disk and network IO. Keep nice IO/cpu graphs going for all of your cloud devices to make sure you're getting what you're paying for. – chicks Nov 12 '15 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.