0

I am running an EC2 m4.xlarge instance with an 500GB Cold HDD EBS volume used as a cache for an image resizing service I am running (thumbor behind nginx reverse proxy).

Every now and then the image service seems to be overloaded and I can't figure out where the bottle neck is. CPU load and memory seem fine, there doesn't appear to be a great deal more traffic at the time. One thing that seems to correlate with the problem is that almost every day at a specific time there is a drop on the VolumeReadBytes and a jump in the VolumeReadOps for like half an hour. I don't have any cron jobs running, the server is dedicated to just serving resized images. I don't know what this could possibly be. Could this be the problem?

Any advice would be greatly appreciated.

Improve this question
1
  • What time of day (UTC)? There are scheduled cronjobs that run every day by default, from /etc/crontab invoking run-parts against pre-defined system tasks... one of which might be the slocate(1) database update, which could potentially generate a lot of small I/O because it's scanning directories.
    – Michael - sqlbot
    Mar 18, 2018 at 14:40

1 Answer 1

Reset to default
0

EBS has quotas on IOPS and throughput. Something is maxing out IOPS, question is what.

There are a couple tools that can find this out. iotop --time --batch is one. iosnoop from Brendan Gregg's perf-tools scripts is another, if you want a ftrace implmentation.

Could even set up auditd to record all the read system calls. But the logging to file overhad would probably use most of your IOPS quota...

However determined, you will need to make the call to either stop that task, use faster storage, or to tolerate poor performance for this half hour.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .