I can't get the cgroup blkio to throttle the iops of processes in a sub-cgroup

Question

I can't get the cgroup blkio to throttle the iops of processes in a sub-cgroup. I am using CentOS 7, 3.10.0-229. I create the cgroup /sys/fs/cgroup/blkio/user.slice/test1 and do

echo 8:32 10 > /sys/fs/cgroup/blkio/user.slice/test1/blkio.throttle.read_iops_device

and likewise for write_iops_device. 8:32 corresponds to HDD at /dev/sdc. I then use stress-ng to read and write to the drive at /srv/hdd-c using

cgexec -g blkio:user.slice/test1 stress-ng --temp-path /srv/hdd-c --hdd 15 -t 180

and

cgexec -g blkio:user.slice/test1 stress-ng --temp-path /srv/hdd-c --readahead 15 -t 180

I confirmed that the 30 processes launched are running in the blkio:user.slice/test1 cgroup using ps eo pid,user,args,cgroup.

When I do iostat 5 the number of tps for sdc is 400 to 600, which is the same as when I run them outside the test1 cgroup. I get the same result if I do echo 8:32 10 > /sys/fs/cgroup/blkio/user.slice/blkio.throttle.{read|write}_iops_device and run them in the user.slice or user.slice/test1 cgroup. However, if I throttle at the root of blkio using echo 8:32 10 > /sys/fs/cgroup/blkio/blkio.throttle.{read|write}_iops_device, and then run stress-ng processes, iostat 5 shows the tps for sdc is 10-11.

Does blkio throttling only work at the root level? The manual pages does not seem to suggest that.

Answer 1

So the issue, which I learned from the answer to this question, is that kernel buffering bypasses the blkio system. Using fio with no buffering and directio I was able to throttle the iops to a device to 10 per second using echo 8:32 10 > /sys/fs/cgroup/blkio/user.slice/blkio.throttle.{read|write}_iops_device

Answer 2

I got the same result as you, but found that if I used

blkio.throttle.{read|write}_bps_device

that will limit throughput when the device is being used as a filesystem.