My vms have been hard hit by IOPS (Causing network errors and other strange behavior) and I wonder what of my pods is causing it (if any) - how can I find out? I couldn't find a clear tool providing this info. I'll be glad for any clues (Cluster running in AKS azure managed kubernetes service)
1
-
I use prometheus operator chart with helm– c4f4t0rCommented Mar 17, 2020 at 15:56
Add a comment
|
2 Answers
With Prometheus and cAdvisor (in GKE included within kubelet metrics), you can use container_fs metrics, and calculate pod IOPS with:
max by (pod, device)(rate(container_fs_reads_total{container!=""}[5m]))
max by (pod, device)(rate(container_fs_writes_total{container!=""}[5m]))
Installing and running iotop on each host could be the easiest way to find out what process is generating the most IO.
