130 reputation
17
bio website
location Seattle, WA
age 54
visits member for 2 years, 7 months
seen Sep 10 at 18:56

Aug
21
awarded  Notable Question
Dec
30
awarded  Popular Question
Feb
28
awarded  Critic
Feb
27
awarded  Teacher
Feb
27
comment High Load Average with modest CPU Utilization and almost no IO
@mindthemonkey while the guest scheduler is only controlling the virt cpu and ESX is scheduling in the larger context, I don't see how this substantially effects the cpu% and load accounting. Both are based on samples taken at some frequency and to the extent the guest is pre-empted by the hypervisor scheduling, that will effect both the slices where real work gets done and slices where the guest takes its samples.
Feb
27
comment High Load Average with modest CPU Utilization and almost no IO
@mindthemonkey There's at least a couple dozen different services in the VMs. Some have substantially different profiles, but the majority look pretty similar to this service. 4GB mem, 2 virt cpus, modest IO (mostly network and basic logging), run 30-60% cpu utlization through the daily curve. IO and/or memory intensive nodes (DBs, SOLR) get dedicated hosts. Most of these other service vm's show the expected correlation between cpu% and load (at least as long as they stay healthily away from 100%).
Feb
27
awarded  Commentator
Feb
27
comment High Load Average with modest CPU Utilization and almost no IO
@ewwhite added requested info. (Except I can't answer "etc." because the world is too big to fully describe. :)
Feb
27
revised High Load Average with modest CPU Utilization and almost no IO
added 34 characters in body
Feb
27
revised High Load Average with modest CPU Utilization and almost no IO
update esx version
Feb
27
answered Is this all I need to deny an IP in nginx?
Feb
27
comment High Load Average with modest CPU Utilization and almost no IO
Any load avg above cpu count means I have more processes waiting to run than cpus to run them. I see many intervals above 2.0 and several la-5s over 4 and up to 6.5. This means I often have processes that are stalling for cpu behind other processes and implies unwanted latency for lack of cpu capacity. I normally expect load average and cpu% to correlate until the system starts approaching 100% cpu saturation; after that load average is the better signal because it indicates how over-committed the system is, not just that it's 100% busy.
Feb
27
comment High Load Average with modest CPU Utilization and almost no IO
I would add that this is a web service, though not a web server. It does have an execution profile akin to a bunch of other similar services we run: receive and deserialize a request, perform some dispatching to upstream services/databases, compute a result based on the answers from the upstreams, serialize a response, scribble a log msg. Median request duration ~60ms, 90% 200ms, 99% 500ms+. We have a bunch of other services with similar profiles running on comparable vm containers that do not exhibit this disconnect between Load and CPU%.
Feb
27
revised High Load Average with modest CPU Utilization and almost no IO
fix header alignment, add some more info
Feb
27
revised High Load Average with modest CPU Utilization and almost no IO
fix header alignment
Feb
27
comment High Load Average with modest CPU Utilization and almost no IO
iotop is boring, everything says approximately 0.
Feb
27
comment High Load Average with modest CPU Utilization and almost no IO
s/uninterpretable/uninterruptible/
Feb
27
comment High Load Average with modest CPU Utilization and almost no IO
Thanks for your response. What constitutes a "fractional time slice"? As I understand the scheduler, a process gets assigned to a cpu and runs on that cpu until the next scheduling interval or until it makes a blocking system call which causes it to yield. It my cpu is idle 70% of time but my run queue lengths average over 2, that puzzles me, why aren't these ready-to-run processes just getting scheduled to the mostly idle cpus?
Feb
27
comment High Load Average with modest CPU Utilization and almost no IO
Linux includes processes in uninterpretable sleep in its load calculation. Such processes show with State 'D' in the usual process inspection tools. This state is usually used by device drivers waiting for disk or network IO. That "usual explanation" is true for Linux, but not most other unixes.
Feb
27
awarded  Student