Skip to main content
31 votes
Accepted

Why is response time exploding when request frequency drops?

This is common behavior in data centers. The times your response time is slow corresponds to what is commonly called the Batch Window. This is a period of time when user activity is expected to be ...
BillThor's user avatar
  • 28k
30 votes

High server load - [jbd2/md1-8] using 99.99% IO

This is not really an answer as there is not enough context to give the exact cause, but it is a description of how I managed to track this when it happened to me. I noticed my jbd2/md0-8 kept ...
Iwan Aucamp's user avatar
12 votes

HA Proxy - roundrobin vs leastconn

It depends on what's the protocol and the use case to balance. For anything where the amount of connections is correlated with the load/usage, it's better to use leastconn. Because of the way networks ...
user5994461's user avatar
  • 3,038
11 votes

Fast pinging without requiring the response

ping -f will not wait for the remote party to reply. However, a 1us delay between consecutive packets can simply be too much to ask from your hardware (CPU) and software (IP stack) combination. To ...
shodanshok's user avatar
  • 51.2k
9 votes

How does a typical http(s) load balancer work?

This depends on the purpose of the load balancer i.e. what type of workload it's supposed to balance. The reverse proxy based HTTP(S) load balancers are distributing the load between application back-...
Esa Jokinen's user avatar
  • 50.4k
9 votes
Accepted

High CPU usage without useful information from top command

About 30% of CPU time seems to be spent in WAIT, and your (1 minute, at least) load is very high. So I would start by checking your storage, and storage usage patterns. A good starting point might be ...
iwaseatenbyagrue's user avatar
8 votes

Why is response time exploding when request frequency drops?

This relation may happen in the other direction if the request senders wait for a previous request to complete before submitting a new one. In that case traffic drops as request times grows (for ...
Karol Nowak's user avatar
7 votes

Why is response time exploding when request frequency drops?

Though @BillThor's answer may be correct, it seems unlikely that the period of low load is entirely taken up by backup processes (i.e. that the periods match precisely). An alternative explanation is ...
abligh's user avatar
  • 285
7 votes

CPU load for high performance computing

Load average shows the queue of threads ready to run. In Linux, this includes also threads waiting for disk. It could happen that a broken NFS server could increase the load average to insane numbers. ...
Mircea Vutcovici's user avatar
6 votes

High CPU usage without useful information from top command

You have not described what the problem is. If there was a performance problem, such as applications not being responsive enough, describe it. The load average is not the number of cores in use. It ...
John Mahowald's user avatar
6 votes

NGINX: Request Limit timeout behavior for queries in the queue (burst)

Seems there is a bit of confusion how nginx rate limiting and timeouts work. There is no timeout for rate limiting. You just set a rate and a queue size. Any requests exceeding the rate are being ...
Bob's user avatar
  • 472
4 votes

Fast pinging without requiring the response

Probably you make a confusion between latency and throughput. It's enough to send 1 packet and measure its delay to determine the latency. All you need is a mechanism to measure the time with high ...
marian adam's user avatar
4 votes

High CPU usage without useful information from top command

You should install atop. It is more informative. Your CPU is used by input/output operations. WA - is waiting and it is more 30%. I see that more useful process is mysqld. It eat CPU time more than ...
Mikhail Khirgiy's user avatar
4 votes

High load average, when should I be worried?

Only worry if it actually corresponds to a slow application. A bit more precisely, load average relates to the number of processes running or waiting. This can be a lot more than 1 and perform just ...
John Mahowald's user avatar
4 votes
Accepted

LAMP stack with AWS performance

Yes, you can use AWS for a site with a scale from 1 request a day to 1 million hits per second and beyond. You really need AWS education to use AWS effectively, including designing for cost savings. ...
Tim's user avatar
  • 33.1k
4 votes
Accepted

How to tune TCP for high-frequency connections between two nodes

In our data center we have an F5 running on BigIP hardware that acts as single ingress point for HTTPS requests from client machines in our various office locations across the country. If this single ...
poige's user avatar
  • 9,605
4 votes
Accepted

How multiple load balancers work

Instead of passing all the traffic through a single load balancer you can load balance on the network level utilizing Equal-cost multi-path routing (ECMP). There are two presentations from HAProxy ...
AlexD's user avatar
  • 10.4k
3 votes
Accepted

Why there are so many "ss" process on my server (CentOS with nginx serving the web)?

Use ps auxw or ps -ef and look at parent process identifier (PPID) column. Both netstat and its successor ss are mainly used for monitoring. These are normal admin tools, not any background services. ...
kubanczyk's user avatar
  • 14k
3 votes

Explain load averages on Solaris 10

Quite a late reply but the accepted answer still has incorrect statements, is missing parts of the point, and suggest statistics lies while there is no reason here not to trust the ones reported by ...
jlliagre's user avatar
  • 8,951
3 votes

How to detect if Apache is (on the verge of) being overwhelmed by requests?

Regarding quantifying the performance of apache to the end users, the time taken to serve a response is useful and it will increase with load on the server. I typically combine logging of this value ...
Tom's user avatar
  • 11.4k
3 votes

How to tune TCP for high-frequency connections between two nodes

Although I also think adding more IP addresses is the simplest way forward, have you considered exploring reusing TCP connections between the F5 and the Traefik nodes instead of creating a new one per ...
Pedro Perez's user avatar
  • 6,562
3 votes

RDS MySQL Insights: Top Query "commit"

We finally were able to tackle this "commit" load issue. A few facts about our system: About 95% of our SQL queries are reads About 5% of our write queries are writes Most of our queries ...
Simon Ninon's user avatar
3 votes

MariaDB crazy at 500% cpu time

High CPU usage usually means lots of in-memory table-scanning. Looking at one of your slow queries: Time=5.64s (5s) Rows_sent=0.0 (0) Rows_examined=4088258.0 (4088258) SELECT udownload_id AS retval ...
Phill  W.'s user avatar
  • 1,531
3 votes

Linux server high load spikes

A few things pop out to me kworker/u256:1 get_write_access php-fpm7.2 call_rwsem_down_write_failed php-fpm7.2 locks_lock_inode_wait What this is telling you is that php is doing some file ...
Matthew Ife's user avatar
  • 23.8k
2 votes
Accepted

How to detect if Apache is (on the verge of) being overwhelmed by requests?

I think you're right watching system resources. Load average, IO load, memory, swap, CPU, etc... You'll probably benefit from some detail on Apache's internal status, like what it's processes are ...
Ryan Babchishin's user avatar
2 votes

How can I handle heavy Database interaction at peak hours?

You need to put monitoring in place to gather metrics and use Scientific Method to identify bottlenecks. You can then plan, implement and test solutions. Iterate as required.
user9517's user avatar
  • 116k
2 votes

Strange values in linux system load

I think that you might have misunderstood how the "load average" work. First of it's not system "load" it's system "load average" - There is a big difference as the numbers are all averages (meaning ...
davidbl's user avatar
  • 86
2 votes

Determine php script causing MySQL huge load

If you enable the slow query log in mysql, it will give you a starting point for any SQL statements that take a lot of resources. It won't tell you which PHP script was the culprit, but you will have ...
Tom's user avatar
  • 11.4k
2 votes

How to detect if Apache is (on the verge of) being overwhelmed by requests?

Disclaimer: I work at www.mosoplot.com, which automates most of the capacity planning needed here. I've used some charts from MOSOplot to demonstrate how to do this. This is how I do capacity ...
Nathan Webb's user avatar
2 votes

High system interrupt rate

I assume you don't have single socket system with 24C CPU. So it's probably NUMA system with 2x12C. In that case I'd suggest to make sure the program uses only one numa node (usually socket) and it's ...
Jaroslav Kucera's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible