2

I'm currently working on optimization of a web server, however I'm quite stuck at one particular problem. I'm using jmeter to simulate simulate load. jmeter is configured as follows:

  • 400 threads
  • Ramp-up 30 seconds
  • Loop count 1
  • Each thread visits 17 different pages on the server with a 1 - 5 second delay between each request.

What I'm experiencing is, up to 350 threads everything seems to be working as it should. The load and cpu usage increases, the site becomes noticeable slower but is till usable.

Somewhere between 350 - 400 threads however something happens. The load drops to nearly nothing , cpu is idling about 75 - 85% and the site hangs for several minutes for everyone.

What I have ruled out:

  • The server does not swap, at least it does not show in top and collectd graphs.
  • There is no MySQL queries that are waiting to finish (as reported by MySQL Administrator). Although I'm seeing a lot of open connections.
  • max_connections in MySQL is 1600 (1 MySQL connection per request, so this limit is far from reached)
  • wait is so to speak non existent in cpu graphs (collectd)
  • We are using memcached, but the timeout is set to 1 second.
  • memcached is run on the same server so network latency should not be an issue.
  • MaxClients and ServerLimit is not reached in apache

I'm running out of ideas of how to track this issue down. Any tips, tricks or ideas to help pin the reason down?

Thanks

Improve this question

1 Answer 1

Reset to default
1

You should check the maximum open file descriptors allowed for each running program.

You can see this limit with ulimit -n and modify the default behaviour in the file /etc/security/limits.conf.

If you have allowed 2000 clients to connect to MySQL but only 1024 (the default limit) file descriptors can be opened (network sockets are included), it will stop responding to new connections without warning.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .