nginx - Takes too long for a connection to be established

Question

I am new to nginx, and I am using it currently on my server to serve videos to users. a few days ago the server started becoming extremely slow while have around 500-600 users online, note that I have a 1gbps line on the server so its not the bandwidth. it takes the server around a minute or 2 to respond but once the file starts streaming the speed is decent, so its taking the server long to spot the file I guess! I was wondering if anyone knows how I can fix this? I have already tried increasing the open file limit and worker limit, but still not luck!!!

I am also interested to hire someone to fix this for me if they know how to do it exactly and have previous experience with a similar situation.

Regards

ADDITIONAL INFO

         total       used       free     shared    buffers     cached
Mem:          7861       7695        166          0          3       4233
-/+ buffers/cache:       3459       4402
Swap:         4095         44       4051

Answer 1

So, you need to work from the bottom-up to find the bottleneck. I wrote a fairly comprehensive article on finding performance bottlenecks a couple of years ago, which should be helpful.

In short, you need to identify the exact system component that's the source of the problem (using tools like sar or it's many equivalents), then fix it. Once you fix that, something else will become the bottleneck, so you repeat the process until performance is acceptable.

In your specific case, disk I/O might be the problem, but the complicating factor is the fact that it takes a long time for requests to be serviced but they stream OK. I'd be looking into where the holdup there is -- is the kernel establishing the TCP connection quickly (look for the three-way-handshake with tcpdump), is nginx getting an accept through quickly (strace nginx and it's workers), is the HTTP request processing taking a while (nginx log files at maximum debug, cap'n)? A methodical approach to problem-solving will cure many ills.

(As an aside, whilst I don't do consulting, I do work for a hosting company that specialises in full-stack system support, so if you need paid help, it's always there)