We have a quad-core i7, 8G machine on a gigabit network. There are 4 WiFi access points to which we connect 50 iPads which has an app that downloads a file from the server.
The server has NGINX running with a location to a folder on the server. The folder has a file 190MB large.
We tested 3 iPads. The download took about 1 minute 10 seconds, or 2.7Mbps.
Then, we start the download of the file as concurrently as we can on the 50 iPads.
What we see is 20 iPads failing and 30 succeeding. In the NGINX access log we see 30 200 OK with the full byte count for 190MB and 20 with byte counts around the 80MB mark.
During the test, I was running atop. I noted that eth0 was no more than 6% at any time. CPU was < 1% usage, RAM was 5GB free unmoving. NGINX processes were sometimes at the top of the process list but most of the time not. Ordered by RAM, then CPU. si was at its height 1900K and so was 90Mbps.
I am not sure how to debug this problem. Our network team are coming in today but I want to be able to tell them something useful. Ideally - it's not the server. I don't think it's the server given the atop readings. I would appreciate other ways to debug this kind of problem where the access log clearly shows clients not completing downloads.
If those that failed got to 80MB, then it was never a problem with them connecting and downloading alongside those that succeeded right? Something at around 80MB interrupted? It seems interesting that a lot of the fails were around the same mark (some weren't). Does this suggest a network blip? Could the server itself for any reason decide not to continue fulfilling existing connections?