I've had a few cases in which the homepage of one of my sites wouldn't load in the browser. The site is running on django/fastcgi/nginx.
It was difficult to reproduce so, in an attempt to understand the frequency of this problem, we've added a 1x1px images to the homepage body, served in the same way as the homepage html. Then, we wrote a script that scans the nginx logs and checks for each request for the homepage if a request for the 1px image occurred, from the same ip, within 10 seconds from the request for the homepage.
The results were shocking - about 30% (!!) of the homepage requests didn't have a request for the image in close proximity, and that's after filtering out requests from obvious searchbots etc. Many of such requests are even from high quality sources, i.e. visitors who are very likely to at the very least want to see the rendered homepage before leaving the site...
Therefore, I strongly suspect there is some sort of technical problem that's causing many requests to fail.
How should I go about troubleshooting this?