We run a cluster of 45 linux machines with a lot of different stuff running (nginx, Varnish, Apache, MySQL, memcache etc) and load getting balanced on different machines. We do around 3000 requests per second and do some very computationally heavy requests in real time.
We do have a good understanding of Linux by now, but we do keep facing issues of max_connections reached, SYN Cookies sent, IP Tables issues, etc etc. We learn from our syslog and google around it and make a fix. I am hoping to understand all the different parameters and stuff that matters in a Linux box (ubuntu / Debian perspective) that I should know before running a very heavy service on a machine. The service is ought to get many thousands of requests per second from people across the globe (different IPs). A server interact with a bunch of other servers to complete a request.
Is there any book / set of books, I should get to understand the linux system and how it handle a request and what parameters matter where. Also any specific documentation on nginx, varnish, memcache, LAMP, MySQL that I should read before scaling things to many thousand requests per second level.