Quick architecture overview:
Nginx serves up static assets and is running as a reverse proxy to PHP (FastCGI, mod_fcgid + Apache).
Memcached is running on the same server as well:
memcached -d -p 11211 -u memcached -m 1024 -c 4096 -P /var/run/memcached/memcached.pid -l 127.0.0.1
I have PECL Memcache extension compiled for PHP.
A couple one-off apps connect directly to Memcache, but typical usage is < 100Kb (for storing short strings or configurations). Point is, I know it's working.
I want to be able to swap out WordPress's object cache, which acts as an in-memory cache of data fetched from the DB (by creating hashes using arrays in PHP), with Memcache. This will allow cached data to persist across multiple requests, eventually across multiple application servers, etc. This is easily possible thanks to a "drop-in" module which replaces WordPress object cache with Memcache. This is pretty much industry standard stuff - people have done it many times before with no issues.
I'm running into a problem though. When I drop in Memcache, I see plenty of data going into Memcache, but also a huge spike in network traffic on my server. I also see more and more php-cgi processes, server load increases, and eventually I'm forced to remove the drop-in and revert back to plain old PHP for caching, which will eventually no longer be viable.
I can rationalize why network traffic increases, if Memcache is being accessed via 127.0.0.1, and that traffic counts in Munin, which I'm using to monitor the server. Here's the Memcache data I'm seeing:
Trying to figure out why the PHP-CGI processes seem to hang is the real problem. When I check Apache error logs after my experiments, I see:
[Fri Feb 21 19:11:20 2014] [warn] [client 220.127.116.11] mod_fcgid: read data timeout in 241 seconds
[Fri Feb 21 19:11:20 2014] [error] [client 18.104.22.168] Premature end of script headers: index.php
Yeah, I know 240 seconds is quite a long time for a timeout, and I can clearly lower that. But this problem remains, and is baffling. One thing I know is a problem is that some objects being sent to Memcached are too large to be stored there... but... Memcache just returns false on failure and continues on as normal, so I don't think that's a problem.
I've turned on verbose mode in Memcache to see if I can find any errors, and the Apache logs aren't much help (because the PHP-CGI processes are hanging, and that's all the info I have).
So I ask for your help! Where should I start in trying to figure out what's going on here? How might I debug Memcached + PHP? Is there a way to add some verbosity to PHP-CGI processes to see what's going on that's causing them to hang?