I finally figured this out.
The problem was a conflict between two plugins installed on the site (and specifically in the way the two plugins were configured.) The iThemes Security plugin (http://ithemes.com/security) is configured to make site backups periodically. The code to make the DB backup does a dump of every table in the database, and it assumes that the contents of each table will fit in memory in its entirety. Unrelated to this, there is another plugin installed on the site called Redirection (http://urbangiraffe.com/plugins/redirection/) which is used to maintain redirects. This plugin has configuration option to log redirects as well a 404 responses. Unfortunately these logs were set to never expire and b/c of the volume of botnet traffic directed at our site, had accumulated nearly 90,000 redirect logs and 30,000 404 logs. Since the iThemes Security plugin tries to load the entire table into memory when it tries to create a backup, the wp_redirection_logs table consumed all available memory from php and crashed the process. I suspect that iThemes Security attempted to re-run the failed backup at each available opportunity, resulting in the errors every 3-4 minutes.
I fixed this by changing the redirection log setting to expire redirection entries after 5 days and not to log 404 errors at all. I then had to repeatedly refresh the log page to get the expired entries to be deleted. The out of memory error no longer occurs.
[edit] I've since heard from other wordpress people running into similar errors with the iThemse Security plugin doing SELECT * on various wordpress tables. Below is an explanation of how I debugged this issue in case yours is similar but not exactly the same:
The way that I went about troubleshooting the error after the normal fixes (increasing php memory and making sure that actually worked, confirming that my memory usage was stable and low over time) was to add some debugging log statements to wp-db.php so I could see what was happening when the error occurred. Here's the code change I made to help narrow down the problem (be sure to make a backup of wp-db.php before trying this so you can easily restore your settings and in case you mess something up when editing the file):
function get_results( $query = null, $output = OBJECT ) {
$this->func_call = "\$db->get_results(\"$query\", $output)";
if ( $query )
$this->query( $query );
else
return null;
$new_array = array();
if ( $output == OBJECT ) {
// Return an integer-keyed array of row objects
return $this->last_result;
} elseif ( $output == OBJECT_K ) {
// Return an array of row objects with keys from column 1
// (Duplicates are discarded)
foreach ( $this->last_result as $row ) {
$var_by_ref = get_object_vars( $row );
$key = array_shift( $var_by_ref );
if ( ! isset( $new_array[ $key ] ) )
$new_array[ $key ] = $row;
}
return $new_array;
} elseif ( $output == ARRAY_A || $output == ARRAY_N ) {
// Return an integer-keyed array of...
if ( $this->last_result ) {
$emited = false;
foreach( (array) $this->last_result as $row ) {
if ( $output == ARRAY_N ) {
// ...integer-keyed row arrays
if (!$emitted) {
error_log("Current Mem: " . memory_get_usage() . ", eak mem: " . memory_get_peak_usage());
error_log($query);
$emitted = true;
}
$new_array[] = array_values( get_object_vars( $row ) );
} else {
// ...column name-keyed row arrays
$new_array[] = get_object_vars( $row );
}
}
}
return $new_array;
} elseif ( strtoupper( $output ) === OBJECT ) {
// Back compat for OBJECT being previously case insensitive.
return $this->last_result;
}
return null;
}
This is 6 lines of added code; the first assigns the $emitted variable to false to keep track of whether we've already logged for this request, and then 5 lines of the if clause to actually log.
What this does is print out the current and peak memory consumed by php before we start reading the query results into memory, and also print out the query that was executed. The memory will give you an idea of if you have reasonable memory available before we start reading the results. If the available memory is close to your limit (within a few MB), then the problem is probably elsewhere and you don't really have the available headroom to run a reasonable size query. If, like me, you have tons of free memory before the query is run, then look to see what the query is that runs right before you run out of memory (my log entries were all in php error log, but if yours are split across and iThemes log and the php error log, correlate between the two based on timestamps.) The query that was run immediately prior to the memory error is the one that blew you up.
In my case it was a SELECT * FROM wp_redirection_logs;. That table had grown out of control b/c the Redirection plugin was misconfigured on my site to never expire log entries. From reading the iThemes security plugin's code, it's clear that the backup action does a SELECT * FROM query on every table in your DB that starts with the wp_ prefix (or other prefix if your site is configured to use a different prefix due to multi-site or something else.) Other areas of iThemes Security (like 404 error logs) also seem to issue SELECT * queries against tables that could exceed available memory. Once you find what the query is, you can start to reason about the cause of your error, and perhaps prune unnecessary DB content to workaround the issue like I did.
I'd be happy to offer suggestions if you go through these steps and report back.