0

Nearly every 4 minutes, I'm seeing the following out of memory error in the php error log:

01-Jul-2014 21:50:03 UTC] PHP Fatal error:  Allowed memory size of 268435456 bytes
exhausted (tried to allocate 72 bytes) in /home/[sitename]/public_html/wp-includes
/wp-db.php on line 1938

The error message seems to support that the php.ini setting of memory_limit = 256M is thought to be true by PHP. However, I've used several memory monitoring plugins in wordpress and they all report that the site is using ~35MB of RAM at steady state, and it never seem to grow at all until the OOME happens. The memory had previously been set at lower levels and increased repeatedly without fixing the symptom. It's nearly always exactly 4 minutes. Occasionally it's exactly 3 minutes, or 3 minutes 30 seconds, etc. I installed a wordpress cron plugin to see if something was scheduled to run at 4 minute intervals, but nothing appears to be.

I have checked the httpd.conf file and confirmed that there is no RLimitMEM setting. I also confirmed via apachectl -V that I'm looking at the right httpd.conf file. Top says there's half a GB of free RAM available to the system. I have found no correlation between entries in the access log and the OOMEs in the php error log.

This server is hosting a fairly large number of sites. I don't manage the server, but I've been helping troubleshoot some issues on the site in question.

I'd be grateful for any suggestions on how to continue troubleshooting this.

3
  • 1
    Two things that will help are finding the URL that causes that error (so you can reproduce the bug at will) and looking at the code mentioned in the error message.
    – Ladadadada
    Jul 3, 2014 at 6:44
  • are you still monitoring this thread? I have a question about this process - I'm not familiar with what you mean by "print out" - like within the file itself it will list the problems? would you mind emailing me?
    – jessica
    Jun 12, 2018 at 0:29
  • (i am having the same problem and have ithemes but not the other plugin so need to figure out which plugin is interacting with ithemes.)
    – jessica
    Jun 12, 2018 at 0:30

1 Answer 1

5

I finally figured this out.

The problem was a conflict between two plugins installed on the site (and specifically in the way the two plugins were configured.) The iThemes Security plugin (http://ithemes.com/security) is configured to make site backups periodically. The code to make the DB backup does a dump of every table in the database, and it assumes that the contents of each table will fit in memory in its entirety. Unrelated to this, there is another plugin installed on the site called Redirection (http://urbangiraffe.com/plugins/redirection/) which is used to maintain redirects. This plugin has configuration option to log redirects as well a 404 responses. Unfortunately these logs were set to never expire and b/c of the volume of botnet traffic directed at our site, had accumulated nearly 90,000 redirect logs and 30,000 404 logs. Since the iThemes Security plugin tries to load the entire table into memory when it tries to create a backup, the wp_redirection_logs table consumed all available memory from php and crashed the process. I suspect that iThemes Security attempted to re-run the failed backup at each available opportunity, resulting in the errors every 3-4 minutes.

I fixed this by changing the redirection log setting to expire redirection entries after 5 days and not to log 404 errors at all. I then had to repeatedly refresh the log page to get the expired entries to be deleted. The out of memory error no longer occurs.

[edit] I've since heard from other wordpress people running into similar errors with the iThemse Security plugin doing SELECT * on various wordpress tables. Below is an explanation of how I debugged this issue in case yours is similar but not exactly the same:

The way that I went about troubleshooting the error after the normal fixes (increasing php memory and making sure that actually worked, confirming that my memory usage was stable and low over time) was to add some debugging log statements to wp-db.php so I could see what was happening when the error occurred. Here's the code change I made to help narrow down the problem (be sure to make a backup of wp-db.php before trying this so you can easily restore your settings and in case you mess something up when editing the file):

function get_results( $query = null, $output = OBJECT ) {
    $this->func_call = "\$db->get_results(\"$query\", $output)";

    if ( $query )
        $this->query( $query );
    else
        return null;

    $new_array = array();
    if ( $output == OBJECT ) {
        // Return an integer-keyed array of row objects
        return $this->last_result;
    } elseif ( $output == OBJECT_K ) {
        // Return an array of row objects with keys from column 1
        // (Duplicates are discarded)
        foreach ( $this->last_result as $row ) {
            $var_by_ref = get_object_vars( $row );
            $key = array_shift( $var_by_ref );
            if ( ! isset( $new_array[ $key ] ) )
                $new_array[ $key ] = $row;
        }
        return $new_array;
    } elseif ( $output == ARRAY_A || $output == ARRAY_N ) {
        // Return an integer-keyed array of...
        if ( $this->last_result ) {
            $emited = false;
            foreach( (array) $this->last_result as $row ) {
                if ( $output == ARRAY_N ) {
                    // ...integer-keyed row arrays
                    if (!$emitted) {
                        error_log("Current Mem: " . memory_get_usage() . ", eak mem: " . memory_get_peak_usage());
                        error_log($query);
                        $emitted = true;
                    }
                    $new_array[] = array_values( get_object_vars( $row ) );
                } else {
                    // ...column name-keyed row arrays
                    $new_array[] = get_object_vars( $row );
                }
            }
        }
        return $new_array;
    } elseif ( strtoupper( $output ) === OBJECT ) {
        // Back compat for OBJECT being previously case insensitive.
        return $this->last_result;
    }
    return null;
}

This is 6 lines of added code; the first assigns the $emitted variable to false to keep track of whether we've already logged for this request, and then 5 lines of the if clause to actually log.

What this does is print out the current and peak memory consumed by php before we start reading the query results into memory, and also print out the query that was executed. The memory will give you an idea of if you have reasonable memory available before we start reading the results. If the available memory is close to your limit (within a few MB), then the problem is probably elsewhere and you don't really have the available headroom to run a reasonable size query. If, like me, you have tons of free memory before the query is run, then look to see what the query is that runs right before you run out of memory (my log entries were all in php error log, but if yours are split across and iThemes log and the php error log, correlate between the two based on timestamps.) The query that was run immediately prior to the memory error is the one that blew you up.

In my case it was a SELECT * FROM wp_redirection_logs;. That table had grown out of control b/c the Redirection plugin was misconfigured on my site to never expire log entries. From reading the iThemes security plugin's code, it's clear that the backup action does a SELECT * FROM query on every table in your DB that starts with the wp_ prefix (or other prefix if your site is configured to use a different prefix due to multi-site or something else.) Other areas of iThemes Security (like 404 error logs) also seem to issue SELECT * queries against tables that could exceed available memory. Once you find what the query is, you can start to reason about the cause of your error, and perhaps prune unnecessary DB content to workaround the issue like I did.

I'd be happy to offer suggestions if you go through these steps and report back.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .