3

Does apache2 support synchronized log for access_log and error_log? How the log in apache2 works for multiple instances? Will it get to lock condition if multiple instances write at the same time?

1
  • Thanks all for the answers. I am new in apache. Only proxy server function is used in our application and it has only access_log and error_log. From APR file_io write function I found it is using a pollset for log file descriptor if it can't do write.
    – user111757
    Mar 1, 2012 at 3:01

5 Answers 5

2

Simply put, no, Apache itself does not lock files--at the very least, this is a performance and resource hit. At most, its a potential deadlock waiting to happen. Since there can be hundreds of threads on a normal setup vying for the lock, you'll see lots of starvation going on.

The more complicated answer is, you can certainly try and implement this yourself. That is, you can use something like piped logs to pass them an intermediary script that handles locking.

The right answer is, why do you need to put these into the same file? If they're hosting the same data, then they should be the same server--splitting it up into two separate overseers (i.e., root owned httpd) and multiple children for each (i.e., apache owned httpd) won't buy you anything; if anything, the additional resources required to launch resources will decrement performance. And if they're NOT hosting the same content, then they shouldn't be logging things to the same log.

If you absolutely do need the content to be merged, there are plenty of things out there that can merge the content from two Apache log files.

Hope this helps!

2

Does apache2 support synchronized log for access_log and error_log?
No. As others have pointed out, there is no locking or mechanism to ensure that entries are written in sync to any two apache log files. The events ARE however timestamped (so you can match an error_log event to the access_log event that triggered it), and tools exist to merge the logs if that's what you want to do.

How the log in apache2 works for multiple instances?
See the Apache manual section on logging, particularly the sub-section on virtual hosts.
Basically the answer is "It works however you configure it, within the limits described in the manual". The advice MrTuttle gave you (use a separate log for each site, and certainly separate logs for separate httpd instances!) is very good advice - I suggest following it...

Will it get to lock condition if multiple instances write at the same time?
No (see above), however you may experience some events written out of order into the log file under extreme load. (I've seen this ONCE, and the site was using piped logs, so it's possible the program that was handling the logs simply lost its mind...)

1

No, you won't getting a locking condition. Apache's logs aren't opened with an exclusive lock (i don't know if this is true on windows).

That said, use separate logs for your own sanity.

1

Multiple apache processes from the same instance can write to the same logfiles without any concern, because the fundamental IO API's guarantee this is safe (won't lose data or interleave) when a file descriptor is shared between N processes or threads.

0

The answer is that Apache2 is, obviously, programmed to support any number of child processes or threads and they all have the ability, at any time, to write to some log file.

For that purpose, Apache2 uses a library named libapr. The log implementation calls a function called apr_file_puts() with the log messages fully processed (i.e. one string). In turn, the apr_file_puts() function calls the apr_file_write() function presented below.

What do we see in that function?

    if (thefile->flags & APR_FOPEN_XTHREAD) {
        apr_thread_mutex_lock(thefile->mutex);
    }

Oh! A lock... so already the people who say there are no locks have not looked too closely. This lock is useful for inter-thread use of this function. It would not prevent interleaved log data from two processes running in parallel (if you have Apache2 setup to fork() child processes instead of using threads). So...

APR_DECLARE(apr_status_t) apr_file_write(apr_file_t *thefile, const void *buf, apr_size_t *nbytes)
{
    apr_status_t rv;
    DWORD bwrote;

    /* If the file is open for xthread support, allocate and
     * initialize the overlapped and io completion event (hEvent). 
     * Threads should NOT share an apr_file_t or its hEvent.
     */
    if ((thefile->flags & APR_FOPEN_XTHREAD) && !thefile->pOverlapped ) {
        thefile->pOverlapped = (OVERLAPPED*) apr_pcalloc(thefile->pool, 
                                                         sizeof(OVERLAPPED));
        thefile->pOverlapped->hEvent = CreateEvent(NULL, FALSE, FALSE, NULL);
        if (!thefile->pOverlapped->hEvent) {
            rv = apr_get_os_error();
            return rv;
        }
    }

    if (thefile->buffered) {
        char *pos = (char *)buf;
        apr_size_t blocksize;
        apr_size_t size = *nbytes;

        if (thefile->flags & APR_FOPEN_XTHREAD) {
            apr_thread_mutex_lock(thefile->mutex);
        }

        if (thefile->direction == 0) {
            /* Position file pointer for writing at the offset we are logically reading from */
            apr_off_t offset = thefile->filePtr - thefile->dataRead + thefile->bufpos;
            DWORD offlo = (DWORD)offset;
            LONG  offhi = (LONG)(offset >> 32);
            if (offset != thefile->filePtr)
                SetFilePointer(thefile->filehand, offlo, &offhi, FILE_BEGIN);
            thefile->bufpos = thefile->dataRead = 0;
            thefile->direction = 1;
        }

        rv = 0;
        while (rv == 0 && size > 0) {
            if (thefile->bufpos == thefile->bufsize)   /* write buffer is full */
                rv = apr_file_flush(thefile);

            blocksize = size > thefile->bufsize - thefile->bufpos ? 
                                     thefile->bufsize - thefile->bufpos : size;
            memcpy(thefile->buffer + thefile->bufpos, pos, blocksize);
            thefile->bufpos += blocksize;
            pos += blocksize;
            size -= blocksize;
        }

        if (thefile->flags & APR_FOPEN_XTHREAD) {
            apr_thread_mutex_unlock(thefile->mutex);
        }
        return rv;
    } else {
        if (!thefile->pipe) {
            apr_off_t offset = 0;
            apr_status_t rc;
            if (thefile->append) {
                /* apr_file_lock will mutex the file across processes.
                 * The call to apr_thread_mutex_lock is added to avoid
                 * a race condition between LockFile and WriteFile 
                 * that occasionally leads to deadlocked threads.
                 */
                apr_thread_mutex_lock(thefile->mutex);
                rc = apr_file_lock(thefile, APR_FLOCK_EXCLUSIVE);
                if (rc != APR_SUCCESS) {
                    apr_thread_mutex_unlock(thefile->mutex);
                    return rc;
                }
                rc = apr_file_seek(thefile, APR_END, &offset);
                if (rc != APR_SUCCESS) {
                    apr_thread_mutex_unlock(thefile->mutex);
                    return rc;
                }
            }
            if (thefile->pOverlapped) {
                thefile->pOverlapped->Offset     = (DWORD)thefile->filePtr;
                thefile->pOverlapped->OffsetHigh = (DWORD)(thefile->filePtr >> 32);
            }
            rv = WriteFile(thefile->filehand, buf, (DWORD)*nbytes, &bwrote,
                           thefile->pOverlapped);
            if (thefile->append) {
                apr_file_unlock(thefile);
                apr_thread_mutex_unlock(thefile->mutex);
            }
        }
        else {
            rv = WriteFile(thefile->filehand, buf, (DWORD)*nbytes, &bwrote,
                           thefile->pOverlapped);
        }
        if (rv) {
            *nbytes = bwrote;
            rv = APR_SUCCESS;
        }
        else {
            (*nbytes) = 0;
            rv = apr_get_os_error();

            /* XXX: This must be corrected, per the apr_file_read logic!!! */
            if (rv == APR_FROM_OS_ERROR(ERROR_IO_PENDING)) {
 
                DWORD timeout_ms;

                if (thefile->timeout == 0) {
                    timeout_ms = 0;
                }
                else if (thefile->timeout < 0) {
                    timeout_ms = INFINITE;
                }
                else {
                    timeout_ms = (DWORD)(thefile->timeout / 1000);
                }
           
                rv = WaitForSingleObject(thefile->pOverlapped->hEvent, timeout_ms);
                switch (rv) {
                    case WAIT_OBJECT_0:
                        GetOverlappedResult(thefile->filehand, thefile->pOverlapped, 
                                            &bwrote, TRUE);
                        *nbytes = bwrote;
                        rv = APR_SUCCESS;
                        break;
                    case WAIT_TIMEOUT:
                        rv = (timeout_ms == 0) ? APR_EAGAIN : APR_TIMEUP;
                        break;
                    case WAIT_FAILED:
                        rv = apr_get_os_error();
                        break;
                    default:
                        break;
                }
                if (rv != APR_SUCCESS) {
                    if (apr_os_level >= APR_WIN_98)
                        CancelIo(thefile->filehand);
                }
            }
        }
        if (rv == APR_SUCCESS && thefile->pOverlapped && !thefile->pipe) {
            thefile->filePtr += *nbytes;
        }
    }
    return rv;
}

If you look closely enough in the apr_file_write() function, you'll notice calls to the apr_file_lock() function, which, obviously?, locks the file (temporarily). You'll notice the flock() function being called.

rc = apr_file_lock(thefile, APR_FLOCK_EXCLUSIVE);

That function will prevent other processes which also use the flock() function from reading from or writing to the file.

APR_DECLARE(apr_status_t) apr_file_lock(apr_file_t *thefile, int type)
{
    int rc;
    int ltype;

    if ((type & APR_FLOCK_TYPEMASK) == APR_FLOCK_SHARED)
        ltype = LOCK_SH;
    else
        ltype = LOCK_EX;
    if ((type & APR_FLOCK_NONBLOCK) != 0)
        ltype |= LOCK_NB;

    /* keep trying if flock() gets interrupted (by a signal) */
    while ((rc = flock(thefile->filedes, ltype)) < 0 && errno == EINTR)
        continue;

    if (rc == -1)
        return errno;

    return APR_SUCCESS;
}

But I can always look at the file so there must not be a lock on it?!

This is true in Linux. Probably not in MS-Windows. The file is always accessible, no matter what. The flock() does not prevent another process from reading the file unless that other process also attempts a lock. This is how tools such as tail and less can read the log files even when Apache2 is still writing to them.

So you do enter a lock condition all the time. However, it's fairly transparent for two reasons:

  1. The write is very short since one line of log is generally under 256 characters.

  2. In most likelihood, Apache2 will use a logging thread;

    the logs are added to a FIFO and processed later the FIFO also uses a lock on a mutex (there is no FIFO implementation that works without a lock), but this is close to transparent. Once the data is in that FIFO your process continues to run. The logging threads can take its time to save the logs to the file.

    (We see that thread implementation, which is called "Overlapped" after the same interface under MS-Windows.)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .