Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We're using LDAP as a backend user database for our drupal sites (drupal reads from/and writes to it). LDAP has been running fine for weeks but for some reason it justs stopped running today. I was looking at syslog using the below command:

$ grep -i "error\|panic\|unable\|warning\|fatal\|issue" /var/log/syslog | grep slapd

The first errors that appear are these ones:

Sep 20 09:59:38 server slapd[18158]: bdb(dc=example,dc=org): PANIC: fatal region error detected; run recovery
Sep 20 09:59:38 server slapd[18158]: conn=1010 op=7 RESULT tag=107 err=80 text=internal error

Q1: Is there a way to find out more as to what caused the fatal region error? (maybe the ldap module of drupal is issuing commands that openldap doesn't like).

Following the advise from the error message I ran the recovery tool:

$ sudo /etc/init.d/slapd stop
$ sudo /usr/bin/db4.7_recover -v -h /var/lib/ldap/

Finding last valid log LSN: file: 3916 offset 298418
Recovery starting from [3916][298273]
Recovery complete at Tue Sep 20 12:17:42 2011
Maximum transaction ID 80000017 Recovery checkpoint [3916][298418]

$ sudo /etc/init.d/slapd start

What I see next in syslog is this:

Sep 20 12:17:47 server slapd[10286]: @(#) $OpenLDAP: slapd 2.4.21 (Jun  2 2011 19:36:19) $#012#011buildd@allspice:/build/buildd/openldap-2.4.21/debian/build/servers/slapd47
Sep 20 12:17:47 server slapd[10286]: PROXIED attributeDescription "DC" inserted.
Sep 20 12:17:48 server slapd[10287]: bdb(dc=example,dc=org): /var/lib/ldap/log.0000003916: log file unreadable: Permission denied  <----- HERE
Sep 20 12:17:48 server slapd[10287]: bdb(dc=example,dc=org): PANIC: Permission denied
Sep 20 12:17:48 server slapd[10287]: bdb(dc=example,dc=org): Invalid log file: log.0000003916: DB_RUNRECOVERY: Fatal error, run database recovery
Sep 20 12:17:48 server slapd[10287]: bdb(dc=example,dc=org): PANIC: DB_RUNRECOVERY: Fatal error, run database recovery

What is happening there is that one of the log files isn't readable by slapd as indeed the owner has changed from openldap to root and permissions are set to 600:

/var/lib/ldap/log.0000003916: log file unreadable: Permission denied

$ sudo ls -lh /var/lib/ldap/
-rw------- 1 openldap openldap  10M 2011-09-09 11:09 log.0000003914
-rw------- 1 openldap openldap  10M 2011-09-20 09:54 log.0000003915
-rw------- 1 root     root      10M 2011-09-20 12:10 log.0000003916 <----- HERE
-rwxr-xr-x 1 openldap openldap  20M 2011-09-20 12:51 mail.bdb
-rwxr-xr-x 1 openldap openldap 1.7M 2011-09-20 12:51 objectClass.bdb
-rwxr-xr-x 1 openldap openldap  12M 2011-09-20 12:51 sn.bdb
-rwxr-xr-x 1 openldap openldap 1.4M 2011-09-20 12:51 uid.bdb

I fixed this by changing the owner from root to openldap:

$ sudo chown openldap:openldap /var/lib/ldap/log.0000003916

Then slapd started properly:

Sep 20 13:01:40 evergreen slapd[18901]: @(#) $OpenLDAP: slapd 2.4.21 (Jun  2 2011 19:36:19) $#012#011buildd@allspice:/build/buildd/openldap-2.4.21/debian/build/servers/slapd
Sep 20 13:01:40 evergreen slapd[18901]: PROXIED attributeDescription "DC" inserted.
Sep 20 13:01:41 evergreen slapd[18902]: slapd starting

I didn't check the permissions of that log file before running db4.7_recover so I don't know what set the owner to root.

Q2: Is it recommended to run db4.7_recover as openldap to avoid permission issues on log files (e.g. do sudo -u openldap /usr/bin/db4.7_recover -v -h /var/lib/ldap)?

Q3: As a more generic question: is there a way to monitor which process last changed the permission on a file? (that way if the issue is not with running db4.7_recover as sudo I can find out what it is).

Q4: Have you got more advises as to how troubleshooting this type of LDAP issues?

Additional info:
-server=ubuntu 10.04
-openldap=https://help.ubuntu.com/community/OpenLDAPServer

share|improve this question
    
Copy one of the logfiles to a separate directory. Run db4.7_recover on it as root, and see if the permissions change. That'll answer what-you're-getting-at with Q2 and Q3. I strongly suspect that it's creating a new file - which will be owned by the user running the recovery - and copying it over the top. –  nickgrim Sep 20 '11 at 13:54

1 Answer 1

up vote 1 down vote accepted

On busy servers I've seen OpenLDAP having problems with default 1024 max file handles. Increase that limit in /etc/security/limits.conf for openldap user and see if that fixes the problem. Usually if that is the reason slapd will complain about it to logs.

Under certain circumstances slapd can also leak memory over time and as a result Linux kernel OOM will kill it. If that happened, dmesg should tell you a tale how out of memory killer bravely killed slapd.

Q2: should not matter as long as you fix the permissions before starting slapd.

Q3: Linux audit subsystem can help you with that. Read man auditd and auditctl; answering to this is out of scope of your original Q :)

Q4: Increase slapd log level to maximum, read the logs. Google.

share|improve this answer
    
Accepted? :) I'm curious what finally was the reason for the dying slapd. Out of memory? Out of file handles? –  Janne Pikkarainen Sep 21 '11 at 9:47
    
I normally accept when I have the feeling that I won't be getting better answers. It sometimes happens that someone comes up with a better answer later on in which case I un-accept the answer I previously accepted and accept the new one instead. Accepting now is also a way not to forget rewarding whoever spent some time helping me out. I haven't started the LDAP investigation so unfortunately can't tell you :( –  user64204 Sep 26 '11 at 11:57

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.