ISC DHCPD memory leak solutions and or workarounds

Question

Background:
I have recently been put in charge of finding the root of and/or fixing a memory leak with an ISC DHCPD (4.2.5-P1) installation on a Debian (Lenny) Unix server.

I have been researching the problem for over a week now and have retrieved a lot of information about the fact that the system is indeed leaking, but I haven't found an actual answer as to why it is or how to stop it.

I have currently:

• used valgrind in vgdb mode to detect memory leaks and allow for line inspection of the code
• using valgrind discovered 2 possible start points for the leak
• compiled the DHCPD source with CFLAGS=-DDEBUG_MEMORY_LEAKAGE_ON_EXIT (this appears to stop memory leaks)
• run the newly compiled DHCPD binary as dhcpd -6 -d -cf /etc/dhcpd6.conf
• taken a snapshot of the vsz and rss binaries in 10 minute intervals over the course of 72 hours using the following script

Script:

#!/bin/bash
#probably could have used watch
while [[ 0 -eq 0 ]]; do
    ps -eo vsz,rss,command | grep "dhcpd6.conf" | grep -v grep >> memory-usage.txt
    sleep 600
done

I did a little bit of research on VSZ and RSS. If the RSS size stays the same, but the VSZ size increases it appears that there is a clear memory leak. However, in my situation the VSZ and RSS are both increasing. [Starting size day 1: VZS=8560 RSS=6292 => Ending size day 3: VZS=67168 RSS=64860]

I also looked at /proc/PID/maps to see if I could get any information there, but I was unable to find anything of use.

/proc/PID/maps information:

08048000-081e3000 r-xp 00000000 08:05 119382     /usr/sbin/dhcpd
081e3000-081e8000 rw-p 0019b000 08:05 119382     /usr/sbin/dhcpd
081e8000-08222000 rw-p 081e8000 00:00 0
09fea000-0a11c000 rw-p 09fea000 00:00 0          [heap]
b72b7000-b72c1000 r-xp 00000000 08:01 6184       /lib/i686/cmov/libnss_files-2.7.so
b72c1000-b72c3000 rw-p 00009000 08:01 6184       /lib/i686/cmov/libnss_files-2.7.so
b72c3000-b7673000 rw-p b72c3000 00:00 0
b7673000-b77c8000 r-xp 00000000 08:01 6192       /lib/i686/cmov/libc-2.7.so
b77c8000-b77c9000 r--p 00155000 08:01 6192       /lib/i686/cmov/libc-2.7.so
b77c9000-b77cb000 rw-p 00156000 08:01 6192       /lib/i686/cmov/libc-2.7.so
b77cb000-b77ce000 rw-p b77cb000 00:00 0
b77cf000-b77d0000 rw-p b77cf000 00:00 0
b77d1000-b77d4000 rw-p b77d1000 00:00 0
b77d4000-b77d5000 r-xp b77d4000 00:00 0          [vdso]
b77d5000-b77ef000 r-xp 00000000 08:01 2022       /lib/ld-2.7.so
b77ef000-b77f1000 rw-p 0001a000 08:01 2022       /lib/ld-2.7.so
bfe0d000-bfe30000 rw-p bffdc000 00:00 0          [stack]

Question(s):
1. How should I go about debugging a memory leak like this?
2. ISC says that the solution is resetting the server every so often and that this is not a bug. If my client doesn't want to reset their server is there any middle ground? (They want hard proof that they have to follow through with a solution.)
3. Has anyone had experience with a dhcpd related leak since January, 2013?
4. Is there a solution or workaround available for this problem?

Related Link(s):
1. https://kb.isc.org/article/AA-00737 (ISC Report)
2. https://access.redhat.com/site/solutions/402713 (This bug report matches the start point of my memory leak [OMAPI FUNCTIONALITY])

If you need any additional information that may help in solving this problem, I ready to provide what I can.

In the mean time I am going to see if I can compile the binary and disable OMAPI functionality.

DHCP 4.3.0a1 was just released, so I'm going to see whether this changes anything (there was no information on the change log about a bug leak fix, but it doesn't hurt to try).

Thanks for your time.

Answer 1

As a workaround, you might consider running dhcpd with memory limits under a process supervisor like runit.

Hopefully dhcpd will abort if it fails to allocate memory, at which point the process supervisor will restart it.

Or you can just restart it periodically from cron -- it's still less invasive than rebooting the entire server.