dhcp server gives "no free leases" error although there should be available IPs

Question

We have a small HPC cluster of about 40 machines running Rocks Linux Distro 6.2 (based on Centos 6.6). I have been trying to add 6 more machines to the system in the last few days. The first 3 were successfull but I can't pxe boot the other 3 machines. /var/log/messages shows "no free leases" error as below:

[root@topaz ~]# tail -n3 /var/log/messages 
Jun 29 17:56:00 topaz dhcpd: DHCPDISCOVER from d0:bf:9c:03:3b:40 via eth0: network 10.1.0.0/16: no free leases
Jun 29 17:56:03 topaz dhcpd: DHCPDISCOVER from d0:bf:9c:03:3b:40 via eth0: network 10.1.0.0/16: no free leases
Jun 29 17:56:07 topaz dhcpd: DHCPDISCOVER from d0:bf:9c:03:3b:40 via eth0: network 10.1.0.0/16: no free leases

All machines only have a local IP except for the frontend. There should be enough IPs availabe to assign to these new machines. There doesn't seem to be any unexpected machines connected to the network:

[root@topaz ~]# arp -a | awk '{ print $2 }' | sort   
(10.1.255.215)
(10.1.255.216)
(10.1.255.217)
(10.1.255.218)
(10.1.255.219)
(10.1.255.220)
(10.1.255.221)
(10.1.255.222)
(10.1.255.223)
(10.1.255.224)
(10.1.255.225)
(10.1.255.226)
(10.1.255.227)
(10.1.255.228)
(10.1.255.229)
(10.1.255.230)
(10.1.255.231)
(10.1.255.232)
(10.1.255.233)
(10.1.255.234)
(10.1.255.235)
(10.1.255.236)
(10.1.255.237)
(10.1.255.238)
(10.1.255.239)
(10.1.255.240)
(10.1.255.241)
(10.1.255.242)
(10.1.255.243)
(10.1.255.244)
(10.1.255.245)
(10.1.255.246)
(10.1.255.247)
(10.1.255.248)
(10.1.255.249)
(10.1.255.250)
(10.1.255.251)
(10.1.255.252)
(10.1.255.253)
(10.1.255.254)
(193.140.197.1)
(193.140.197.110)
(193.140.197.160)
(193.140.197.30)
(193.140.197.36)

Our leases file do not show any entries:

[root@topaz ~]# cat /var/lib/dhcpd/dhcpd.leases
# The format of this file is documented in the dhcpd.leases(5) manual page.
# This lease file was written by isc-dhcp-4.1.1-P1

server-duid "\000\001\000\001\037\006r\351x\347\321\350\031\006";

And our configuration file looks like this:

[root@topaz ~]# cat /etc/dhcp/dhcpd.conf
ddns-update-style none;
subnet 10.1.0.0 netmask 255.255.0.0 {
        default-lease-time 1200;
        max-lease-time 1200;
        option routers 10.1.1.1;
        option subnet-mask 255.255.0.0;
        option domain-name "local";
        option domain-name-servers 10.1.1.1;
        option broadcast-address 10.1.255.255;
        option interface-mtu 1500;
        group "local" {
                host topaz-eth0 {
                        hardware ethernet 78:E7:D1:E8:19:06;
                        option host-name "topaz";
                        fixed-address 10.1.1.1;
                        filename "pxelinux.0";
                        next-server 10.1.1.1;
                }
                host topaz-eth2 {
                        hardware ethernet 78:E7:D1:E8:19:0A;
                        option host-name "topaz";
                        fixed-address 10.1.1.1;
                        filename "pxelinux.0";
                        next-server 10.1.1.1;
                }
                host topaz-eth3 {
                        hardware ethernet 78:E7:D1:E8:19:0C;
                        option host-name "topaz";
                        fixed-address 10.1.1.1;
                        filename "pxelinux.0";
                        next-server 10.1.1.1;
                }
                host compute-2-0-eth0 {
                        hardware ethernet 00:23:7d:9f:6f:ca;
                        option host-name "compute-2-0";
                        fixed-address 10.1.255.254;
                        filename "pxelinux.0";
                        next-server 10.1.1.1;
                }
                host compute-2-0-eth1 {
                        hardware ethernet 00:23:7d:a0:20:e4;
                        option host-name "compute-2-0";
                        fixed-address 10.1.255.254;
                        filename "pxelinux.0";
                        next-server 10.1.1.1;
                }
                host compute-0-0-eth0 {
                        hardware ethernet 1c:c1:de:76:6f:40;
                        option host-name "compute-0-0";
                        fixed-address 10.1.255.253;
                        filename "pxelinux.0";
                        next-server 10.1.1.1;
                }
                host compute-0-0-eth1 {
                        hardware ethernet 1c:c1:de:76:6f:44;
                        option host-name "compute-0-0";
                        fixed-address 10.1.255.253;
                        filename "pxelinux.0";
                        next-server 10.1.1.1;
                }

                (goes on like this)
}

I don't understand why am I getting this error. Any help is appreciated.

Answer 1

It looks like you miss either range definition or a specific host entry for those boxes, matching their MACs. Unless your failing boxes are on the list that "goes on like this", they won't have any address assigned.

Answer 2

Your ISC dhcpd config file (posted June 2016) has no range statement. No leases are free because the server can't find an IP address, in a defined range, to hand out. You need a line like

range 10.1.42.0 10.1.49.255;

inside your subnet{} block. Any range inside that subnet, that nothing else is already using, will work. If you have multiple subnet blocks, each one needs a range definition inside it.

Answer 3

I just had a weird case of this error, I'll mention it here in case it helps someone.

My dhcp3 is binding on two adapters. I couldn't get DHCP to work on one of them. It turns out I had an old MAC address reservation in place for the client computer. Since this reservation was for the other subnet, DHCP could only find a pre-determined IP address to give that was outside the range for this adapter, and so gave this no free leases error.