Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

This problem from what i can tell is isolated to PowerDNS. The servers are running two packages pdns-static-3.0.1-1.i386.rpm and pdns-recursor-3.3-1.i386.rpm on the most recent version of Amazon Linux.

The amazon ec2 loadbalancers are assigned a CNAME with multiple hosts. Below is an example of the actual behavior. Notice how the hosts are always in the same order.

[root@localhost ~]# host is an alias for has address has address bbb.bbb.bbb.bbb
[root@localhost ~]# host is an alias for has address has address bbb.bbb.bbb.bbb
[root@localhost ~]# host is an alias for has address has address bbb.bbb.bbb.bbb

Expected behavior is round robin for the hosts

[root@localhost ~]# host is an alias for has address has address bbb.bbb.bbb.bbb
[root@localhost ~]# host is an alias for has address bbb.bbb.bbb.bbb has address
[root@localhost ~]# host is an alias for has address has address bbb.bbb.bbb.bbb

The addresses eventually do swap but it seems to be on a 30 minute cache timer changing the TTL of the record doesn't appear to affect anything. It appears as though the resolver has a cache of the response. This adversely affects my application because all of the load is only being sent to one of the loadbalancers (Availability Zones) so if I have servers in two zones then only one zone is under load at a time.

Do you know how I can fix this so that each time the host is resolved the order of the addresses is alternating.


;  DiG 9.7.6-P1-RedHat-9.7.6-1.P1.18.amzn1
;; global options: +cmd
;; Got answer:
;; HEADER opcode: QUERY, status: NOERROR, id: 54610
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:           IN      A

;; ANSWER SECTION:    100     IN      CNAME                                                                                                         3 IN A 3 IN A bbb.bbb.bbb.bbb

;; Query time: 0 msec
;; SERVER: ccc.ccc.ccc.ccc#53(ccc.ccc.ccc.ccc)
;; WHEN: Mon Jul  2 15:09:27 2012
;; MSG SIZE  rcvd: 130

Recursor config

local-port=530                                                                  # Port should be changed to 530 because its not good to run on the same port as dns server
forward-zones=domain.local=LOCALIP,                         # Forward the two zones we care about back to the local dns server,compute-1.internal=  # Forward queries for amazons domains to the resolver for amazon


add the following lines to recursor.conf


add the following line to pdns.conf

share|improve this question
regarding the bounty: you can't have both a detailed canonical answer, and a specific problem solved. Furthermore, configuring the PowerDNS Recursor is the solution for this specific problem. – Habbie Jul 3 '12 at 7:27
up vote 2 down vote accepted

The PowerDNS Recursor caches at two levels.

It caches responses from authoritative servers for up to the TTL specified in the response it got (limited by max-cache-ttl but never exceeding the TTL it got from an auth).

Additionally, when a response packet from the recursor to a client (your clients that are generating load) is generated and sent, this packet is cached as a whole, so that the same question can be answered extremely quickly (without any parsing) if it comes in again. This is called the packetcache.

Shuffling happens in between these two levels. This means that your results are in fact shuffled, but their shuffle order is kept stable by the packetcache (for up to an hour, by default). If you want per-response shuffle, set 'disable-packetcache' or 'packetcache-ttl=0'.

share|improve this answer
disable-packetcache actually disables the caching code; packetcache-ttl=0 pushes the same check slightly deeper down. Thus, disable-packetcache should be slightly faster. Specifying both does not hurt :) – Habbie Jul 1 '12 at 20:11
I added both those to the configuration for the recursor and still get no luck. I updated the original post with a dig and the configuration i have. – bwight Jul 2 '12 at 15:20
I tested with recursor 3.3 and your exact config. The A records for an 'A' query shuffle for me. Can you share a name that fails for you? Obviously is fake :) – Habbie Jul 2 '12 at 17:40
can you try this and post the output? for d in {1..100} ; do echo $(dig +short a @ -p 530 | grep '^[0-9]') ; done | sort -u | wc -l – Habbie Jul 2 '12 at 19:21
Its solved, i'll award you the bounty points as soon as i can. Thanks for your help. – bwight Jul 3 '12 at 14:11

Not necessarily a "fix" - but do you need to use the CNAME from your application rather than directly querying the underlying A record? Presumably the CNAME => A record mapping doesn't change that often.

Sometimes the simplest dumb fixes are actually enough, and avoid having to solve all the worlds' problems just to get the results you need!

share|improve this answer
I do actually need to use the CNAME instead of the A record. The CNAME points to a list of load balancers that are used for redundancy within the amazon cloud. The A records are changed as more load is put onto the system because amazon will provision more powerfull load balancers as my needs inrease. – bwight Jul 2 '12 at 17:24
Oh well... so much for that plan then. – Bron Gondwana Jul 2 '12 at 18:42

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.