Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This problem from what i can tell is isolated to PowerDNS. The servers are running two packages pdns-static-3.0.1-1.i386.rpm and pdns-recursor-3.3-1.i386.rpm on the most recent version of Amazon Linux.

The amazon ec2 loadbalancers are assigned a CNAME with multiple hosts. Below is an example of the actual behavior. Notice how the hosts are always in the same order.

[root@localhost ~]# host cache.domain.com
cache.domain.com is an alias for xxxxx.us-east-1.elb.amazonaws.com.
xxxxx.us-east-1.elb.amazonaws.com has address aaa.aaa.aaa.aaa
xxxxx.us-east-1.elb.amazonaws.com has address bbb.bbb.bbb.bbb
[root@localhost ~]# host cache.domain.com
cache.domain.com is an alias for xxxxx.us-east-1.elb.amazonaws.com.
xxxxx.us-east-1.elb.amazonaws.com has address aaa.aaa.aaa.aaa
xxxxx.us-east-1.elb.amazonaws.com has address bbb.bbb.bbb.bbb
[root@localhost ~]# host cache.domain.com
cache.domain.com is an alias for xxxxx.us-east-1.elb.amazonaws.com.
xxxxx.us-east-1.elb.amazonaws.com has address aaa.aaa.aaa.aaa
xxxxx.us-east-1.elb.amazonaws.com has address bbb.bbb.bbb.bbb

Expected behavior is round robin for the hosts

[root@localhost ~]# host cache.domain.com
cache.domain.com is an alias for xxxxx.us-east-1.elb.amazonaws.com.
xxxxx.us-east-1.elb.amazonaws.com has address aaa.aaa.aaa.aaa
xxxxx.us-east-1.elb.amazonaws.com has address bbb.bbb.bbb.bbb
[root@localhost ~]# host cache.domain.com
cache.domain.com is an alias for xxxxx.us-east-1.elb.amazonaws.com.
xxxxx.us-east-1.elb.amazonaws.com has address bbb.bbb.bbb.bbb
xxxxx.us-east-1.elb.amazonaws.com has address aaa.aaa.aaa.aaa
[root@localhost ~]# host cache.domain.com
cache.domain.com is an alias for xxxxx.us-east-1.elb.amazonaws.com.
xxxxx.us-east-1.elb.amazonaws.com has address aaa.aaa.aaa.aaa
xxxxx.us-east-1.elb.amazonaws.com has address bbb.bbb.bbb.bbb

The addresses eventually do swap but it seems to be on a 30 minute cache timer changing the TTL of the record doesn't appear to affect anything. It appears as though the resolver has a cache of the response. This adversely affects my application because all of the load is only being sent to one of the loadbalancers (Availability Zones) so if I have servers in two zones then only one zone is under load at a time.

Do you know how I can fix this so that each time the host is resolved the order of the addresses is alternating.


DIG OUTPUT


;  DiG 9.7.6-P1-RedHat-9.7.6-1.P1.18.amzn1  cache.domain.com
;; global options: +cmd
;; Got answer:
;; HEADER opcode: QUERY, status: NOERROR, id: 54610
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
cache.domain.com.           IN      A

;; ANSWER SECTION:
cache.domain.com.    100     IN      CNAME   xxxxx.us-east-1.elb.amazonaws.com.                                                                                                                 
xxxxx.us-east-1.elb.amazonaws.com. 3 IN A aaa.aaa.aaa.aaa
xxxxx.us-east-1.elb.amazonaws.com. 3 IN A bbb.bbb.bbb.bbb

;; Query time: 0 msec
;; SERVER: ccc.ccc.ccc.ccc#53(ccc.ccc.ccc.ccc)
;; WHEN: Mon Jul  2 15:09:27 2012
;; MSG SIZE  rcvd: 130

Recursor config

allow-from=0.0.0.0/0
dont-query=
local-address=127.0.0.1
local-port=530                                                                  # Port should be changed to 530 because its not good to run on the same port as dns server
quiet=yes
setgid=pdns
setuid=pdns
disable-packetcache=
packetcache-ttl=0
forward-zones=domain.local=LOCALIP,domain.cloud=LOCALIP                         # Forward the two zones we care about back to the local dns server
forward-zones-recurse=amazonaws.com=172.16.0.23,compute-1.internal=172.16.0.23  # Forward queries for amazons domains to the resolver for amazon

SOLUTION

add the following lines to recursor.conf

disable-packetcache=
packetcache-ttl=0

add the following line to pdns.conf

recursive-cache-ttl=0
share|improve this question
    
regarding the bounty: you can't have both a detailed canonical answer, and a specific problem solved. Furthermore, configuring the PowerDNS Recursor is the solution for this specific problem. –  Habbie Jul 3 '12 at 7:27

2 Answers 2

up vote 2 down vote accepted
+50

The PowerDNS Recursor caches at two levels.

It caches responses from authoritative servers for up to the TTL specified in the response it got (limited by max-cache-ttl but never exceeding the TTL it got from an auth).

Additionally, when a response packet from the recursor to a client (your clients that are generating load) is generated and sent, this packet is cached as a whole, so that the same question can be answered extremely quickly (without any parsing) if it comes in again. This is called the packetcache.

Shuffling happens in between these two levels. This means that your results are in fact shuffled, but their shuffle order is kept stable by the packetcache (for up to an hour, by default). If you want per-response shuffle, set 'disable-packetcache' or 'packetcache-ttl=0'.

share|improve this answer
    
disable-packetcache actually disables the caching code; packetcache-ttl=0 pushes the same check slightly deeper down. Thus, disable-packetcache should be slightly faster. Specifying both does not hurt :) –  Habbie Jul 1 '12 at 20:11
    
I added both those to the configuration for the recursor and still get no luck. I updated the original post with a dig and the configuration i have. –  bwight Jul 2 '12 at 15:20
1  
I tested with recursor 3.3 and your exact config. The A records for an 'A www.google.com' query shuffle for me. Can you share a name that fails for you? Obviously cache.domain.com is fake :) –  Habbie Jul 2 '12 at 17:40
1  
can you try this and post the output? for d in {1..100} ; do echo $(dig +short a www.google.com @127.0.0.1 -p 530 | grep '^[0-9]') ; done | sort -u | wc -l –  Habbie Jul 2 '12 at 19:21
1  
Its solved, i'll award you the bounty points as soon as i can. Thanks for your help. –  bwight Jul 3 '12 at 14:11

Not necessarily a "fix" - but do you need to use the CNAME from your application rather than directly querying the underlying A record? Presumably the CNAME => A record mapping doesn't change that often.

Sometimes the simplest dumb fixes are actually enough, and avoid having to solve all the worlds' problems just to get the results you need!

share|improve this answer
    
I do actually need to use the CNAME instead of the A record. The CNAME points to a list of load balancers that are used for redundancy within the amazon cloud. The A records are changed as more load is put onto the system because amazon will provision more powerfull load balancers as my needs inrease. –  bwight Jul 2 '12 at 17:24
    
Oh well... so much for that plan then. –  Bron Gondwana Jul 2 '12 at 18:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.