As I am the skeptic's CTO, I'd like to throw some color around the discussion.
As Gary has mentioned, the application needs to be able to publish a very large manifest. The manifest will be partitioned into a few (30-100) groups. Each key will average about 55 bytes, but could be much larger.
Whichever product we choose needs to support the following:
1) Full redundancy and load balancing
2) High transaction volume (15k-20k reads/sec) with <500ms response time.
The nice to haves are:
1) Hierarchical structure
2) Record expiration
3) Self-describing topology
DNS popped into mind only after thinking about using UDP instead of TCP to retrieve data from an in-memory distributed cache. Naturally, DNS is one of the old and largest DNS applications.
DNS is known to support zone files with very large amounts of records. For example .com. is a zone file afterall (albeit distributed across many servers globally). It is also known to support very high levels of traffic.
We have run DNS through some preliminary tests. We loaded up a single zone file with 10M TXT records with a representative amount of data. From a different server on the same LAN, we then ran tests of 300,000 queries in a multi-threaded fashion and got about 5,000 requests per second. The server and client barely flinched during the test. We are either running into bottlenecks in the testing app itself or in the network stack on the client.
I am intrigued by DNS because it supports everything I want natively, and has so for many years. The features I like are:
Zone Delegation - we can define which server(s) handle particular partitions. For example 1.mycompany.local is handled by servers 10.1.1.1 and 10.1.1.2.
Redundancy - DNS was built with resiliency and redundancy in mind. It can also be easily load balanced.
Performance - Proven to support high request volumes
With all that said, DNS does sound like a bizarre tool to use as a cache. If it does end up making it through the selection phase, we would absolutely use it only on the internal LAN, and it would not be on the same DNS as our any other internal or external DNS systems. One side note is that we might want to share the data we're storing with 3rd party partners. DNS is a well known entity that anyone could easily query to receive zone transfers from.
Thank you for your continued feedback.