Trying to setup a Graphite/Carbon cluster. I have an elastic load balancer that directs traffic between two nodes in my cluster, each with one web app, relay, and cache.
In this example, I sent 1000 counts for Metric1 to the cluster.
Here's a diagram:
As seen above in the diagram, each server holds approximately half of the actual metric count. When queried via the web app, it only returns one half of the actual count. According to this fantastic post, this is expected behavior, because the web app returns the first result it sees. This implies (and is documented) that only complete counts should be stored on nodes (in my example, one or both of the nodes should have 1000.)
So my issue appears to be the improper sharding and replication of the count. In my example above, when a new count comes in from the web, it can be redirected to either NodeA or NodeB. I had assumed that counts can enter the cluster via any relay. To test this assumption, I removed the load balancer from the cluster, and directed all incoming counts to NodeA's relay. This worked: the full count appeared on one node, then replicated to the second, and the full count was returned correctly from the web app.
carbon-relay appears to act as an application-level load balancer. This is fine, however I'm concerned that when inbound traffic becomes too great, using a single
carbon-relay as a load balancer will become a bottleneck, and single point of failure. I'd much prefer to use an actual load balancer to evenly distribute incoming traffic across the cluster's relays. However,
carbon-relay doesn't seem to play nice, hence the problem illustrated above.
- Why did the relay cluster split Metric1 between the two caches in the above scenario? (When a load balancer distributed the input into different relays?)
- Can I use an elastic load balancer in front of my Graphite/Carbon cluster? Have I misconfigured my cluster for this purpose?
- If I can't, should I put my primary
carbon-relayon its own box to function as a load balancer?