Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have Graphite setup on three instances on EC2:

  • carbon-relay - relay1.graphite.prod.example.ec2
  • carbon-cache + webapp - cache3.graphite.prod.example.ec2
  • carbon-cache + webapp - cache4.graphite.prod.example.ec2

The relay is working perfectly with consistent-hashing. The problem is the two web servers are not communicating with each other, so I only see the metrics from one server.

I spent a lot of time looking at https://answers.launchpad.net/graphite/+question/114206 and I can't figure out what I have setup incorrectly. I can run a wget from cache3 against cache4, get data back and see it in the Apache logs. So I don't think it's a firewall issue. I tried enabling suppressError = False in remote_storage.py and turned on DEBUG in local_settings.py, but I don't see any errors in Firebug.

cache3 - local_settings.py

CLUSTER_SERVERS = [ 'cache4.graphite.prod.example.ec2', 'localhost' ]

cache4 - local_settings.py

CLUSTER_SERVERS = [ 'cache3.graphite.prod.example.ec2', 'localhost' ]

I have tried using IP addresses as well and that had no impact.

I did a little more debugging and modified storage.py to directly hard code my remote hosts:

STORE = Store(settings.DATA_DIRS, remote_hosts=["cache4.graphite.prod.example.ec2", "127.0.0.1"])

That worked. So, somehow my CLUSTER_SERVERS value isn't getting pulled in from local_settings.py correctly.

Any suggestions?

share|improve this question

1 Answer 1

up vote 0 down vote accepted

Turns out the permissions on local_settings.py were too restrictive and Apache was unable to read it:

-rw------- 1 root root  4006 May  4 13:40 local_settings.py

Fixing the permissions to 644 (instead of 600) resolved the problem.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.