0

I have two Ubuntu EC2 instances running Scrapy, both using scrapy-redis to control dupes. I am trying to use an ElastiCache Redis as the scrapy-redis store. I have the exact same set up in a different AWS account, and it works fine. I am trying to replicate the setup in a new account, and it will not work. The two EC2 instances and the ElastiCache instance are all in the same VPC, and I followed this article for allowing security group access from the Scrapy instances to ElastiCache. You can see the setup below:

enter image description here

The ElastiCache instance is in the selected security group; the other two groups are the two Scrapy EC2 instances.

The problem is isolated to scrapy-redis, because I can ssh into both of the Scrapy boxes and connect to the ElastiCache instance with redis-cli without issue. I have the correct settings for scrapy-redis in my settings.py file, like so:

REDIS_HOST = 'redis.xxxx.ng.0001.use2.cache.amazonaws.com'
REDIS_PORT = 6379

The error I get from scrapy-redis is

File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 486, in connect
    raise TimeoutError("Timeout connecting to server")
redis.exceptions.TimeoutError: Timeout connecting to server

I'm grasping at straws here, but I created my Scrapy instances from an AMI I created in the original account, where the setup is working correctly, so maybe that's the problem? Any help would be appreciated.

0

I'm not strong enough in Python to understand exactly why, but we realized that the issue had to do with which directory we were in when starting ScrapyD. Apparently you need to be in the same directory as settings.py, not above or below it, when you fire up ScrapyD, otherwise the Redis connections aren't recognized. This solves the issue, but if anybody knows the reason why, feel free to add on.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.