I have two Ubuntu EC2 instances running Scrapy, both using scrapy-redis to control dupes. I am trying to use an ElastiCache Redis as the scrapy-redis store. I have the exact same set up in a different AWS account, and it works fine. I am trying to replicate the setup in a new account, and it will not work. The two EC2 instances and the ElastiCache instance are all in the same VPC, and I followed this article for allowing security group access from the Scrapy instances to ElastiCache. You can see the setup below:
The ElastiCache instance is in the selected security group; the other two groups are the two Scrapy EC2 instances.
The problem is isolated to scrapy-redis, because I can ssh into both of the Scrapy boxes and connect to the ElastiCache instance with redis-cli without issue. I have the correct settings for scrapy-redis in my settings.py file, like so:
REDIS_HOST = 'redis.xxxx.ng.0001.use2.cache.amazonaws.com'
REDIS_PORT = 6379
The error I get from scrapy-redis is
File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 486, in connect
raise TimeoutError("Timeout connecting to server")
redis.exceptions.TimeoutError: Timeout connecting to server
I'm grasping at straws here, but I created my Scrapy instances from an AMI I created in the original account, where the setup is working correctly, so maybe that's the problem? Any help would be appreciated.