Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have just setup a load balancer with AWS for a few instances as search engine crawlers were bringing down the site (it has millions of pages). Parts of the site allow you to login so I selected:

Enable Application Generated Cookie Stickiness

and everything works fine.

I now wonder how this will effect my SEO and the crawlers. As I selected sticky load balancing does this mean that a crawler will be stuck on one server and therefore defeat the point in the load balancer?

Any recommendations will be appreciated.

share|improve this question

2 Answers 2

up vote 2 down vote accepted

Since ELB uses cookie-oriented stickiness, a crawler shouldn't ever get stuck to a single instance. However, since it's obviously a crap crawler you're dealing with here (since it's DoSing your site), it might be stupid enough to hold on to cookies it gets, in which case you're going to get hosed.

Personally, I'd recommend getting rid of the session affinity and fixing your app so it doesn't need it -- there are plenty of ways to do shared session storage.

share|improve this answer
    
Thanks for your answer I have solved the problem by storing the sessions on an external memcached server. –  John Wheal Jul 2 '12 at 20:57

Application Generated Cookie Stickiness means your app has to drop a cookie that the load balancer looks for to assign affinity. Without the cookie, it's going to be random what server the request is assigned to.

Unless you've modified your app to do drop the affinity cookie, you probably want to check "Enable Load Balancer Generated Cookie Stickiness" instead. This way the ELB itself drops a cookie for the duration value you set and keeps the user on the same server until the cookie expires.

share|improve this answer
    
Thanks for your answer I have solved the problem by storing the sessions on an external memcached server. –  John Wheal Jul 2 '12 at 20:57
    
That's a nice, sneaky way of doing it... detect the spiders and make sure you don't give them back a sticky cookie... –  womble Jul 3 '12 at 10:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.