11

I have a basic-ish ELB v2 site. No clustering or anything yet. I'm pretty inexperienced with AWS.

My stack is nginx/uwsgi/django + some other services.

I was wondering if anyone had thoughts about making a "sorry, the website is currently down..."-style page (custom text I can update planned downtime is a bonus!) whenever there is downtime for whatever reason, and the health of the instance is red. It doesn't seem that Amazon offers this capability - am I missing anything? Is there a way to create a separate, super-tiny instance that is ONLY served if the main one is red, or something?

Thanks!

4 Answers 4

26

The simple and cool solution here is to put your ELB behind CloudFront.

If the origin server (the ELB in this case) throws a 5XX error (or 4XX if you like), CloudFront can return a custom error page, which you can configure CloudFront to fetch from an S3 bucket by creating a second origin pointing to the bucket and creating a cache behavior routing (e.g.) /errors/static/* to the bucket.

This works better than Route 53 failover for an important reason... a fatal flaw, if you will... browsers are terrible about caching DNS lookups for far longer than you expect. The DNS TTL isn't relevant.

Essentially, once a browser has a DNS entry in hand, it just keeps trying to use it... typically, until all browser windows are closed.

So if your site goes down for a visitor who was already on the site, they unlikely to see the alternate site.

Worse, if a visitor hits your site for the first time while it's down, they'll "stick" on the maintenance page until they close all browser windows.

If you use failover DNS, that's really only good if the failover target is still your application, maybe just further away.

You can turn CloudFront's caching off if you don't need it.

You can also configure CloudFront's error caching TTL to a nonzero value if you want it to quit hammering your site while it's down and trying to recover. For a given page that throws an error, it will keep showing the error page and not bother your server with more requests for that page until the Error CachingTTL expires.

2
  • Interesting. AWS don't mention this drawback in their documentation. This underlines how important testing is.
    – Tim
    Jan 25, 2017 at 1:24
  • Well @Tim, AWS recommends doing rolling updates. They didn't have their "Docker Service" that they offer now, so EBS it was for our Docker app. Only needed one, though. Jul 28, 2017 at 21:52
6

Use Route53 DNS and failover routing. You should be able to get up an S3 bucket hosting a single page static website. I don't think you can do it with just ELB.

Amazon have a blog post that tells you how to do it here.

Update: as Michael says, there's a drawback around browser DNS caching, see his answer for more information. Route 53 is probably a simpler option than CloudFront, but CF has other advantages, performance and in some use cases can lower costs.

5
  • I do say +1 here... this is a good solution, but it does seem to have an achilles heel, making it less viable in some use cases. Jan 25, 2017 at 1:14
  • @Michael-sqlbot what's the drawback of this approach? Browser DNS caching time?
    – Tim
    Jan 25, 2017 at 1:22
  • Yes, exactly. People "sticking" to the error page is something I find unnerving. Jan 25, 2017 at 5:09
  • Here is an updated blog post from AWS with an newer simpler method for ELB failover routing. aws.amazon.com/blogs/aws/… See my post below for more details.
    – AstroTom
    Jul 24, 2018 at 12:43
  • Here's an updated AWS support page (from Oct 2020) for using Route 53 health checks for DNS failover
    – Matt
    Sep 15, 2022 at 16:21
2

Several solutions have been mentioned already, including CloudFront and Route53. CloudFront is an excellent solution and in my experience hasn't slowed things down at all, but it does bring added cost. And Route53 has the DNS caching issues already mentioned.

Until ALB supports custom error pages out of the box (which may or may not happen), there's potentially a new solution after the recent announcement of ALB fixed responses, but it's not point and click: you could set up a Lambda function that temporarily adds a rule to your load balancer, providing a fixed response with your 'error page' contents.

Aside from writing the Lambda, you'll need to find a way to trigger it 'on' and 'off', which could possibly be via a Route53 health check or load balancer target group health check (probably via CloudWatch alarm -> SNS -> Lambda).

It's not exactly simple, but would probably work well once set up!

0

As written by @Tim and @Micheal, you have your choice of using Route53 DNS and failover routing, or CloudFront with custom error pages. Both methods have their pros and cons.

If you are not already using Cloudfront, I think Route53 is a simpler solution. See the updated blog post from AWS (which now includes a simpler method for ELB integration).

CloudFront is much more complicated to setup, and it will take about 15 minutes for every update. Cloudfront also caches (as to be expected) so it is not clear if that will be any slower to respond, than the DNS cache issues with Route 53.

Note that if your ELB website only responds to SSL then you cannot use the simple S3 solution as described in the AWS blog3. In that case you will have to add Cloudfront in front the the S3 bucket to add the SSL, this making the DNS failure routing solution more complicated.

1
  • 1
    That blog post does not offer a clean solution -- it has exactly the problem I mentioned in my post and discussed with Tim in comments. It is perfectly viable when you have multiple deployments that can serve your requests but wholly unsuited to an error page because of the way browsers cache DNS lookups. The content of the AWS post unfortunately fails to take this reality into consideration. DNS failover does not reliably "fail back" from the end-user's perspective. CloudFront also has entirely separate cache settings for error responses. Sep 28, 2018 at 7:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .