I've recently moved an application to Elastic Beanstalk. I'm currently in the process of setting up log files to be archived. One of the business requirements is x days of log retention, which is pretty sensible of course. Since log files are stored on the instance, and the instance data is not persisted when terminating/re-creating the instance, the logs need to be stored elsewhere.

Elastic Beanstalk offers the option of "S3 log storage", which seems to upload rotated log files to S3, which would cover most of my needs. The one exception would be when an instance is terminated/re-created: how can I easily assure that before termination, the not-yet-rotated log files are uploaded to S3?

  • 1
    You could use Cloudwatch logs. Install an agent (it may be there already), configure it all, and it will push logs into CW Logs which you can then view or extract to any other system for analysis. – Tim Nov 23 '18 at 18:33
  • Yes, I've been leaning towards just using Cloudwatch instead, much simpler. Still, you'd think that the S3 backup option could be made workable as well, am I the first one to think about losing logs when terminating instances? – Michel Nov 24 '18 at 19:16
  • CloudWatch logs is made to solve your problem. – Tim Nov 24 '18 at 23:35

Elastic Beanstalk logs backup to S3 is done from time to time as a batch upload by logrotate which means, as you rightly pointed out, that the latest logs may not be in S3 when the instance terminates.

You can create a shutdown script (e.g. here) that will upload all log files to S3 when the instance is being gracefully shut down. It may not be executed if the instance crashes or fails in some abrupt way but that's a pretty rare event.

However Elastic Beanstalk supports CloudWatch Log Streaming which feeds the logs to CloudWatch Logs in real time, as soon as they are generated. That means even in case of a shutdown you'll still have the latest logs in CloudWatch Logs up to the point where the awslogs agent gets terminated.

Have a look here how to set it up.

It's also easier to search CloudWatch Logs than S3 logs, you can watch for a particular keyword (e.g. "Error") and action on it (e.g. send email or run Lambda function), etc.

You can probably have both S3 and CloudWatch Logs enabled in your Elastic Beanstalk application at the same time.

Hope that helps :)

  • Thanks. The shutdown script was something I considered, yes, but I hoped that there might be a canned solution. It seems like log streaming would be the better choice, as it's indeed easier to access/search through old log files. – Michel Nov 26 '18 at 7:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.