AWS Elastic Beanstalk random spike in NetworkIn traffic once to twice a day

Question

Unusual traffic on my Elastic Beanstalk application. NetworkIn, NetworkOut both spike once to twice a day at random times. Sometimes it crashes my load balancer and I have to go in and manually kill the SEVERE instance to get it to fire another one up.

I have downloaded the logs and looked at nginx.access. The activity looks normal.

I have looked for patterns... Time-wise there is not much of a pattern. Can happen between 8 and 10am, or later in the afternoon, but only during working hours ususally (no spikes at 2am for example).

I've considered someone running an automated job slamming our APIs, but did not see that in ngix.access log.

I've considered someone trying to upload a large file, but wouldn't know how to see that in a log.

I am a Software Engineer that has somewhat inherited the role of AWS admin until we find more help so my experience is limited.

So the question: What else could I be looking at in AWS to try an identify the reason for these spikes?

Any advice that can be offered would be helpful. Thanks.

EDIT: FOLLOW UP

I did manage to "fix" the crashes. Hopefully this may help someone.

Adjusting auto scaling properties.

Min instances to 2 instead of 1.

Scaling triggers > Period > changed to 1 minute instead of 5.

Same for "Breach Duration" - 1 min instead of 5.

No more crashing since I made these changes.

Answer 1

For the NetworkIn traffic:

I've just been looking for a similar answer and i've discovered that elastic beanstalk does count the initial installing of the application on an ec2 instance in it's NetworkIn stat.

I set my instance to scale by 1 manually and watched the traffic - my application package is roughly 300mb - and minutes later it showed up in the monitored traffic.

So maybe your NetworkOut traffic is triggering your auto scaling policy which explains the NetworkIn spike.