I have completed my AWS ELB architecture for our website and successfully created the Launch Configuration and Target Groups for which instances are created behind the Load Balancer.
My configuration is as follows :
HTTPS through port 443
Path : /healthy.html
Healthy threshold : 10
Unhealthy threshold : 2
Timeout : 5
Interval : 30
Desired : 2
Min : 2
Max : 3
No scaling policy for now.
Application load balancer that listens on HTTP:80 and HTTPS:443 that both forward to the mentioned target group.
I have one remaining issue where my instances keep being terminated because of failed health checks. Although, querying the path mentioned in health check configuration works all the time and the instance seems to have no issue at all.
Every now and then, an instance becomes unhealthy and then a new one is instantiated back.
I have read all the documentation regarding health checks and I understand the theory behind it but I don't understand why hosts keep failing health checks at times while there are near to no traffic or load on the application. The application works great apart from the mentioned fact that instances keep failing health check... But I cannot figure out why.
I have a hard time investigating the issue as all configuration seem adequate.
It seems the aws s3 sync command executed every minute in my crontab stalls...
20343 bitnami 20 0 192392 48884 9696 R 34.3 2.4 1:06.32 /usr/bin/python3 /home/bitnami/.local/bin/aws s3 sync --delete /opt/bitnami/apps/wordpress/htdocs s3://nutriti-code
20351 bitnami 20 0 192108 48608 9680 R 32.7 2.4 0:37.24 /usr/bin/python3 /home/bitnami/.local/bin/aws s3 sync --delete /opt/bitnami/apps/wordpress/htdocs s3://nutriti-code
20375 bitnami 20 0 339360 48748 9728 R 32.7 2.4 0:10.88 /usr/bin/python3 /home/bitnami/.local/bin/aws s3 sync --delete /opt/bitnami/apps/wordpress/htdocs s3://nutriti-code
(This is from my writer node but I suspect the same happens on my reader nodes that get terminated.)
The aws s3 sync command runs for minutes until the server fails health check.
Attached are all timestamps for health check failures : https://photos.app.goo.gl/sdU1yzL4r5q8q5hz5
Some insights would be greatly appreciated!