docker-compose ELK container fails to start

Question

I am deploying a docker-compose stack of 5 applications on a single AWS EC2 host with 32GB RAM. This includes: 2 x Java containers, 2 x Rails containers, 1 x Elasticsearch/Logstash/Kibana (ELK) container (from https://hub.docker.com/r/sebp/elk/).

When I bring the stack up for the first time, all containers start. The ELK container takes about 3 minutes to start. The others come up straight away.

But the ELK containers exits after about 5 minutes. I can see from the logs that the elasticsearch service will not start. The log messages indicates a memory limitation error.

However, when I then tear everything down, and bring it up again, all the containers start straight away, including the ELK container, and everything remains stable. The issue only occurs the first time I start the stack on a new EC2 instance.

I can see from the docker stats that the ELK container is only using 2-3GB of the 32GB RAM available on the instance.

The ELK container is configured as follows:

elk:
  image: sebp/elk
  hostname: elk
  container_name: elk
  volumes:
    - ./pipeline/:/etc/logstash/conf.d/
  tty: true
  expose:
    - "12201/udp"
  network_mode: host
  ports:
    - "5601:5601"
    - "9200:9200"
    - "12201:12201"
  ulimits:
    nofile:
      soft: 65536
      hard: 65536

There are no dependencies between the containers on start up.

What is happening with elasticsearch when it first runs that cause the container to fail when starting?

Answer 1

The issue here was that the sebp/elb image contains a timeout parameter that was expiring before elasticsearch could start.

The is controlled by the ES_CONNECT_RETRY environment variable, which is set to 30 secs by default. Elasticsearch takes longer than this to start the first time, so when I set this to 300 secs, it worked. You can add this as an environment variable for the elk container in your docker-compose manifest.

elk:
  image: sebp/elk
  hostname: elk
  container_name: elk
  environment:
   - ES_CONNECT_RETRY=300

In addition to this, you also need to set the COMPOSE_HTTP_TIMEOUT environment variable before you run docker-compose, as otherwise, docker-compose will time out before elasticsearch can start. You should set the value to something greater than you set for ES_CONNECT_RETRY eg

COMPOSE_HTTP_TIMEOUT=360

Its possible that this is occurring because I am using a larger Ec2 instance (32GB RAM). It may not be an issue on smaller instances.