AWS Elastic Load Balancer: Forwarding Requests to all Instances

Question

Using Terraform to create Autoscaling Group (with 2 instances) and Elastic Load Balancer (ELB) in AWS.

The instances are running simple http-echo server written in Go on Port 3000.

When the DNS name of Load Balancer is visited through browser, the request is logged in both the instances instead of one. Expected behavior should be sending request to one of the instances.

Log of Instance1:

2019/01/23 05:03:53 <DNS Name of LB> 
10.0.21.217:31904 "GET /favicon.ico HTTP/1.1" 200 58 
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/71.0.3578.98 Safari/537.36" 9.018µs

Log of Instance2:

2019/01/23 05:03:53 <DNS Name of LB> 
10.0.21.217:47620 "GET / HTTP/1.1" 200 58 
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/71.0.3578.98 Safari/537.36" 9.074µs

But when request is sent through curl to the same address, the load balancer works as expected and only sends request to one of the instance, and also cycles through the instances on repeated requests. This is desired behavior.

Log of curl request:

2019/01/23 05:43:15 <DNS Name of LB> 10.0.21.217:49364 
"GET / HTTP/1.1" 200 58 "curl/7.47.0" 8.397µs

Both instances are "healthy" and respond to Load Balancer's health requests.

Configuration of Load Balancer is below:

resource "aws_elb" "go_app" {
  name               = "terraform-asg-go-app"
  security_groups    = ["${aws_security_group.elastic_lb.id}"]
  subnets            = ["${aws_subnet.public.*.id}"]

  listener {
    lb_port           = 80
    lb_protocol       = "http"
    instance_port     = 3000
    instance_protocol = "http"
  }

  cross_zone_load_balancing   = true
  idle_timeout                = 400
  connection_draining         = true
  connection_draining_timeout = 400
}

The security group the of the Load Balancer had the following configuration:

resource "aws_security_group" "elastic_lb" {
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

While the security group of the instance had this configuration:

resource "aws_security_group" "go_app" {    
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    from_port   = 3000 
    to_port     = 3000
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Answer 1

This is expected behavior. Look at the paths being fetched in both instances -

Instance1

10.0.21.217:31904 "GET /favicon.ico HTTP/1.1" 200 58 

Instance2

10.0.21.217:47620 "GET / HTTP/1.1" 200 58

The second one fetches the page itself and the first request fetches the favicon of the page. When you deploy a web app, the browser makes separate requests to fetch all assets related to your web app, favicon being one such asset. Other assets could be css/js files, images etc. All referenced assets are fetched separately and then rendered on the page.

When the browser makes multiple requests, they hit the load balancer which then distributes the request among instances behind it, generally in a round robin fashion which is why you're seeing separate requests on the two instances.

curl on the other hand simply fetches the html page and does not make any extra requests.

Answer 2

Carefully check the posted logs above again. Your Chrome makes two requests:

1. GET / HTTP/1.1
2. GET /favicon.ico HTTP/1.1

And because the Load balancer is load balancing it sends one request to each node as expected.

It's the standard behaviour of modern desktop browsers to request /favicon.ico from all websites so that they can display the site icon in the bookmarks, tabs, etc. On the other hand curl doesn't do this so you see only one request.

Hope that helps :)