1

I'm Having some wierd issues trying to automate the deployment of docker containers using nginx's upstream directive. For some reason when I manually do the following it works but when I try to automate it I do not get consistent results.

Basically I pull the latest image stop, remove the 1 of 2 running containers Then start the container Then update nginx upstream conf file Reload nginx config. Repeat for 2nd container.

Seems simple but for some reason I can' get this to work in a bash script all the time. I'm using seige to load test the application.

siege -d1 -t75S -c25 http://192.168.49.4:8087

My Script to automate the deployment The latest images is pulled prior to running the script.

#! /bin/bash

appname=appnamexyz;
appport=8000;
host_ip=192.168.49.4;
registry=192.168.254.96;

echo "upstream api_servers {${nl}
  server $(docker port ${appname}-1 ${appport}) max_fails=2 fail_timeout=1s;${nl}
  server $(docker port ${appname}-2 ${appport}) max_fails=2 fail_timeout=1s;${nl}
}" > /etc/nginx/conf.d/api_upstream.conf

sed -i "s/server $(docker port $appname-1 $appport)/server xxx/g" /etc/nginx/conf.d/api_upstream.conf

docker stop $appname-1
docker rm -f $appname-1
docker run -d --name $appname-1 -p $host_ip::$appport $registry:5000/$appname -APIKey=e5e1c4b8e46d563c3 

sed -i "s/server xxx/server $(docker port $appname-1 $appport)/g" /etc/nginx/conf.d/api_upstream.conf
nginx -s reload



sleep 15s

sed -i "s/server $(docker port $appname-2 $appport)/server xxx/g" /etc/nginx/conf.d/api_upstream.conf

docker stop $appname-2
docker rm -f $appname-2
docker run -d --name $appname-2 -p $host_ip::$appport $registry:5000/$appname -APIKey=e5e1c4b8e46d563c350b7

sed -i "s/server xxx/server $(docker port $appname-2 $appport)/g" /etc/nginx/conf.d/api_upstream.conf
nginx -s reload
cat /etc/nginx/conf.d/api_upstream.conf

The application will return 200s then while restarting the second container I'll receive 500,502 errors. Here is my conf file for the app.

server {
  listen 8087;
  server_name 192.168.49.4;

  location / {
    proxy_pass http://api_servers;
    proxy_next_upstream     error timeout invalid_header http_500 http_502 http_504;
    proxy_connect_timeout   1;
  }
}

What should I do to remediate this?

  • I see a typo in the command line for starting the 2nd container: $ap pname should become $appname – Bernard Rosset Nov 4 '14 at 7:53
0

While 502 corresponds to backend timout, 500 is a server error, indicating a misconfiguration. You might wish to fix the following on first hand:

Before the sleep, you actually replaced the first line of api_upstream.conf with

sed -i "s/server xxx/server $(docker port $appname-1 $appport)/g" /etc/nginx/conf.d/api_upstream.conf

(You even do it twice... purpose? Double copy-paste?)

But what you have not done yet at that time is replacing the second line, which remains being

server $(docker port ${appname}-2 ${appport}) max_fails=2 fail_timeout=1s;${nl}

I suppose $appname and $appport are empty, making the subshell fail/being empty, meaning that this line is actually interpreted as

server max_fails=2 fail_timeout=1s;

When load-balancing the requests with the default round-robin, nginx might face a problem there...

You should ensure that the api_upstream.conf always contains valid data as soon as it is used. You may add servers in there later, but letting a partially written placeholder will won't produce any good.

  • Your statement about 5xx errors is not true. It only means this is a server side error. Nginx will return 504 if it timed out connecting / reading from upstream servers or 502 if all upstream servers are down. – Xavier Lucas Nov 4 '14 at 10:05
  • Right, answer corrected – Bernard Rosset Nov 4 '14 at 10:24
  • I do it twice to be able to identify the server in the upstream that I want to replace with the new ip:port. The second one isn't replaced until the new container is up and running. – user2108258 Nov 4 '14 at 11:45
  • So while you only have the first container running, you have one server in upstream which is incorrect. nginx would not even load that incorrect configuration since it does does pass nginx -t. I am surprised it works at all. – Bernard Rosset Nov 4 '14 at 12:19
  • Are you saying that having one server listed in the upstream would not pass nginx -t or that the upstream has one live server accepting requests so it will always fail? I tried leaving one server listed in the upstream and nginx -t passed. I'm using nginx 1.6.2 on centos 6.5. However I think I understand you what you mean now. – user2108258 Nov 4 '14 at 13:24
1

I've had something of the same issue myself: the container startup times just aren't the same accross deployments of the image... Thus restarting a container from a container isn't always as fast... It might just be that you'd have to setup some kind of consistency check before rotating the other container ... You might even want to hit the container from the shell script to do a healthcheck instead of just waiting an arbitrary 15sec... ?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.