3

I have a Kubernetes cluster which is running Django application within docker container served by uWSGI. The ingress controller is ingress-nginx (this one: https://github.com/kubernetes/ingress-nginx).

Recently I did an upgrade of the whole cluster from 1.9 to 1.11, and due to some issues I had to run kubeadm reset and kubeadm init again.

Since then (I guess), sometimes I'm getting weird 502 errors that are reported by the users: upstream prematurely closed connection while reading response header from upstream.

The biggest problem for me is that those requests are not visible in uWSGI logs within container so I have no idea what is happening.

Here is my uwsgi.ini file:

[uwsgi]

http = 0.0.0.0:8000
# Django-related settings
# the base directory (full path)
chdir = /app
# Django's wsgi file
module = in_web_server.wsgi:application
pythonpath = /app

static-map = /static=/app/static

# process-related settings
# master
master = true
# maximum number of worker processes
processes = 10
# clear environment on exit
vacuum = true

# spooler setup
spooler = /spooler
spooler-processes = 2
spooler-frequency = 10

Dockerfile CMD: CMD ["/usr/local/bin/uwsgi", "--ini", "/app/in_web_server/docker/in/in_web_server_uwsgi.ini"]

Kubernetes Ingress: 

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: in-debug
  namespace: in-debug
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      if ($request_uri ~ "^[^?]*?//") {
          rewrite "^" $scheme://$host$uri permanent;
      }
spec:
  rules:
  - host: test-in
    http:
      paths:
      - path: "/"
        backend:
          serviceName: in-debug
          servicePort: 8000

Those errors are only for larger (but not very large) PUT requests. By larger I mean ~300KB, so it is not a big deal.

Also, the 502 error is returned after around 1 minute, so there is possibly some timeout issue. However I'm not able to locate it since there is no trace within uwsgi log. Any hints what I'm doing wrong?

Improve this question
3
  • Where you able to find it the solution @Djent? Is there a load balancer in front of Kubernetes?
    – Diego Gallegos
    Nov 5, 2018 at 1:57
  • @DiegoGallegos ,Djent have you found a solution?
    – dina
    Dec 13, 2018 at 14:24
  • No, I haven't found solution, but it stopped at some point. I have no idea what helped.
    – Djent
    Dec 14, 2018 at 5:58

2 Answers 2

Reset to default
2

From https://monicalent.com/blog/2013/12/06/set-up-nginx-and-uwsgi/ I have found out "limit-as" option which restrict process virtual memory size and may be responsible for 502 error code with message "upstream prematurely closed connection".

Improve this answer
0

Based on the issues (see the list below) I would recommend trying to turn off buffering, and experimenting with timeout values. For example, try to put them to the nginx-ingress controller config-map and remove them from the Ingress object configuration.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: in-debug
  namespace: in-debug
  annotations:
    nginx.org/proxy-buffering: "False"
...

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
data:
  proxy-connect-timeout: "600s"
  proxy-read-timeout: "600s"
  client-max-body-size: "5m"

Issues list:

Manuals:

Improve this answer
1
  • Unfortunately, it doesn't help. None of it helps.
    – Djent
    Aug 30, 2018 at 11:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .