I have an ingress that routes to a custom endpoint external to the kubernetes cluster. The service listens only on HTTPS on port 8006.

apiVersion: v1
kind: Service
  name: pve
    - protocol: TCP
      port: 8006
apiVersion: v1
kind: Endpoints
  name: pve
  - addresses:
      - ip:
      - port: 8006
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
  name: pve

    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "off"
    nginx.ingress.kubernetes.io/whitelist-source-range: ""

    - hosts:
        - pve.example.com
      secretName: pve-tls
    - host: pve.example.com
          - backend:
              serviceName: pve
              servicePort: 8006
            path: /

Gives the error in the nginx pod: - - [28/Aug/2020:01:17:58 +0000] "GET / HTTP/1.1" 502 157 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0" "-"

2020/08/28 01:17:58 [error] 2609#2609: *569 upstream prematurely closed connection while reading response header from upstream, client:, server: pve.example.com, request: "GET / HTTP/1.1", upstream: "", host: "pve.example.com"


After removing the proxy protocol, I get the error - - [28/Aug/2020:02:19:18 +0000] "GET / HTTP/1.1" 400 59 "-" "curl/7.58.0" "-"

2020/08/28 02:19:26 [error] 2504#2504: *521 upstream prematurely closed connection while reading response header from upstream, client:, server: pve.example.com, request: "GET / HTTP/1.1", upstream: "", host: "pve.example.com" - - [28/Aug/2020:02:19:26 +0000] "GET / HTTP/1.1" 502 157 "-" "curl/7.58.0" "-"


And in case it is relevant, my nginx configuration, deployed through the helm char nginx-stable/nginx-ingress

  ## nginx configuration
  ## Ref: https://github.com/kubernetes/ingress/blob/master/controllers/nginx/configuration.md
        hsts-include-subdomains: "false"
        ssl-protocols: "TLSv1.1 TLSv1.2"
    ingressClass: nginx
      externalTrafficPolicy: Local
        metallb.universe.tf/address-pool: default
    enabled: true
    22: "gitlab/gitlab-gitlab-shell:22"
  • This appears to be two separate and distinct problems, unrelated to each other. It would be best to post them as separate posts. I've answered one of them below. I have no idea about the other. You can just edit it out of here and paste it into a new question. – Michael Hampton Aug 28 at 1:50
  • @MichaelHampton I split up the questions, second one here serverfault.com/questions/1031810/… – cclloyd Aug 28 at 2:02
  • @cclloyd Could you share some more details regarding that custom endpoint external to the kubernetes cluster? – Wytrzymały Wiktor Sep 1 at 11:03

This annotation is probably the cause of the problem.

    nginx.ingress.kubernetes.io/use-proxy-protocol: "true"

The docs state:

Enables or disables the PROXY protocol to receive client connection (real IP address) information passed through proxy servers and load balancers such as HAProxy and Amazon Elastic Load Balancer (ELB).

If you don't have a load balancer in front of your Ingress which is passing connections in using the PROXY protocol, then this is not what you want, and this annotation should not be present (or should be "false").

| improve this answer | |
  • Removing it doesn't seem to change anything. – cclloyd Aug 28 at 2:15
  • @cclloyd Very odd. Check to see if you are getting different error messages from nginx now. – Michael Hampton Aug 28 at 2:17

This is a community wiki answer. Feel free to expand it.

The error you see: upstream prematurely closed connection while reading response header from upstream comes from Nginx and means that the connection was closed by your "upstream".

It's hard to say what might be the exact cause of this issue without the necessary details but what you can do regardless is to try to increase the timeout value according to this documentation, specifically:

  • proxy_read_timeout

  • proxy_connect_timeout

You can also adjust code/configuration on your "upstream", whatever it is in your use case.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.