4

Hi I'm learning kubernetes and I'm having trouble exposing the service. I want to route traffic to my cluster from HAProxy. I'm using my own bare-metal server.

EDIT: I've also created an ingress controller.

Now, when I describe my ingress I can see IP Address of worker machine but still I've got Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) And no idea how to get access to my pod...

example config:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache
  labels:
    app: apache-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: apache-test
  template:
    metadata:
      labels:
        app: apache-test
    spec:
      containers:
      - name: apache
        image: httpd
        ports:
        - containerPort: 80

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: apache-test-service
spec:
  selector:
    app: apache-test
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    name: http

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: apache-test-ingress
spec:
  rules:
  - host: apache-test.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: apache-test-service
            port:
              number: 80

What's wrong?

describe ingress:

Name:             apache-test-ingress
Namespace:        default
Address:          192.168.6.72
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host         Path  Backends
  ----         ----  --------
  apache-test  
               /   apache-test-service:80 (10.44.0.1:80)
Annotations:   <none>
Events:        <none>

describe service:

Name:              apache-test-service
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=apache-test
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.104.63.167
IPs:               10.104.63.167
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.44.0.1:80
Session Affinity:  None
Events:            <none>

describe controller:

Name:         ingress-nginx-controller-55bc4f5576-vpsgb                                                                                                                                                    
Namespace:    ingress-nginx
Priority:     0
Node:         kubernetes-node02/192.168.6.72
Start Time:   Sun, 16 May 2021 16:47:26 +0200
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx 
              pod-template-hash=55bc4f5576
Annotations:  <none>
Status:       Running
IP:           10.36.0.1
IPs:
  IP:           10.36.0.1
Controlled By:  ReplicaSet/ingress-nginx-controller-55bc4f5576
Containers:
  controller:
    Container ID:  docker://7daf566a039aba0d06f856b0adcc03659423ec2462c33d9a79f820b58dfcbf98
    Image:         k8s.gcr.io/ingress-nginx/controller:v0.46.0@sha256:52f0058bed0a17ab0fb35628ba97e8d52b5d32299fbc03cc0f6c7b9ff036b61a
    Image ID:      docker-pullable://k8s.gcr.io/ingress-nginx/controller@sha256:52f0058bed0a17ab0fb35628ba97e8d52b5d32299fbc03cc0f6c7b9ff036b61a
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-controller-leader
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Sun, 16 May 2021 16:47:28 +0200
    Ready:          True
Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-55bc4f5576-vpsgb (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ftnfs (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-ftnfs:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

logs from POD doesn't show anything... logs from ingress controller:

I0516 14:47:28.871207       8 flags.go:208] "Watching for Ingress" class="nginx"
W0516 14:47:28.871287       8 flags.go:213] Ingresses with an empty class will also be processed by this Ingress controller
W0516 14:47:28.872068       8 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0516 14:47:28.872594       8 main.go:241] "Creating API client" host="https://10.96.0.1:443"
I0516 14:47:28.887394       8 main.go:285] "Running in Kubernetes cluster" major="1" minor="21" git="v1.21.0" state="clean" commit="cb303e613a121a29364f75cc67d3d580833a7479" platform="linux/amd64"
I0516 14:47:29.768986       8 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0516 14:47:29.772688       8 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"
W0516 14:47:29.775841       8 main.go:127] No IngressClass resource with name nginx found. Only annotation will be used.
I0516 14:47:29.793896       8 ssl.go:532] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0516 14:47:29.829161       8 nginx.go:254] "Starting NGINX Ingress controller"
I0516 14:47:29.848934       8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"0cf6bc98-71b3-4387-a535-7d3dcb956fc8", APIVersion:"v1", ResourceVersion:"401441", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0516 14:47:30.936661       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"apache-test-ingress", UID:"6e3c5757-28cf-4a68-be98-827fd69ee86f", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"400092", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0516 14:47:31.030103       8 nginx.go:296] "Starting NGINX process"
I0516 14:47:31.030266       8 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I0516 14:47:31.030658       8 nginx.go:316] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0516 14:47:31.031274       8 controller.go:146] "Configuration changes detected, backend reload required"
I0516 14:47:31.040799       8 leaderelection.go:253] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0516 14:47:31.041189       8 status.go:84] "New leader elected" identity="ingress-nginx-controller-55bc4f5576-vpsgb"
I0516 14:47:31.054203       8 status.go:204] "POD is not ready" pod="ingress-nginx/ingress-nginx-controller-55bc4f5576-vpsgb" node="kubernetes-node02"
I0516 14:47:31.129614       8 controller.go:163] "Backend successfully reloaded"
I0516 14:47:31.129922       8 controller.go:174] "Initial sync, sleeping for 1 second"
I0516 14:47:31.130053       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-55bc4f5576-vpsgb", UID:"16d9fca9-8ac9-4fc1-be40-056540857035", APIVersion:"v1", ResourceVersion:"401513", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0516 14:48:31.054140       8 status.go:284] "updating Ingress status" namespace="default" ingress="apache-test-ingress" currentValue=[] newValue=[{IP:192.168.6.72 Hostname: Ports:[]}]
I0516 14:48:31.067947       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"apache-test-ingress", UID:"6e3c5757-28cf-4a68-be98-827fd69ee86f", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"401625", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

describe POD

Name:         apache-67487b7c8b-8jbgb
Namespace:    default
Priority:     0
Node:         kubernetes-node01/192.168.6.71
Start Time:   Sun, 16 May 2021 15:13:07 +0200
Labels:       app=apache-test
              pod-template-hash=67487b7c8b
Annotations:  <none>
Status:       Running
IP:           10.44.0.1
IPs:
  IP:           10.44.0.1
Controlled By:  ReplicaSet/apache-67487b7c8b
Containers:
  apache:
    Container ID:   docker://70e4e3c4e01dffa11aa3c945f297e2cf3bc8af249c8d900c8aa30381ce7f56e6
    Image:          httpd
    Image ID:       docker-pullable://httpd@sha256:e4c2b93c04762468a6cce6d507d94def02ef4dc285278d0d926e09827f4857db
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sun, 16 May 2021 15:13:10 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c8dfx (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-c8dfx:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

EDIT: I used an ingress controller:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.46.0/deploy/static/provider/baremetal/deploy.yaml
Improve this question
8
  • Hi Paweł Zając welcome to S.F. Be aware that traffic is only routed to Ready pods, so if you apache is in CLBO or similar, it will also not work. You'll want some combination of kubectl describe and kubectl logs across the components to see what the misalignment is. No one can help you with the minimum level of detail you've provided here. Good luck!
    – mdaniel
    May 16, 2021 at 18:07
  • Hi, thanks for your answer. I've added more details. I want to be sure: curl 192.168.6.72 -H 'Host apache-test' should give me right output?
    – Paweł Zając
    May 16, 2021 at 20:42
  • Not as written, because your -H is missing the colon separator between the Host: header and its value -- but otherwise, yes. If that for some reason does not pan out, you may next try curl -vH 'host: apache-test' http://10.36.0.1 to go directly to the Ingress controller's Service. All of that, of course, should happen only after, as I originally said, checking that the Pod is Ready and otherwise behaving sanely: curl -v 10.44.0.1 and curl -v 10.104.63.167
    – mdaniel
    May 16, 2021 at 22:32
  • Pod says it's running (I've added full describe) When I'm trying all of this curl commands...still no output. Trying to use curl from pod with apache, it returns 200, so it's working...but only inside cluster...
    – Paweł Zając
    May 16, 2021 at 23:16
  • I find the assertion "No output" from a curl with -v highly suspicious. Since you don't want to include the output of your own investigation, I think this "troubleshooting via comments" is now over. Good luck with your question
    – mdaniel
    May 16, 2021 at 23:18

1 Answer 1

Reset to default
2

TL;DR

Instead of using the port of 80/443, you should use the ports that are associated with the Service of type NodePort that you've created during the Ingress controller provisioning.

  • $ kubectl get services -n
NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.233.39.209   <none>        80:30983/TCP,443:32636/TCP   4h30m
ingress-nginx-controller-admission   ClusterIP   10.233.54.211   <none>        443/TCP                      4h30m

In this example, you should use the IP address of one of your Nodes and the corresponding port (when trying to connect to it from outside):

  • curl http://IP_ADDRESS:30983 or curl -v -k https://IP_ADDRESS:32636

Explanation

Focusing on the part of the YAML manifest that you've used:

# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    <-- REDACTED --> 
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: NodePort # <-- IMPORTANT 
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller

When you applied the whole manifest you've created a Service of type NodePort.

Citing the official documentation:

  • NodePort: Exposes the Service on each Node's IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You'll be able to contact the NodePort Service, from outside the cluster, by requesting <NodeIP>:<NodePort>.

-- Kubernetes.io: Docs: Concepts: Services networking: Service: Publishing services service types

Type Nodeport

If you set the type field to NodePort, the Kubernetes control plane allocates a port from a range specified by --service-node-port-range flag (default: 30000-32767). Each node proxies that port (the same port number on every Node) into your Service. Your Service reports the allocated port in its .spec.ports[*].nodePort field.

-- Kubernetes.io: Docs: Concepts: Services networking: Service: Nodeport

This NodePort is the entrypoint for your Ingress controller. You will need to send a request to it's ports to contact your Ingress controller (and then Ingress controller will route the traffic accordingly to the Ingress resource).

You can check to which port you should send the traffic to by invoking (previously mentioned):

  • $ kubectl get services -n
NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.233.39.209   <none>        80:30983/TCP,443:32636/TCP   4h30m
ingress-nginx-controller-admission   ClusterIP   10.233.54.211   <none>        443/TCP                      4h30m

In this example it should be following:

  • curl http://IP_ADDRESS:30983 for HTTP
  • curl -v -k https://IP_ADDRESS:32636 for HTTPS

If you would like to expose your Nginx Ingress controller on a port 80/443 you could opt to use Service of type LoadBalancer that is backed with:

Metallb will allow you to create a pool of IP address that will be available for allocation for Service of type LoadBalancer.

A side note!

Please remember that with the Ingress resource you've specified, you should send a request with a Host: apache-test.com otherwise you will get a 404. For testing purposes you can set:

  • - host: instead of:
  • - host: apache-test.com

Addressing following part of the question:

Now, when I describe my ingress I can see IP Address of worker machine but still I've got Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) And no idea how to get access to my pod...

This won't limit your ability to connect to your Services as this is a resource to send the traffic to when it doesn't match any of the rules in the Ingress resource.

Additional resources:

Improve this answer
2
  • Hello @PawełZając and welcome to ServerFault! Please remember to react to answers for your questions. That way we know if the answers were helpful and other community members could also benefit from them. Try to accept answer that is the final solution for your issue, upvote answers that are helpful and comment on those which could be improved or require additional attention. Enjoy your stay!
    – Wytrzymały Wiktor
    May 19, 2021 at 11:57
  • Thanks a lot for the clarification. I implement and learn further :)
    – Paweł Zając
    May 20, 2021 at 17:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .