Hi I'm learning kubernetes and I'm having trouble exposing the service. I want to route traffic to my cluster from HAProxy. I'm using my own bare-metal server.
EDIT: I've also created an ingress controller.
Now, when I describe my ingress I can see IP Address of worker machine but still I've got
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
And no idea how to get access to my pod...
example config:
deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: apache
labels:
app: apache-test
spec:
replicas: 1
selector:
matchLabels:
app: apache-test
template:
metadata:
labels:
app: apache-test
spec:
containers:
- name: apache
image: httpd
ports:
- containerPort: 80
service.yaml
apiVersion: v1
kind: Service
metadata:
name: apache-test-service
spec:
selector:
app: apache-test
ports:
- protocol: TCP
port: 80
targetPort: 80
name: http
ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: apache-test-ingress
spec:
rules:
- host: apache-test.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: apache-test-service
port:
number: 80
What's wrong?
describe ingress:
Name: apache-test-ingress
Namespace: default
Address: 192.168.6.72
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
apache-test
/ apache-test-service:80 (10.44.0.1:80)
Annotations: <none>
Events: <none>
describe service:
Name: apache-test-service
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=apache-test
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.104.63.167
IPs: 10.104.63.167
Port: <unset> 80/TCP
TargetPort: 80/TCP
Endpoints: 10.44.0.1:80
Session Affinity: None
Events: <none>
describe controller:
Name: ingress-nginx-controller-55bc4f5576-vpsgb
Namespace: ingress-nginx
Priority: 0
Node: kubernetes-node02/192.168.6.72
Start Time: Sun, 16 May 2021 16:47:26 +0200
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
pod-template-hash=55bc4f5576
Annotations: <none>
Status: Running
IP: 10.36.0.1
IPs:
IP: 10.36.0.1
Controlled By: ReplicaSet/ingress-nginx-controller-55bc4f5576
Containers:
controller:
Container ID: docker://7daf566a039aba0d06f856b0adcc03659423ec2462c33d9a79f820b58dfcbf98
Image: k8s.gcr.io/ingress-nginx/controller:v0.46.0@sha256:52f0058bed0a17ab0fb35628ba97e8d52b5d32299fbc03cc0f6c7b9ff036b61a
Image ID: docker-pullable://k8s.gcr.io/ingress-nginx/controller@sha256:52f0058bed0a17ab0fb35628ba97e8d52b5d32299fbc03cc0f6c7b9ff036b61a
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--election-id=ingress-controller-leader
--ingress-class=nginx
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Running
Started: Sun, 16 May 2021 16:47:28 +0200
Ready: True
Restart Count: 0
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-55bc4f5576-vpsgb (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ftnfs (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
kube-api-access-ftnfs:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
logs from POD doesn't show anything... logs from ingress controller:
I0516 14:47:28.871207 8 flags.go:208] "Watching for Ingress" class="nginx"
W0516 14:47:28.871287 8 flags.go:213] Ingresses with an empty class will also be processed by this Ingress controller
W0516 14:47:28.872068 8 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0516 14:47:28.872594 8 main.go:241] "Creating API client" host="https://10.96.0.1:443"
I0516 14:47:28.887394 8 main.go:285] "Running in Kubernetes cluster" major="1" minor="21" git="v1.21.0" state="clean" commit="cb303e613a121a29364f75cc67d3d580833a7479" platform="linux/amd64"
I0516 14:47:29.768986 8 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0516 14:47:29.772688 8 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"
W0516 14:47:29.775841 8 main.go:127] No IngressClass resource with name nginx found. Only annotation will be used.
I0516 14:47:29.793896 8 ssl.go:532] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0516 14:47:29.829161 8 nginx.go:254] "Starting NGINX Ingress controller"
I0516 14:47:29.848934 8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"0cf6bc98-71b3-4387-a535-7d3dcb956fc8", APIVersion:"v1", ResourceVersion:"401441", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0516 14:47:30.936661 8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"apache-test-ingress", UID:"6e3c5757-28cf-4a68-be98-827fd69ee86f", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"400092", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0516 14:47:31.030103 8 nginx.go:296] "Starting NGINX process"
I0516 14:47:31.030266 8 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I0516 14:47:31.030658 8 nginx.go:316] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0516 14:47:31.031274 8 controller.go:146] "Configuration changes detected, backend reload required"
I0516 14:47:31.040799 8 leaderelection.go:253] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0516 14:47:31.041189 8 status.go:84] "New leader elected" identity="ingress-nginx-controller-55bc4f5576-vpsgb"
I0516 14:47:31.054203 8 status.go:204] "POD is not ready" pod="ingress-nginx/ingress-nginx-controller-55bc4f5576-vpsgb" node="kubernetes-node02"
I0516 14:47:31.129614 8 controller.go:163] "Backend successfully reloaded"
I0516 14:47:31.129922 8 controller.go:174] "Initial sync, sleeping for 1 second"
I0516 14:47:31.130053 8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-55bc4f5576-vpsgb", UID:"16d9fca9-8ac9-4fc1-be40-056540857035", APIVersion:"v1", ResourceVersion:"401513", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0516 14:48:31.054140 8 status.go:284] "updating Ingress status" namespace="default" ingress="apache-test-ingress" currentValue=[] newValue=[{IP:192.168.6.72 Hostname: Ports:[]}]
I0516 14:48:31.067947 8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"apache-test-ingress", UID:"6e3c5757-28cf-4a68-be98-827fd69ee86f", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"401625", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
describe POD
Name: apache-67487b7c8b-8jbgb
Namespace: default
Priority: 0
Node: kubernetes-node01/192.168.6.71
Start Time: Sun, 16 May 2021 15:13:07 +0200
Labels: app=apache-test
pod-template-hash=67487b7c8b
Annotations: <none>
Status: Running
IP: 10.44.0.1
IPs:
IP: 10.44.0.1
Controlled By: ReplicaSet/apache-67487b7c8b
Containers:
apache:
Container ID: docker://70e4e3c4e01dffa11aa3c945f297e2cf3bc8af249c8d900c8aa30381ce7f56e6
Image: httpd
Image ID: docker-pullable://httpd@sha256:e4c2b93c04762468a6cce6d507d94def02ef4dc285278d0d926e09827f4857db
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Sun, 16 May 2021 15:13:10 +0200
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c8dfx (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-c8dfx:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
EDIT: I used an ingress controller:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.46.0/deploy/static/provider/baremetal/deploy.yaml
Ready
pods, so if you apache is in CLBO or similar, it will also not work. You'll want some combination ofkubectl describe
andkubectl logs
across the components to see what the misalignment is. No one can help you with the minimum level of detail you've provided here. Good luck!-H
is missing the colon separator between theHost:
header and its value -- but otherwise, yes. If that for some reason does not pan out, you may next trycurl -vH 'host: apache-test' http://10.36.0.1
to go directly to the Ingress controller's Service. All of that, of course, should happen only after, as I originally said, checking that the Pod isReady
and otherwise behaving sanely:curl -v 10.44.0.1
andcurl -v 10.104.63.167
-v
highly suspicious. Since you don't want to include the output of your own investigation, I think this "troubleshooting via comments" is now over. Good luck with your question