0

I'm pretty new to the kubernetes. I need some suggestions how to access app which is deployed in cluster.

I have a docker image with Django app and Gunicorn to run .wsgi. I'm using this image to deploy containers in cluster with 3 nodes (environment to learn k8s, 1 master 2 workers). If I expose service with nodePort and use curl <node'sIP>:[port], from server not included to the cluster, I can reach the app.

But now I want to have a single entry point, which would be able to connet with any of replicas.

I thought about creating one more machine, not included to the cluster with installed Nginx or HAproxy, but I can't find any solution how to connect external reverse proxy with kubernetes service.

Can you give me any better solution, or a suggestions how can I move on with my thought?

2
  • Can you tell more about your "environment to learn k8s"? Is this a cluster created locally in VM's (like in Virtualbox) or VM's in the cloud (like GCP or AWS)? You could use nginx-ingress-controller with service of type LoadBalancer to direct the traffic to your application pods (and have single entrypoint by that). – Dawid Kruk Jul 6 '20 at 17:02
  • I forgot to tell about that, sorry. This cluster is created on Virtualbox VM's. If I'm right, LoadBalancers are available in cloud only, because you have to buy them from provider, tell me if I'm wrong. So this is problem. I searched the internet, and there is only guides for ingress with loadbalancer in cloud. I tried to set up ingress with nodeport service, to have access to app on each node in cluster, but unfortunately I did something wrong, because I still can't reach app – aFku Jul 7 '20 at 6:58
0

TL;DR

In the managed Kubernetes clusters like GKE, EKS or AKS you have an ability to request IP addresses for services from your cloud provider.

Kubernetes clusters created for example in Virtualbox using either kubespray,kubeadm will require additional tools to be able to get the External-IP for the services.

You can use metallb to assign External-IP in clusters not managed by cloud provider. Please take a look on official site:

In that case steps to expose an app would be:

  • Deploy the app pods
  • Create a service attached to the pods
  • Deploy metallb and configure it
  • Expose the app by either:
    • Create a service of type LoadBalancer
    • Create an Ingress resource

Disclaimer!

Creating an Ingress resource requires an Ingress controller. I've attached below a link to one of it.

I've included an example and more explanation below of how you can expose your app in the cluster created in Virtualbox with metallb and nginx-ingress-controller.


This is an example showing how to expose application on Kubernetes cluster created in Virtualbox with:

  • metallb
  • nginx-ingress-controller

I've divided this answer on 2 parts:

  • Virtualbox
  • Kubernetes

Virtualbox

When you create your VM's in Virtualbox you have multiple ways to configure networking. Some of them are:

  • Bridged networking - attaching your Kubernetes cluster to physical network
  • Host-only - allowing single adapter to be connected to nodes and the host allowing network connectivity. This solution requires either another adapter with access to the Internet or a gateway (like for example PfSense) with the same Host-only network adapter attached.

This example uses client and 3 nodes with Bridged adapters:

  • client with IP address: 192.168.0.2
  • master with IP address: 192.168.0.117
  • worker1 with IP address: 192.168.0.118
  • worker2 with IP address: 192.168.0.119

To make sure this example works outside of the host, I used another device in the same network.

Please make sure the nodes have full connectivity between each other as well as your host have full access to them.

Kubernetes

Let's assume that:

  • Cluster is configured correctly and has access to the Internet
  • You have full access to this cluster
  • You can "ping" every node from any other host
  • Your pod responds to request on port of your choosing (for example 8000)

Steps:

  • Create a pod with gunicorn
  • Create a service definition for gunicorn pod
  • Deploy Metallb
  • Expose app with service of type LoadBalancer
  • Expose app with Ingress resource

Create a pod with gunicorn

I used below deployment definition to create a pod that will respond to requests:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gunicorn
spec:
  selector:
    matchLabels:
      app: gunicorn
  replicas: 1
  template:
    metadata:
      labels:
        app: gunicorn
    spec:
      containers:
      - name: gunicorn
        image: ubuntu
        command:
        - sleep
        - "infinity"

I used ubuntu image which I exec`d into and run below commands:

  • $ apt update && apt install -y python3 python3-pip
  • $ pip3 install gunicorn

Example code I used myapp.py:

def app(environ, start_response):
    data = b"Hello, World!\n"
    start_response("200 OK", [
        ("Content-Type", "text/plain"),
        ("Content-Length", str(len(data)))
    ])
    return iter([data])
  • $ gunicorn -w 1 -b 0.0.0.0 myapp:app &

Above example is based on:

After this steps your pod should respond on port 8000 with Hello, World!.

Create a service definition for gunicorn pod

Service definition for gunicorn pod:

apiVersion: v1
kind: Service
metadata:
  name: gunicorn-service
spec:
  selector:
    app: gunicorn
  ports:
    - name: gunicorn-port
      port: 8000
      targetPort: 8000
  type: NodePort

Deploy Metallb

Following official Metallb documentation: Metallb.universe.tf: Installation:

  • Edit kube-proxy to change strictARP mode from false to true:
    • $ kubectl edit configmap -n kube-system kube-proxy
  • $ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
  • $ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
  • $ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)

You will also need to add a configMap that will tell metallb which IP addresses he could allocate for services:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.0.240-192.168.0.250

Please take a specific look on part:

      addresses:
      - 192.168.0.240-192.168.0.250

Please make sure that this address range does not overlap any used addresses in your network and it's in the same network as the nodes and the client.

After that you should be able to create service of type LoadBalancer that will get an IP address.

Expose app with service of type LoadBalancer

You should be able to run:

  • $ kubectl expose deployment gunicorn --type=LoadBalancer --port=8000

Output of $ kubectl get services should show:

NAME               TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
gunicorn           LoadBalancer   10.233.43.104   192.168.0.240   8000:32591/TCP   3s
gunicorn-service   NodePort       10.233.34.96    <none>          8000:30862/TCP   73s
kubernetes         ClusterIP      10.233.0.1      <none>          443/TCP          23h

as you can see gunicorn service has the EXTERNAL-IP of 192.168.0.240 which is in IP pool of metallb.

Physical network used in this example is 192.168.0.0/24

You can run: curl 192.168.0.240:8000 and get: Hello, World!

Expose app with Ingress resource

You will need to deploy Ingress controller before exposing your app. You can deploy nginx-ingress-controller by running:

  • $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml

This Ingress controller will have a service type of LoadBalancer attached to it and this will be the entrypoint for the request.

You can get it's IP address by invoking: $ kubectl get svc -n ingress-nginx:

NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.233.52.8     192.168.0.241   80:31380/TCP,443:30066/TCP   29m
ingress-nginx-controller-admission   ClusterIP      10.233.37.255   <none>          443/TCP                      29m

Here is the Ingress definition to route the traffic from controller to your pod:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gunicorn-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: 
    http:
      paths:
      - path: /
        backend:
          serviceName: gunicorn-service
          servicePort: gunicorn-port

After applying it, you should be able to:

  • curl 192.168.0.241 and get the response from gunicorn
10
  • Thank you very much for your answer. This is best guide on SE I've ever seen. To be honest, Metallb was a missing part in my infrastructure. I didn't know this type of software exist. I appreciate your hard work, but I stuck after exposing loadbalancer service. I changed strictARP to true, applied namespace, created controller and speakers, generated secret. Also I applied configMap with correct IP range .2.240-.2.250 (192.168.2.0/24 network). I exposed deployment with loadbalancer, but all time status of External-IP is pending :( – aFku Jul 22 '20 at 18:53
  • I forgot to add this but please check if you have the kube-proxy in iptables or ipvs mode. If you have it in iptables please do not change the strict mode on arp. – Dawid Kruk Jul 22 '20 at 19:08
  • Also, please check the logs of the controller and the speaker. – Dawid Kruk Jul 22 '20 at 19:15
  • I looked to the logs and controller is not able to reach kubernetes Cluster-IP service (10.96.0.1) on port 443 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98: Failed to list *v1.ConfigMap: Get https://10.96.0.1:443/api/v1/namespaces/metallb-system/configmaps?fieldSelector=metadata.name%3Dconfig&limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host also it shows failed to list *v1.Service. Speakers had some problem with memberlist: Failed to join <ip> : dial tcp 192.168.2.3:7946: connect: no route to host – aFku Jul 22 '20 at 19:38
  • How did you configure/deploy your cluster? What CNI plugin are you using? – Dawid Kruk Jul 22 '20 at 19:40
0

Your missing a couple of fundamental Kubernetes concepts that I recommend you look at:

  1. Services - these provide an abstraction over your pods and provide a way to load balance traffic over multiple replicas. These can be used internally, or exposed externally
  2. Ingress - this allows you to setup an Ingress controller inside your cluster, like Nginx, and have this reverse proxy your traffic to the appropriate services. This is a much better approach than creating an Nginx instance outside your cluster, as an Ingress controller will automatically pickup any new ingresses you create in your cluster.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.