1

I have a Kubernetes cluster setup with 2 StatefulSets in a Kube Namespace:

  • NFSserver Statefulset defined with a "headless" service that has 1 pod with an attached persistent volume and running an NFS server exporting that Volume.
  • DataClient Statefulset has multiple pods mounting the NFS share exported above using NFSv4

The DataClients are responsible for data creation which is then stored on the NFS server volume.

The DataClients do a lot of data read/writes to the NFS volume so I am attempting to optimize the transfers as much as possible. I am currently using a "headless" service (eg the IP address of the POD is provided by KUBEDNS) for the NFSserver which means the NFS packets are directly using the NFSserver POD's IP address.

My issue is that using a headless service the IP address of the NFSserver may change (eg. pod is deleted for a DockerImage update). The clients need to monitor the server IP and update when it changes? All clients then need to 'force' umount the NFS volume (mounted at the old IP address) and immediately remount with the new IP. Using KUBEDNS the new IP address will be provided.

The first question I have is: would there be a performance impact if I use a CLUSTER IP service but allocates a service IP address that will not change when the PODs are deleted/recreated? My thinking has been that having a service IP and a separate pod IP would lead to a 'two hop' situation for every NFS packet. I don't know much about kube-proxy and can't access the kuberenetes nodes to see the IPtables config. I just read the article on kube-proxy using IPtables for routing packets so perhaps there is not the performance hit after all.

Other details:

  • Kubernetes cluster runs on 20+ openstack VM nodes
  • Persistent Volumes are openstack volumes attached to the openstack VM node where the NFS server POD is running, and then mounted into the POD container
  • The NFS clients may be on any of the 20+ VM nodes. It is not possible to have all clients on the same node as the NFS server.
  • I do not have node level access, so I cannot see what is going on at the openstack/Linux level of the node. I am confined to container access only.
  • In this cluster a kubernetes namespace identifies the lone NFS server and all its clients. There may be multiple namespaces each with an NFS server and multiple clients.

Unfortunately the service providing the Kubernetes infrastructure does not have any support for ReadWriteMany volumes which I suppose (not an expert here) could replace NFS.

NFS does provide what we need if it works correctly. I find it very finnicky with respect to hangs, and unexplained slowness in a client. Advice appreciated.

Improve this question
3
  • In practice, you'ld better ask your IT to setup a separate openstack VM, serving as your NFS host. They could probably manage it themselves, even, if they don't want to give you more than kube API access. If that's not possible: look for OpenEBS (openebs.io/docs/concepts/rwm). Kubernetes Services, or the way volumes are attached during containers creation, loading kernel modules on host, not to mention performances, ... NFS server in a Pod sounds over-complicated, and failure-prone. While more generally, NFS is rarely recommended with k8s, regardless how your server is deployed
    – SYN
    Jul 22, 2022 at 18:32
  • @SYN thanks for the suggestions. At present IT is a non-starter. I am probably going to start by switching from a "headless" service to a service with a "clusterIP" and see if that has any impact. I want to know if there is anybody with direct experience.
    – DavidG
    Jul 24, 2022 at 11:24
  • @DavidG was the answer useful?
    – Gabriel Robledo Ahumada
    Jul 27, 2022 at 22:37

1 Answer 1

Reset to default
0

That two-hop situation that you are describing could be definitely an issue when you are running multiple nodes and a request comes into one where the NFS Pod is not living because of the load balancing mechanism implemented by default in Kubernetes Services. To avoid an asymmetric routing issue, the destination node sends back the request to the node where it came originally before it goes out to the client.

Fortunately, since you are running a single NFS Pod, you can set the externalTrafficPolicy to local at the Service level; this effectively disables load balancing on the cluster node, and the traffic is sent directly to the destination node, handling all the traffic locally.

Here is an example of a Kubernetes Service with the externalTrafficPolicy set to local:

kind: Service
apiVersion: v1
metadata:
  name: store-backend
spec:
 type: LoadBalancer
 externalTrafficPolicy: Local
 selector:
    app: store
    role: backend
  ports:
  - name: https
    port: 443

Additionally, you could implement the Use of a specific IP address with a pod feature provided in the Project Calico.

Improve this answer
1
  • I did convert the service from headless to ClusterIP and remounted the 10 client pods via the serviceIP and so far so good. When I delete the NFS server pod it restarts and the clients reconnect after a minute. This cluster is 40 local nodes and your answer caused me to find internalTrafficPolicy which is more applicable in my case since all traffic to this service will be from internal pods in the cluster. But since all of my nodes are "local" to each other, I think these TrafficPolicies will have little effect. I marked this as solution since a non changing IP is the way to go.
    – DavidG
    Jul 28, 2022 at 1:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .