zypper patch'ing NFS server on openSUSE Leap 15.2 to latest version and rebooting, nodes in kubernetes cluster (Openshift 4.5) can no longer mount NFS volumes.
NFS server version: nfs-kernel-server-2.1.1-lp18.104.22.168.x86_64
Affected pods are in ContainerCreating status
kubectl describe pod/<pod_name> gives a following error:
Warning FailedMount 31m kubelet MountVolume.SetUp failed for volume "volume" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/c86dee2e-f533-43c9-9a1d-c4f00a1b8eef/volumes/kubernetes.io~nfs/smart-services-http-video-stream --scope -- mount -t nfs nfs.example.invalid:/nfs/volume /var/lib/kubelet/pods/c86dee2e-f533-43c9-9a1d-c4f00a1b8eef/volumes/kubernetes.io~nfs/pv-name Output: Running scope as unit: run-r83d4e7dba1b645aca1e4693a48f45191.scope mount.nfs: Operation not permitted
Server is running NFSv4 only, so rpcbind is turned off and showmount commands are not working.
Mounting directly on kubernetes node results in following error:
sudo mount.nfs4 nfs.example.invalid:/core tmp/ -v; echo $? mount.nfs4: timeout set for Wed Jul 21 12:16:49 2021 mount.nfs4: trying text-based options 'vers=4.2,addr=192.168.11.2,clientaddr=192.168.11.3' mount.nfs4: mount(2): Operation not permitted mount.nfs4: Operation not permitted 32
firewalld rules on NFS server:
services: ssh dhcpv6-client nfs mountd rpc-bind samba http tftp ports: 2049/tcp 2049/udp
AppArmor was working, turning it off haven't changed the outcome.
Before updating NFS server, everything was working fine and no other configuration changes were made. How can i debug this further and make shares mountable again?