I have a CentOS 6.4 fileserver running NFS 4, serving a couple of XFS filesystems. There are a few dozen clients connected to it. Today it slowed to a crawl for the clients - the clients would hang or only respond after a few minutes when accessing the mounted NFS share from the server. On the server itself I could access the shared filesystems without trouble. The trouble went away after about four hours, but I don't know why - see below.
top showed several
migration processes and
xfssyncd process consuming unusual amounts of cpu, jumping between 0% and anywhere up to 100% every few seconds. No other processes were noticeably active. The overall cpu usage reported by top was low, like so:
Cpu(s): 0.0%us, 4.2%sy, 0.0%ni, 95.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
I haven't been able to find anything online talking about this in particular, apart possibly from a RHEL support entry that's in their subscriber-only section and I can't see.
service nfs restart. Then
service nfs status showed running daemons, except
nfsd dead but subsys locked. After another restart this was gone and nfsd was running, but clients were still hung.
I tried some things that were suggested for xfssyncd-related issues:
mount –o remount /mnt/data on the exported fs. Interestingly, this command took about a minute to run, and during this time the 'wild' processes settled down. But once the command finished running, the processes were back to high cpu usage.
echo 720000 > /proc/sys/fs/xfs/xfssyncd_centisecs in order to change the sync interval for xfssyncd. This didn't make any noticeable difference, which is not too surprising since the fs is busy with NFS clients and the issue must be something else.
I had an issue with this server 3 weeks ago in which a .nfsNNN file (from a removed file is still open and being accessed) was filling up fast with a looping error message on a client. Killing the problem process fixed the NFS slowdown. [However the file server then proceeded to slow down again a couple days later without such a .nfsNNN file issue, and I eventually had to just reboot it. At the time I saw some processes with unusal cpu levels, but didn't note what they were and can't remember now if they were the same as the current issue.]
I searched today again for open .nfsNNN files that were maybe problems, but found none. I did delete some from a few months ago, but they weren't being currently modified so figured they weren't a problem. I noticed no difference after deleting these files.
About an hour after trying the various things above, the server went back to normal and the
xfssyncd processes no longer had high cpu usage. I don't know what changed, but I'd like to try and get ahead of figuring this out since it seems it might happen again.
Thanks for any suggestions.