Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a CentOS 6.4 fileserver running NFS 4, serving a couple of XFS filesystems. There are a few dozen clients connected to it. Today it slowed to a crawl for the clients - the clients would hang or only respond after a few minutes when accessing the mounted NFS share from the server. On the server itself I could access the shared filesystems without trouble. The trouble went away after about four hours, but I don't know why - see below.

top showed several migration processes and xfssyncd process consuming unusual amounts of cpu, jumping between 0% and anywhere up to 100% every few seconds. No other processes were noticeably active. The overall cpu usage reported by top was low, like so:

Cpu(s): 0.0%us, 4.2%sy, 0.0%ni, 95.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

I haven't been able to find anything online talking about this in particular, apart possibly from a RHEL support entry that's in their subscriber-only section and I can't see.

I ran service nfs restart. Then service nfs status showed running daemons, except nfsd dead but subsys locked. After another restart this was gone and nfsd was running, but clients were still hung.

I tried some things that were suggested for xfssyncd-related issues:

1) mount –o remount /mnt/data on the exported fs. Interestingly, this command took about a minute to run, and during this time the 'wild' processes settled down. But once the command finished running, the processes were back to high cpu usage.

2) echo 720000 > /proc/sys/fs/xfs/xfssyncd_centisecs in order to change the sync interval for xfssyncd. This didn't make any noticeable difference, which is not too surprising since the fs is busy with NFS clients and the issue must be something else.

I had an issue with this server 3 weeks ago in which a .nfsNNN file (from a removed file is still open and being accessed) was filling up fast with a looping error message on a client. Killing the problem process fixed the NFS slowdown. [However the file server then proceeded to slow down again a couple days later without such a .nfsNNN file issue, and I eventually had to just reboot it. At the time I saw some processes with unusal cpu levels, but didn't note what they were and can't remember now if they were the same as the current issue.]

I searched today again for open .nfsNNN files that were maybe problems, but found none. I did delete some from a few months ago, but they weren't being currently modified so figured they weren't a problem. I noticed no difference after deleting these files.

About an hour after trying the various things above, the server went back to normal and the migration and xfssyncd processes no longer had high cpu usage. I don't know what changed, but I'd like to try and get ahead of figuring this out since it seems it might happen again.

Thanks for any suggestions.

-M

share|improve this question
    
Did you gather any information about disk i/o while it was slow? How about data from /proc/meminfo? –  sciurus Dec 17 '13 at 22:09
    
No, I didn't. Hopefully I would've ended up there if the problem had persisted. –  Michael S Dec 17 '13 at 22:38
    
For i/o stats, anything you recommend besides iostat? Looking around some more I see 'iotop' which gives process-level stats - any recommendation on that? –  Michael S Dec 17 '13 at 22:49
    
I'm a huge fan of atoptool.nl , which you can install from EPEL. –  sciurus Dec 18 '13 at 15:17
    
Thanks, atop looks nice - I'll try it out. –  Michael S Dec 18 '13 at 17:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.