1

I have a large multi user system, NFS mounting a large piece of shared storage. I can identify the source of all my network traffic - narrowing it down to a particular host - thanks to nfsstat.

However I'm still having a little difficulty tracing which of the users is doing this - there's a few hundred, and there aren't any obvious culprits in the process list. (I usually start with looking for instances of find)

But we do definitely have 2.5k IOPS coming from it, and it's causing resource issues on the host.

Is anyone able to offer me suggestions to figure out culprit processes/users?

Box is a RedHat Linux, talking to a NetApp filer.

Improve this question
4
  • I hope you get a good answer here, but if you don't, you might want to consider joining the netapp administrators' mailing list, and asking on there. The list's at teaparty.net/mailman/listinfo/toasters (disclaimer: I started this list, in 1997, and still administer it). – MadHatter Jun 22 '15 at 12:48
  • Have you tried iotop? (I don't know if it handles NFS.) – Mat Jun 22 '15 at 12:51
  • I'm already on the NetApp list - I didn't think it appropriate, because this is really a Linux question. – Sobrique Jun 22 '15 at 12:51
  • Fair enough, Sobrique, and thank you for thinking about appropriate fora - so many don't bother. – MadHatter Jun 23 '15 at 6:48
2

Assuming that there is indeed a culprit (a single user) responsible for the majority of the 2.5k IOPS:

I'd start with top - at that rate you should see one or a few users and processes standing out in the active processes on the box, mostly sleeping but quite often in ready state as well - I'd press i (to hide inactive processes) then press and hold the space bar for very fast updates.

I'd single out the users showing up more often and only display their processes in top for a more stable view.

If you see many processes from the same user (for example a complex build performed on NFS) - check the process tree for that user to confirm pstree -ps <user>. It may be difficult to prove that such process collection is the cause other than by starting/stopping it and watching for correlations in activity changes on the netapp side.

If the culprit is a single process I'd expect it would be a steady presence in the top output. In addition to find I'd also look for:

  • rsync, dd, cp, rcp, scp, tar, backup solutions, etc.
  • the typical high-level programming interpretors: python, java, etc (running custom scripts)
  • web/database apps
  • CI/QA systems (if the server is involved in sw development)

But it could also be something completely custom, you won't find it "by name".

It's also possible for the rate to be the collective effect of so many users (is the NFS server holding their homedirs and/or shared project partitions?) - not much you can do about it - maybe time to scale up the NFS storage?

Improve this answer
2

Maybe a good hint is to find how many opened NFS files a user have (from the client side).

I would use lsof -N. This one liner might help you:

for user in $(who|awk '{print $1}'|sort -n|uniq); do echo $user ; lsof -N -u $user -a |wc -l; done
Improve this answer
1

There is a great tool wireshark . With it's terminal version, tshark, you can find client and uid:

$ tshark -i any -Y 'rpc.msgtyp == 0' -f 'port 2049' -Tfields -e frame.time -e ip.src -e nfs.main_opcode -e rpc.auth.uid

You will get output like:

Jun 23, 2015 11:41:13.306857000 aaa.bbb.ccc.ddd 9   0

with timestamp, client ip, nfs operation and uid. Collect data for some minutes and then you can use your favorite tools to find top users:

$ cat nfs-capture.txt | sort -n -k 7 | uniq -c
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.