Assuming that there is indeed a culprit (a single user) responsible for the majority of the 2.5k IOPS:
I'd start with
top - at that rate you should see one or a few users and processes standing out in the active processes on the box, mostly sleeping but quite often in ready state as well - I'd press
i (to hide inactive processes) then press and hold the space bar for very fast updates.
I'd single out the users showing up more often and only display their processes in
top for a more stable view.
If you see many processes from the same user (for example a complex build performed on NFS) - check the process tree for that user to confirm
pstree -ps <user>. It may be difficult to prove that such process collection is the cause other than by starting/stopping it and watching for correlations in activity changes on the netapp side.
If the culprit is a single process I'd expect it would be a steady presence in the
top output. In addition to
find I'd also look for:
- rsync, dd, cp, rcp, scp, tar, backup solutions, etc.
- the typical high-level programming interpretors: python, java, etc
(running custom scripts)
- web/database apps
- CI/QA systems (if the server is involved in sw development)
But it could also be something completely custom, you won't find it "by name".
It's also possible for the rate to be the collective effect of so many users (is the NFS server holding their homedirs and/or shared project partitions?) - not much you can do about it - maybe time to scale up the NFS storage?