Trying to determine, from the nfs server side, what files/directories are being accessed on a read-only nfs share

Question

Is it possible to find out what files and applications are being used on a nfs share from the server side?

We make files/applications available to our users from a (read-only) nfs share. It has not been cleaned in the last 10 years. Can I find out what our users are still using without monitoring them client side?

I've tried looking at access times, but they don't get updated on the server. I've tried looking at the locking daemon, but no files/directories seems to get locked. I guess because it's mounted read-only.

EDIT
The server is mostly up to date, we update once or twice a year and unattended-upgrades is running for security updates. The data has been accumulating for 10 years.
Debian 9
Kernel 4.9.0-4-amd64
nfs-kernel-server 1:1.3.4-2.1

Answer 1

How about using Wireshark and capturing or filtering for NFS port?

You would be able to see the different actions and what is being requested.

Answer 2

You could use lsof -N <share> It will only show the files that are currently being used. But you could maybe script something over a course of a few days to build a list of documents. Just a thought

Answer 3

I've only found two ways to do this. One is to use Systemtap on the server.

The other is to query every NFS client using lsof -N. You can get a list of clients by running showmount. If you have ssh keys set up, this script is a starting point. If you're looking for a specific volume/mount you can add in a pattern to grep for.

#!/bin/bash
mkdir /var/tmp/output.$$
for ip in `showmount --no-headers` ; do
  ssh -n $ip /usr/sbin/lsof -N > /var/tmp/output.$$/$ip &
done
wait
cat /var/tmp/output.$$/* | awk '{print $9,$10}' | sort -u
rm -rf /var/tmp/output.$$

Answer 4

In addition to using Wireshark (apt-get install tshark on a Debian system; then use something like tshark -f 'port 2049 && host 1.2.3.4' -i eth0 -w savefile -P -t ad and e.g. read it with Wireshark later ), one might want to apply bcc -- for example, use trace.py utility as follows:

# trace.py -n nfsd \
  'vfs_open(const struct path *pP, struct file * pF)  (pP->dentry) "%s (%d)", pP->dentry->d_name.name,pP->dentry->d_inode->i_ino'

PID     TID     COMM            FUNC             -
12333   12333   nfsd            vfs_open         testfile (1337)
12333   12333   nfsd            vfs_open         testdir (1336)

This was the result of a simple cd testdir; echo 'abc' >testfile on the client mounting the share .

d_name.name here would be of course the name of the file, and d_inode->i_ino would be its inode number. So, if the exported filesystem is known, a single file could be found by e.g. find $path -inum $inode.

( In the case if there are inner filesystems exported with nohide or crossmnt options under the same root, find may find more than one entry -- in which case the filename may help to identify the correct one. )