I think that your problem is that the cacheing that's going on is not specifically NFS-related. Here's what I see on the network when I do an
md5sum of a small NFS-mounted file for the second time in five seconds:
12:29:27.615255 IP6 2a02:xxxx:6049:11:76d4:35ff:fe8d:39ec.1003 > 2a02:xxxx:6049:11:7271:bcff:feac:445a.2049: Flags [P.], seq 3028:3236, ack 34857, win 1432, options [nop,nop,TS val 453693630 ecr 2105657370], length 208: NFS request xid 3660266482 204 getattr fh 0,2/53
12:29:27.617595 IP6 2a02:xxxx:6049:11:7271:bcff:feac:445a.2049 > 2a02:xxxx:6049:11:76d4:35ff:fe8d:39ec.1003: Flags [P.], seq 34857:35153, ack 3236, win 500, options [nop,nop,TS val 2105661555 ecr 453693630], length 296: NFS reply xid 3660266482 reply ok 292 getattr NON 3 ids 0/-2006101416 sz 1610612736
12:29:27.617632 IP6 2a02:xxxx:6049:11:76d4:35ff:fe8d:39ec.1003 > 2a02:xxxx:6049:11:7271:bcff:feac:445a.2049: Flags [.], ack 35153, win 1432, options [nop,nop,TS val 453693632 ecr 2105661555], length 0
All the kernel has done is check that the file hasn't been modified, with a
getattr (because it's correctly not cacheing any attributes, but instead going back to the server for fresh ones). Then it knows it the file is unmodified, so it can use the pages that are still in-core that contain the content. This is exactly what it would do with a local file as well, if I understand correctly.
So it's not that you want to turn off NFS cacheing so much as you want to modify file cacheing in the VM subsystem so that it specifically doesn't apply to NFS-mounted content. This may be quite a lot more difficult, because you're not trying to turn off something that's already there, you're trying to introduce a whole new distinction that probably isn't there right now, if you follow me.
I think I agree with Andrew Henle that it's sounding like an XY problem. Perhaps you should tell us what you are really trying to do.
Edit: you assert that the same is not true for local file access. However, I think you're wrong. I've connected a HDD to my desktop to which only one process has access, and from that I have twice in a row
md5summed a file while using
iostat to monitor traffic to and from the physical device.
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 0.00 0.80 0.00 49.60 0.00 124.00 0.08 95.75 95.75 0.00 94.00 7.52
Note the read traffic (rkB/s) from the drive; now the second time:
sdc 0.00 0.20 0.00 0.40 0.00 2.40 12.00 0.00 1.00 0.00 1.00 1.00 0.04
so it looks like there, too, the VM subsystem is getting in the way. So at the moment I see no reason to think you aren't seeing on your netapp all the traffic that clients correctly generate, and I ask again, what are you actually trying to do here? Why is it that you "must disable [cacheing], and send the actual FILE READ from the redhat"? Because at the moment, it really looks like a non-problem.