-1

I have a netapp server exportting NFS share (NFS3).
I am monitoring the fpolicy on the netapp server, this means, I listen to all File read operation of the volume (and share..).

From a CentOS machine, I mount that NFS and perform File read (cat) operation.
In the first time I will do "cat", in the netapp I can see that there was a FILE READ event.

But then if I do additional "cat filename", I dont get the event FILE READ counter to increase.
If I got to a different machine, and mount the NFS Share, and do cat, I will see the counter increase by one.
I assume - the NFS Client is having internal cache mechanism.

I want to disable the NFS Client Cache.

I mounted the NFS with the following command: 

mount -o lookupcache=none,actimeo=0,noac 10.10.10.1:/vol/vol0_unix /mnt/nfs1

Notice the lookupcache=none,actimeo=0,noac options - taken from link.

Are there additional NFS Client cache mechanism i am missing ?

My NFS Client is: Linux CENTOS 6.4 2.6.32-358.el6.x86_64

Machine NFS Version:

[ilan@DevCore-Centos6 nfs1]$ rpm -qa|grep -i nfs
nfs-utils-1.2.3-36.el6.x86_64
nfs-utils-lib-1.1.5-6.el6.x86_64
nfs4-acl-tools-0.3.3-6.el6.x86_64
[ilan@DevCore-Centos6 nfs1]$

I Assume that cachefilesd is not running by default.

Improve this question
3
  • 5
    I want to disable the NFS Client Cache. Why? What problem are you trying to solve? This looks like an XY problem. See meta.stackexchange.com/a/66378
    – Andrew Henle
    Feb 10, 2017 at 12:33
  • My actual problem is that I am monitoring FILE READ events on the Netapp, There are many NFS Client (different users BUT on the SAME Redhat machine) that performing FILE READ operations on the netapp server, and I only get SOME of the FILE READ events, I need to get all the FILE READ events, I dont care if its from different users or the same user.
    – ilansch
    Feb 12, 2017 at 8:41
  • Sorry, this question has gone way past the sensible point. You've asked a question, and had an answer (two, in fact) - but you have some weird business need to outguess the linux kernel's VM subsystem. that you won't tell us, no matter how many times we ask. I'm voting to close as "unclear what you're asking".
    – MadHatter
    Feb 12, 2017 at 20:39

2 Answers 2

Reset to default
4

I think that your problem is that the cacheing that's going on is not specifically NFS-related. Here's what I see on the network when I do an md5sum of a small NFS-mounted file for the second time in five seconds:

12:29:27.615255 IP6 2a02:xxxx:6049:11:76d4:35ff:fe8d:39ec.1003 > 2a02:xxxx:6049:11:7271:bcff:feac:445a.2049: Flags [P.], seq 3028:3236, ack 34857, win 1432, options [nop,nop,TS val 453693630 ecr 2105657370], length 208: NFS request xid 3660266482 204 getattr fh 0,2/53
12:29:27.617595 IP6 2a02:xxxx:6049:11:7271:bcff:feac:445a.2049 > 2a02:xxxx:6049:11:76d4:35ff:fe8d:39ec.1003: Flags [P.], seq 34857:35153, ack 3236, win 500, options [nop,nop,TS val 2105661555 ecr 453693630], length 296: NFS reply xid 3660266482 reply ok 292 getattr NON 3 ids 0/-2006101416 sz 1610612736
12:29:27.617632 IP6 2a02:xxxx:6049:11:76d4:35ff:fe8d:39ec.1003 > 2a02:xxxx:6049:11:7271:bcff:feac:445a.2049: Flags [.], ack 35153, win 1432, options [nop,nop,TS val 453693632 ecr 2105661555], length 0

All the kernel has done is check that the file hasn't been modified, with a getattr (because it's correctly not cacheing any attributes, but instead going back to the server for fresh ones). Then it knows it the file is unmodified, so it can use the pages that are still in-core that contain the content. This is exactly what it would do with a local file as well, if I understand correctly.

So it's not that you want to turn off NFS cacheing so much as you want to modify file cacheing in the VM subsystem so that it specifically doesn't apply to NFS-mounted content. This may be quite a lot more difficult, because you're not trying to turn off something that's already there, you're trying to introduce a whole new distinction that probably isn't there right now, if you follow me.

I think I agree with Andrew Henle that it's sounding like an XY problem. Perhaps you should tell us what you are really trying to do.

Edit: you assert that the same is not true for local file access. However, I think you're wrong. I've connected a HDD to my desktop to which only one process has access, and from that I have twice in a row md5summed a file while using iostat to monitor traffic to and from the physical device.

First time:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdc               0.00     0.00    0.80    0.00    49.60     0.00   124.00     0.08   95.75   95.75    0.00  94.00   7.52

Note the read traffic (rkB/s) from the drive; now the second time:

sdc               0.00     0.20    0.00    0.40     0.00     2.40    12.00     0.00    1.00    0.00    1.00   1.00   0.04

so it looks like there, too, the VM subsystem is getting in the way. So at the moment I see no reason to think you aren't seeing on your netapp all the traffic that clients correctly generate, and I ask again, what are you actually trying to do here? Why is it that you "must disable [cacheing], and send the actual FILE READ from the redhat"? Because at the moment, it really looks like a non-problem.

Improve this answer
18
  • he's trying to test fpolicy, a data feed of who accesses what data on the Netapp.
    – Basil
    Feb 12, 2017 at 1:21
  • Seems that one would be okay with allowing a cache to operate in that case. It's hardly a real-world scenario for an NFS client to have to read from the server for every single file operation on the same file. I agree that it's not at all clear what OP wants to test here.
    – Spooler
    Feb 12, 2017 at 2:00
  • 2
    Why do you care? I'm not trying to be a pain about this, but it's really difficult to understand the underlying issue when you'll only tell us about a symptom - which isn't a symptom, it's correct behaviour.
    – MadHatter
    Feb 12, 2017 at 10:07
  • 1
    @ilansch Doesn't that mean that you've confirmed that there is no problem with your monitoring?
    – Håkan Lindqvist
    Feb 12, 2017 at 10:39
  • 1
    @ilansch I think you could clarify things a whole deal by actually explaining your end goal, a goal which probably ties in with why you are not satisfied despite also saying "there is no problem". Less reads due to caching should be a win-win situation (better performance, less network traffic, less load on the server side), why do you want to disable the cache? Example: Is your goal some form of file access auditing rather than monitoring NFS events?
    – Håkan Lindqvist
    Feb 12, 2017 at 17:58
3

Assuming you're testing fpolicy and are simply looking for a way to reliably generate some traffic, I would skip the OS NFS stack completely and just use a python script using libnfs or something. That you have complete, documented control over when NFS operations are generated. The OS isn't designed to do this, and a quick google tells me that while it should work, it's not obvious and certainly not the default behaviour.

Improve this answer
2
  • Phyton script for what? NFS file sharing?
    – ilansch
    Feb 8, 2017 at 19:58
  • For generating traffic for testing fpolicy on the Netapp. Rather than letting the OS decide when to do an NFS operation, this would allow you to force the issue.
    – Basil
    Feb 10, 2017 at 12:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .