Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I know very little about DFS filesystems but have come across an issue with one of our deployments.

Our application writes files to a designated location, closes them, and then writes a record into the database. Another part of the application picks up these DB records and reads the file that was previous written.

In some cases the reader is getting a "file not found" and it fails. Restarting it without touching anything else and it finds the file correctly and everything is fine.

I believe I have ruled out a problem with our application as the file is definitely flushed/closed before the database record is created.

Therefore I'm led to believe that the OS or filesystem is delaying the file write internally so it isn't immediately available.

The filesystem in question is Windows 2003 SP2 DFS. Is this a likely scenario with this DFS? If so is it possible to switch it into some sort of write-through/no caching policy to ensure the files are written promptly?

share|improve this question
1  
What is the client OS? Are there multiple replicas configured in the DFS? –  Harry Johnston Sep 27 '11 at 3:39
    
@Harry, client OS is Windows 2008 R2, multiple replicas - not sure but will find out shortly –  Mike Q Sep 27 '11 at 9:00
    
The reason I ask about the client OS is that we've experienced some similar issues which I think indicate a bug in file sharing which may have been introduced with Windows Vista or Windows 7. I'm not sure yet whether the problem is at the client or server end. –  Harry Johnston Sep 28 '11 at 20:55

2 Answers 2

up vote 3 down vote accepted

DFS is Distributed File System, which is exactly what the name says: a "virtual" file share that is distributed and replicated across multiple servers. Everytime your application writes to it, it's actually accessing one of its copies on one of the servers that are part of it, and if another application tries to read the same data soon after, it could very well be accessing another server, which didn't receive the updated data yet.

With DFS, you can't never be absolutely sure that data written to it will be available on a subsequent read: there could always be replication latency; you also don't have any way to tell your application to "talk" to a specific DFS server: it is free to connect to any one of the servers running it.

If you want this application to work in real time, you should use a standard file share, not a DFS.

share|improve this answer
    
More info. The site is apparently using DFS in a non-replicated fashion. e.g. dfs/mydir1 on server1, dfs/mydir2 on server2. If they are not using replication does the above still hold true? Can there still be a delay in the filesystem when writing? –  Mike Q Sep 28 '11 at 7:36
    
Are the first and second applications running on the same server, or on different servers? –  Massimo Sep 28 '11 at 8:09
    
The file is being written to and read by the same application/process on a single machine. –  Mike Q Sep 28 '11 at 8:18
    
Ok, this is quite strange then. Even if DFS replication and/or caching was involved, this shouldn't happen. Are you absolutely sure the database record is created after the file has been written and flushed? –  Massimo Sep 28 '11 at 9:21
    
99% sure yes. We have not observed this on any other deployments which use exactly the same setup except for the filesystem difference. Btw are you saying that even with DFS replication if you access the file from the same machine/process then you should always see an up to date file? –  Mike Q Sep 28 '11 at 10:59

You are making the common, but incorrect, assumption that there is one universal notion of "after" and that if you do one thing "after" another, you are guaranteed to see the effects. This is simply a false notion, and nothing you can do will ever make it work the way you expect.

The analogy would be sending someone a letter, getting back a return receipt, calling the person on the phone and assuming they must have read the letter.

As you mentioned, delayed writes will screw this up. Many other things can screw it up too. Trying to find every possible way it can break and fix them all is just crazy.

Instead, if you need ordering between operations, use something that is specifically guaranteed to provide the particular ordering you need. Since there is no guaranteed ordering between the filesystem and the database, it won't do.

Most filesystems do provide guaranteed ordering with respect to themselves and their own operations when accessed through processes running on the same operating system instance. So after the file is correctly set up, you can create a 'trigger' file in the same filesystem. If the reader sees the trigger file, then it can know that the data file is complete and valid. It can remove the trigger file when it's done.

share|improve this answer
    
This is not true. Of course the O.S. (or the controller) might be caching disk writes, but any decent caching algorithm will make sure to return the most up-to-date result on subsequent reads, even if that data only resides in che buffer cache and haven't been physically written to the disk yet. –  Massimo Sep 26 '11 at 22:07
    
@Massimo: That's exactly why I said, "Most filesystems do provide guaranteed ordering with respect to themselves and their own operations when accessed through processes running on the same operating system instance." What do you think I said that is not true? –  David Schwartz Sep 27 '11 at 1:41
    
You said "there is no guaranteed ordering between the filesystem and the database"; this is not true, and if the client application was reading from the same filesystem, it would indeed find the data it was looking for. What's screwing up things here is the fact that he's (unknowingly?) using a distributed and replicated filesystem. This is like saying "it's normal to create an Active Directory account and not see it after a while"; no, this is not normal per se: it can happen only if you look for it on a different DC and in a different AD site, before replication takes place. –  Massimo Sep 27 '11 at 5:57
    
Why is that not true? What standard, documentation, or argument provides the guarantee that there's ordering between the filesystem and the database? And the second part of your comment is really, really bad advice. Ordering simply isn't guaranteed. If it was, what he was doing would already work. Trying to synthesize a guarantee by avoiding everything you suspect might cause your method to break is suicide. The environment can always find some other way to break what you did. That's why standards provide guarantees -- so you don't have to guess and hope. –  David Schwartz Sep 27 '11 at 6:00
    
This is not a problem of "ordering between the filesystem and the database"; he's not doing both things at the same time. He's doing a filesystem write, closing and flushing it, and then he's doing a database write. The filesystem write is already done when he accesses the DB. The problem is, since this is a distributed filesystem, succesfully completing a write operation is not enough, as then the data needs to get replicated to other servers, and this can take a while. –  Massimo Sep 27 '11 at 6:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.