Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Is there a way in Linux or any other *NIX platforms to have a remote filesystem mounted locally backed with a local filesystem backed caching.

I have some archive data on S3 that I need to access. The way they are accessed is; once touched application(s) need to read it off S3 several times.

I know I might be asking the file-system to to do the applications job of caching data; but the trouble is I do not have control on the application to modify it.

share|improve this question
add comment

4 Answers

up vote 2 down vote accepted

This used to work well under Solaris using CacheFS; local HDD space was allocated as the buffer, in which the contents of frequently-accessed NFS-mounted files were cached, and from which they were served if the CacheFS service determined that the contents on the NFS server remained unchanged since they were cached. When I used it under Solaris, it worked well.

According to the Wikipedia page on CacheFS, it was ported to (rewritten for?) Linux, with the most recent release having been in the past year.

I can't comment on the Linux implementation, but assuming it does what the Solaris one used to do, I think it may be what you're looking for.

share|improve this answer
    
Thanks @MadHatter! Thats exactly what I was looking for! It went much further that what I expected. Per application profile for cache management... beautiful! Its a shame that the Linux port is lagging behind! More info here code.google.com/p/cachefs/wiki/ApplicationArchitecture –  CodeMedic Mar 28 '11 at 13:51
add comment

s3fs has a caching option.

If enabled via "use_cache" option, s3fs automatically maintains a local cache of files in the folder specified by use_cache. Whenever s3fs needs to read or write a file on s3 it first downloads the entire file locally to the folder specified by use_cache and operates on it. When fuse release() is called, s3fs will re-upload the file to s3 if it has been changed. s3fs uses md5 checksums to minimize downloads from s3. Note: this is different from the stat cache (see below).

Local file caching works by calculating and comparing md5 checksums (ETag HTTP header).

The folder specified by use_cache is just a local cache. It can be deleted at any time. s3fs re-builds it on demand. Note: this directory grows unbounded and can fill up a file system dependent upon the bucket and reads to that bucket.

share|improve this answer
add comment

Have you considered something like mirroring the filesystem using a filesystem like DRBD? If not DRBD then there are others that could be used, it might give you a jump-off point from which to Google.

It's not caching in the real sense but it would duplicate the data locally and transmit changes to a remote system. Whether it's practical or not depends on how many changes you make to the files and why you are using S3 in the first place.

share|improve this answer
    
DRBD lloks very tempting, but it abstracts the block device as I understand. What I am after is something that abstracts the file-object. Again, the S3 already provides the network / remote file-system via s3fs. What I am after is some kind of glue ( similar to Unionfs). Thinking of which; Unionfs might be the answer to this. –  CodeMedic Mar 28 '11 at 13:34
    
I'm not an expert in it, but it does abstract the block device so you'd have a local copy to work on and the changes are synced up. You'd see a speed improvement since you're working on a local copy, which would be sort of what you were looking for with caching. –  Bart Silverstrim Mar 28 '11 at 13:36
add comment

There is sshfs which might be what you're looking for. My cursory understanding of sshfs is that it's a bit like using encrypted NFS mounts.

share|improve this answer
1  
@Johnathan: You need not post a "sig" or tagline in your answers. –  jscott Mar 28 '11 at 12:45
3  
encryption != caching –  Iain Mar 28 '11 at 13:08
    
Fair enough, points taken. –  Jonathan Ross Mar 28 '11 at 13:48
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.