Say you are confronted with 25 TB worth uncompressed log files and you have at your disposal an array of 20 commodity boxes with collective free storage capacity of 25 TB.

How would you store these ?.

a) Which distributed file system to use ?

b) Which compression / decompression format/algorithm ?

c) Log file size is 1MB to max 7MB all text and lot of whitespace

d) Usage is a) people want latest log files more than previous so what caching system to use b) people will only read log files not delete them c) people want listing of log files against a date range

e) Operating system running on the commodity boxes is Linux,

f) As for backup well we have a storage array that takes care of that. So ability to restore data from array exist.

I don't want them to access the file system directly. What should I do ? How do I get them a REST based API for this ?

Please spare you 2 cents and what would you do ?


  • Which operating systems are the commodity boxes running? Do you require fault tolerance, or if you lose all the data stored on one box, is that OK? Oct 11, 2010 at 3:53
  • @farseeker edited the question to answer you questions. Thanks Oct 11, 2010 at 4:01
  • Just re-read the question, and the first question I would ask is: Where are the 25TB of log files stored right now, and can they stay there? Oct 12, 2010 at 4:28
  • @farseeker on a NFS filesystem Oct 12, 2010 at 6:32
  • How about en.wikipedia.org/wiki/… Dec 16, 2012 at 2:50

6 Answers 6


I'm not a distributed file system ninja, but after consolidating as many drives I can into as few machines as I can, I would try using iSCSI to connect the bulk of the machines to one main machine. There I could consolidate things into hopefully a fault tolerant storage. Preferably, fault tolerant within a machine (if a drive goes out) and among machines (if a whole machine is power off).

Personally I like ZFS. In this case, the build in compression, dedupe and fault tolerance would be helpful. However, I'm sure there are many other ways to compress the data while making it fault tolerant.

Wish I had a real turnkey distributed file solution to recommend, I know this is really kludgey but I hope it points you in the right direction.

Edit: I am still new to ZFS and setting up iSCSI, but recalled seeing a video from Sun in Germany where they were showing the fault tolerance of ZFS. They connected three USB hubs to a computer and put four flash drives in each hub. Then to prevent any one hub from taking the storage pool down they made a RAIDz volume consisting of one flash drive from each hub. Then they stripe the four ZFS RAIDz volumes together. That way only four flash drive were used for parity. Next of course the unplugged one hub and that degraded every zpool, but all the data was available. In this configuration up to four drive could be lost, but only if any two drive were not in the same pool.

If this configuration was used with the raw drive of each box, then that would preserve more drives for data and not for parity. I heard FreeNAS can (or was going to be able to) share drives in a "raw" manner via iSCSI, so I presume Linux can do the same. As I said, I'm still learning, but this alternate method would be less wasteful from drive parity stand point than my previous suggestion. Of course, it would rely on using ZFS which I don't know if would be acceptable. I know it is usually best to stick to what you know if you are going to have to build/maintain/repair something, unless this is a learning experience.

Hope this is better.

Edit: Did some digging and found the video I spoke about. The part where they explain spreading the USB flash drive over the hubs starts at 2m10s. The video is to demo their storage server "Thumper" (X4500) and how to spread the disks across controllers so if you have a hard disk controller failure your data will still be good. (Personally I think this is just a video of geeks having fun. I wish I had a Thumper box myself, but my wife wouldn't like me running a pallet jack through the house. :D That is one big box.)

Edit: I remembered comming across a distributed file system called OpenAFS. I hadn't tried it, I had only read some about it. Perhaps other know how it handles in the real world.


First, log files can be compressed at really high ratios. I find my log files compress at a 10:1 ratio. If they compress to even a 5:1 ratio, that's only 5GB, or 20% of your storage capacity.

Given that you have more than enough storage, the specific compression algorithm isn't too important. You could...

  • Use zip files if Windows users will be accessing the files directly.
  • Use gzip if they'll be accessed through Linux and quick decompression is important.
  • Use bzip2 if they'll be accessed through Linux and it's important to have the smallest possible files.

The bigger question is: how are you going to provide your users with easy access to these files? Part of this depends on how your machines are configured.

If you can put enough storage into a single machine, then you can do something extremely simple, like a read-only Windows file share. Just organize the files in subdirectories, and you're ready to go.

If you can't create a single file server for these files, then you may find that you need a distributed filesystem. Windows has a Distributed File System (DFS) which might suit your needs.

If your needs are more advanced, you may want a web application as a front-end where your users can browse and download log files. In this case, I recommend using MogileFS, which is a distributed file system designed to be used with a front-end application server. It's very easy to integrate with most web programming languages. You can't mount it as a shared drive on your computer, but it's top-notch as a data store for a web application.

  • FYI: Windows DFS is a way to keep files/folders on multiple server in sync. It won't allow you to use the storage on multiple servers as a single storage drive. microsoft.com/windowsserversystem/dfs/default.mspx Oct 11, 2010 at 21:39
  • After thinking about it, you’re right; DFS may be able to be used if you have a DFS root point to the folders living on other machines. That way the user would see one file structure and wouldn't need to know which machines the data actually lives on, DFS would know. That would work. Usually when I have people ask me about Windows DFS, they usually think it is a way to pool together storage space, and that is why I just to that conclusion. Sorry and your right that could work. Oct 11, 2010 at 22:17

lessfs is a deduplicating, compressing file system. While it won't solve whole problem, it may be worth a look to look at as a backend.


export these folders via NFS

mount them on a single machine with apache running (under document root) as a tree

use zip to compress them- good compress ratio, zip can be opened from all OSes

list files in Apache -so you are giving users readonly access (log files are not suppose to be edit, right)

  • 1
    Agree on nfs+httpd, disagree on zip. gzip interacts way better with http.
    – Tobu
    Nov 14, 2010 at 2:54
  • +1 for gzip comment from @Tobu - With the right configuration, Apache can serve up gzip'ed files to a web browser that will transparently decompress and display them. Users don't even need to know about the compression. Apr 2, 2012 at 21:18

You ever think about compressing the log files? Then do something on the frontend to uncompress them before serving them to the end user. Maybe a CGI script of sorts.


@Ankur and @Porch. I strongly agree with the necessity to compress these logs.

@jet I think the simpler scheme is better - thus httpd for end user is close to ideal. And backend could be any.

My opinion - divide logs into 2 groups - folders 'old' and 'new'.

Merge them into document root of httpd. Use strong compression for old ones (either xz or 7z archives, popular for all OSes) with large dictionary and block sizes, may be even solid archives.

Use compressing fs for new ones: lessfs(rw,deduplication+light compression methods),fusecompress 0.9.x(rw,light to strong compression methods),btrfs/zfs, squashfs(ro,light to strong compression methods,some dedup, use for newly rotated logs).

You can even transparently write logs into compressed fs (fusecompress,lessfs,btrfs/zfs). Provide R/o access by httpd to logs being written. They will be transparent for users and transparently decompressed for them.

Warnings about fusecompress: 1)use only 0.9.x - it is stable. Clone from here https://github.com/hexxellor/fusecompress

Later versions either don't support lzma well or lose data.

2)it uses only 1 cpu core for compressing one file, thus may be slow.

Recompress each log in 'new' folder, older than some time (several months) and move to 'old'.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .