Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I currently run a small website which is being used to host large amounts of (generated) static HTML. The problem is that disk space is limited, and the HTML is growing by 1GB a week. (The files are grouped into directories, with around 500 files each, and total to 10B-100MB. The files compress very well [under 10%])

Ideally I'm looking at a way to compress all the HTML files on the HDD while still being able to serve them easily.

share|improve this question
1  
One possibility is to bzip the directories and use Fuse to mount each as a directory, but I would prefer a not-so hackish solution –  Matthew May 28 '09 at 3:20

7 Answers 7

up vote 6 down vote accepted

The mod_gunzip mentioned by Matt Simmons doesn't appear to exist for Apache 2.x. The replacement mentioned by the developer is to use Apache Module mod_ext_filter. I haven't tested, but it looks like it should be pretty easy to build a filter that decompresses files to be served.

The other thing you should consider is that most current browser accept content that has been gzip compressed. It may be possible for you to gzip your files and serve the gzip'd files without doing anything special. Here is a link with some details.

share|improve this answer
    
Thankyou, those links were very useful. –  Matthew May 29 '09 at 2:18

Have you looked at mod_gunzip? I'm too new to link to it, but a google search should point you in the right direction.

share|improve this answer
    
Is this the url? (oldach.net) it has both a mod_gunzip and mod_bunzip2 package. These packages appear to be for apache 1.x only. –  Zoredache May 28 '09 at 5:29
    
You're right. I did find this while researching, though: log.samat.org/2005/10/06/… –  Matt Simmons May 28 '09 at 11:48

Compressed Filesystem - Another solution is to deal with the compression a layer below the web server and files - at the filesystem level.

I've not done with myself, but you can try out something like fusecompress - so separate out your www if you already haven't, and make it a compressed fs of some sort.

Obviously this will cost you in some performance, but if the processor is decent, then it might be ok.

share|improve this answer
    
I looked at Fuse solutions, but they all seemed to be very young, and not really ready for general use –  Matthew May 29 '09 at 2:19

You could wrap all of your pages in a script that looks something like this:

bzcat $1.bz2

Where $1 is the requested file. A quick PHP/Perl/whatever script can pretty effectively pull the path out of the request variables, and there you go.

You do lose the speed of static files, but that might not matter for your use case.

share|improve this answer

One answer would be to run the website on a Windows host and simply compress the NTFS filesystem

Another option would be an OpenSolaris system running ZFS

share|improve this answer

Most browsers understand gzipped HTML pages. One solution is to gzip each page and to have your web server add a "Content-Encoding: gzip" header to each response.

share|improve this answer

gzip all the files and use Options +MultiViews if using Apache.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.