Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

The easiest answer is CDN but I'd like to ask. A friend of mine has a server that is used for mirror downloads. He says he is doing about 10TB of bandwidth a month which shocked me (I wonder if he is lying).

I seen his site and he has no ads. I suspect he might close his website once he gets the bill. Anyways I was wondering since his CPU/RAM is not being used and his HD usage is around 15gb what he can do to lower cost if he continues this site. I said put up ads but I don't know if ads would cover it I found one CDN which offers $0.070 / GB. 10240gb (10TB) * .07 = $717 a month. That seems a little steep but he is using lots of traffic due to it being a mirror site.

Also using a CDN doesnt make sense as he doesn't need multiple servers hosting the files in different areas (which is one reason he isn't using that now). He just needs a big upload pipe

Is there something he can do? At the moment he is paying $200 a month on a dedicated server and he is using WAY more bandwidth then he should be using.

Side question: Can gz-ing files large already compressed files help? like on (zip, rars, etc)

share|improve this question
2  
You may be able to use of BitTorrent to reduce the bandwidth - some cloud storage providers (e.g. Amazon's S3) will let you easily create torrent files that are seeded from their server - it should be possible to implement the same idea using your own server as a seed. The advantage here is that for sizeable content that is being downloaded hundreds of times each month, each downloader may contribute some bandwidth to the total required. Of course, depending on the nature of the mirror, this might not be feasible. (If the files are video files - streaming will save bandwidth over downloading) –  cyberx86 Nov 17 '11 at 1:03
    
@cyberx86 excellent answer. You should have made it an answer. I also found out theres such a thing as ummetered VPS which seems to solve the problem –  acidzombie24 Nov 17 '11 at 1:07
    
Common unmetered plans typically come in 10Mbps and 100Mbps - with 100Mbps being much more expensive. To transfer 10TB in a month you need: (10TB)*(1024^4B/TB)*(8b/B)/((730hr/mo)*(3600s/hr)) = 32Mbps. That is for a constant transfer (and represents the minimum bandwidth needed to transfer 10TB/mo). Since you won't be uploading data at a constant rate, you will need more bandwidth - probably a 100Mbps connection. (At $200/mo for primarily serving static files, it is possible that the server is already unmetered - a quick search does return 100Mbps unmetered results in that price range) –  cyberx86 Nov 17 '11 at 1:43
add comment

3 Answers 3

up vote 1 down vote accepted

Before looking into CDN, I would validate the site is optimized correctly. The very first thing is to check your logs and look at what content is using the most bandwidth.

You don't state the nature of the site, but if the majority of the bandwidth is coming from people downloading zip and rar files, then there may not be too much you can do to optimize. Again, without knowing the nature, it's very hard to suggest anything here, but it may be worth considering:

  • Are the files actually necessary? Eg, are the people that are downloading them actually looking for the complete file? Are there cases where they're downloading the zip file just to view one small piece of content inside?
  • Is there an alternative to them downloading? Can a preview or viewer be provided in the browser that removes the need to download files?
  • Can you use bittorrent to host the files? Your server can seed it, but the more people downloading/seeding, the less traffic you'll have to handle.

As far as optimizing the actual site:

  • ensure that static content (images, javascript, css) has appropriate cache headers (which ensure that browsers don't continuously re-download them)
  • Same goes for static HTML, and even dynamic HTML if the content isn't changing.
  • Minify your css, js, and html.
  • Make sure your files are split appropriately (eg, don't include css/js directly in HTML files -- especially if the HTML is dynamic)
  • Use YSlow to verify all of the above, and point out more optimizations

As @databyte says, compressing something (gzip) that's already compressed won't help.

CDN are designed primarily to offload handling of static content, and improve performance of your site. They do this partially by alleviating load and letting your server concentrate on serving the dynamic content (sounds like load is not an issue for you, though). They also put servers on very fast networks, and locate them at ISPs. The reason for this is to make the speed as fast as possible, and latency as low as possible. If an end-user accesses content from a CDN and the content is served directly from their ISPs POP, that's the shortest possible path: it's going entirely over gigabit network in the POP and then directly to the user -- nothing ever travels across the actual "internet".

share|improve this answer
add comment

Compressing a file twice will not make it smaller(1). The person downloading the file may not know that you meant to compress it twice as well - imagine their amusement in having to decompress it twice.

Since you're mirroring content, I don't recommend recompressing all the files to a different format. However if you had your own content, that's an option. Take everything, compress it with different formats, and simply pick the smallest size. Different file types may end up doing better under different algorithms, say zip vs 7z.

As for lowering the costs of mirroring content, pushing down 10TB is a lot - period. I doubt you could beat $200 using a proper CDN that distributes the content over multiple servers across different locations.

If the files are only 15GB but you're simply transferring them a lot - you could build your own CDN. Setup several cheap hosts on different platforms, rsync the files, and then have one web site pick the closest mirror. Most shared hosts give you enough space and probably could handle 1TB /mo @ $5-10/mo.

Not a proper "CDN" but it's surely cheaper than $700.


(1) Try taking a large file and compressing it twice, it should vary only by +/- 0-1%. I tried it on two files myself just now and one was about 300KB smaller on a 165MB file.

share|improve this answer
    
I didnt mean actually re-compress the files. I meant gz-ing so the browser which supports gz receives it gz compressed and saves it normally on PCs. Thats the thing he doesnt need the files to be on multiple servers which CDNs do. He just wants a really big upload pipe –  acidzombie24 Nov 16 '11 at 23:32
1  
@acid gzipping a file is compressing an already compressed file. The algorithm is different, but the end result will generally be the same. –  MDMarra Nov 16 '11 at 23:46
add comment

I will make sure that the the server supports Range HTTP request header. And that the browser caching is allowed by the server (See: http://www.mnot.net/cache_docs/#TIPS). Use gzip and/or deflate compression for the content that is not already compressed (e.g. text files, html, css)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.