Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We had an interesting outage today on one of our client's websites. Out of nowhere, the website was inaccessible. The website runs by itself on a dedicated physical Windows 2003 R2 server (probably overkill, I know, but that's a discussion for a different day). After restarting IIS and ColdFusion Application Service, the problem came back several times. My initial thought was that it was a DNS issue, which happens occasionally - the last time it happened was after Hurricane Sandy when we our ISP was out, and we had to make some network config changes. But, it was not a DNS issue. My second thought was that it was a DDOS attack, but, there's very little reason anyone would want to take this site down. When we called our ISP, the operator on the other end noted that traffic was spiking significantly. As it turned out, the client had unintentionally caused a DDOS on the website, after they FTPed a very large video file, and then mass emailed a link to it. Hundreds of people clicked the link and brought the site to its knees.

I am primarily a Website Programmer, but I often have to contribute to server administration at times. Sadly, I'm the resident ColdFusion and IIS expert, but I don't have a lot of experience with this issue. What are some basic steps that I can take to prevent this from happening in the future, since we cannot always control what files the client posts to the website.

Here are some ideas I had, but I'm unsure of the impact:

  1. Limit the number of connections in IIS.
  2. Put media files on a separate server (like an Amazon site, etc.).
  3. File requests of this type currently behind a server-script (i.e. /www.site.com/viewFile.cfm?fileId=1424545, where the fileId references a file off the webroot) that logs requests, and pushes the file to the browser using CFCONTENT. I could edit this script to reject requests when they exceed a certain amount in a given time-frame (i.e. a 5MB can be accessed globally 10 times in an hour). This may cause some users frustration, but, if hundreds of users are attempting to view the file, the site is going to crash anyways, as it did today, which is way more frustrating, since there is no "pretty" message explaining why they can't get to the file.
  4. Update Request Tuning settings in ColdFusion Administrator. Maximum number of simultaneous Template requests is currently set to 20. I could reduce this number to something like 5 just to prevent occurrences like this, but that would likely have an adverse affect on normal use of the website.

I'm open to any suggestions, as I'm continuing my research to report to the CTO with the best options, so that we can put a solution into effect.

Thank you.

UPDATE: Usage Report from the time surrounding the outage:

Usage Report

share|improve this question
2  
Windows Server 2000 is past end of life and hasn't gotten a security update in a year and a half - you may want to consider an upgrade. Anyway, it sounds like your bandwidth was saturated - how much bandwidth is available to that system? –  Shane Madden Nov 30 '12 at 3:04
    
Whatever you do you need to take the user experience into account, not just the technical aspects. Some type of rate/bandwidth limiting might work. As for rejecting requests based on load, that doesn't sound like a good idea. The only reason I'd ever come back to a web site where I was previously rejected is if I had a job requirement to do so. If I can't get what I need from a web site on the first try within about 7 seconds I'm moving on and am probably never coming back. –  joeqwerty Nov 30 '12 at 3:35
    
@ShaneMadden I apologize, we're running Windows Server 2003 R2, not 2000 –  Eric Belair Nov 30 '12 at 13:20
    
@joeqwerty funny you say that, as this is a propietary website for use only by the client - a financial company. It is used primarily for ordering and viewing their marketing materials, but occasionally, they upload a video for their employees. So, the employees really have no other choice but to use the website. That's why option 3 is even up there. –  Eric Belair Nov 30 '12 at 13:23
    
@ShaneMadden The website runs on a dedicated 3 Mbps line. Seems low, but as you can tell from the usage report above, we normally don't exceed 1 Mbps. I guess throwing more bandwidth at the problem could help. But we're already paying through the nose for the 3 Mbps line. I can't imagine the CTO would approve spending more. Looking for a more cost-effective method here. –  Eric Belair Nov 30 '12 at 13:39

2 Answers 2

For large static items, move them to a CDN like Amazon CloudFront.

share|improve this answer
    
There are too many files (tens of thousands) to maintain this way. The files come in automatically via FTP, and they contain propietary information, which I don't think the client would want to exist in the cloud. –  Eric Belair Nov 30 '12 at 18:14
    
Unencrypted proprietary information, transferred over the internet via unencrypted FTP? That's a dangerous combination. Whether you trust the cloud or not, it would be safer to encrypt the files securely and distribute them via some kind of cloud storage service. –  Skyhawk Nov 30 '12 at 18:40
    
I'm only suggesting putting the large ones on CDN. –  longneck Nov 30 '12 at 18:51

Here is the solution I am working with right now:

All of these links to load documents go to one page, with additional information attached in the query string:

/www.site.com/viewFile.cfm?fileId=1424545

I have updated this file to simply contain an iframe that points to a separate site which serves up the media file:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title></title>

    <style type="text/css">
        html {
            overflow: hidden;
        }

        iframe {
            position: absolute;
            left: 0px;
            top: 0px;
            width: 100%;
            height: 100%;
            border: none;
        }
    </style>
</head>
<body>
    <cfoutput>
    <iframe src="http://media.site.com/index.cfm#CGI.PATH_INFO#" frameborder="0"></iframe>
    </cfoutput>
</body>
</html>

That separate site (media.site.com) resides on a different server (which is served by a separate ISP line, which is actually a 15Mbps line but also handles other traffic for the company) and index.cfm on that site contains all of the logic that previously resided in the current landing page, and serves the file up from the local directory which syncs the media files from where they originally reside.

A bit hack-ish, but it gets the job done. This way, if the users overload the requests for media files, only that site/server/IPS-line will go down, but the original website will still remain "operational", so that users can still perform normal functions, even if they can't view the media files.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.