3

Here's the environment:

  1. Website that hosts a forum/journal/bboard/email/socialmedia application in walled garden (ie you pay to get to use it or are invited to do so

  2. Many Clients pay to use the site during specific chunks of time (ie they lease access to site) in order to interact with their clients. There are dozens of clients in a broad range of fields.

  3. There is a very broad service level agreement. Meaning that it's not stated that the website can't go down for more than ten minutes but there's a gentleman's agreement that it won't. They don't pay for the 24/7 support be we give it to them because we love what we do.

  4. Site runs in 7 different languages throughout multiple time zones.

Here's the situation:

The site goes down at 5:30EST and stays "offline" for approximately two hours due to DDOS attack. The clients reactions vary from annoyed to livid. The clients are also not very tech savvy. The clients are accustomed to 24/7 support and typically receive great support.

Here's the question:

How much to you divulge to the client about the DDOS attack? They want a reason as to why the site went down.

Improve this question

7 Answers 7

Reset to default
10

Be honest. A DDoS attack is likely to be beyond your control (or at least beyond your ability to predict).

If it is a DoS caused by a bug in your code (or by someone exploiting a bug in your code specifically to create a DoS) then things get more difficult as there is blame that could be sent your way, but for a DDoS that is genuinely beyond your control then honest is definitely the best policy.

If your users want an uptime policy that states "won't down down for longer then X in Y or for any period longer than Z for any reason" then they need to be paying you for a service level agreement that states those rules rather than living on a gentlemen's agreement.

Improve this answer
5
  • @David - you're right on about there being a bug in our code. Once the attack fired off it was compounded by our code sending off thousands of email alerts to our team.
    – Patrick R
    Jan 26, 2010 at 16:59
  • +1 for uptime policies. Uptime isn't cheap and clients need to know there is a price attached to it.
    – David
    Jan 26, 2010 at 17:33
  • 3
    +1. Amen! I'm shocked sometimes at how many customers expect 99.999% uptime but don't expect that there's any cost associated with building and managing a system to meet that requirement. We lay it out to our customers when they ask us for it. Us: "Yes, we can provide that level of uptime to you, and here's what it's going to cost you". Customer: "On second thought, we're happy with our current SLA".
    – joeqwerty
    Jan 26, 2010 at 18:12
  • 2
    Even if a bug is part of the problem I would still be inclined to be honest. "X happened, it was compounded by bug Y, and we've done Z to try ensure it doesn't happen again". Admitting a small flaw now will probably look better than being found to have been economical with the truth later, or just looking undefinably unreliable, especially with the "what we've done and/or plan to do" part.
    – David Spillett
    Jan 26, 2010 at 19:22
  • I'm going to open this one back up to see if it gets some more traction. +1 for the time being.
    – Patrick R
    Feb 21, 2010 at 20:38
4

IMHO, be straight forward with them. Explain to them what you believe to be the cause of the outage. Explain what you're doing to analyze\verify the cause of the outage and what you'll do to try to prevent it in the future. Even the largest, most technically savvy entities have problems: Microsoft mucking up their DNS, TechCrunch getting hit with a DDOS attack, Facebook accounts being defaced, the Washington Post letting their domain name expire, etc., etc.

If you've performed due dilligence in securing your site\assets then that's all a customer can ask of you as far as I'm concerned. IMHO, honesty and straight talk are the best policy.

Improve this answer
2

I've always been in favor of maximum transparency. I was impressed by FogCreek's openness in their reporting of an unplanned outage of their hosted FogBugz service a couple of years ago. They didn't have to tell us any of that stuff, but honesty builds trust.

Improve this answer
1
  • 2
    +1. Nothing irks me more than hearing some service provider (Google mail, Blackberry) respond to an issue or outage with a press release stating "A small subset of our users experienced some minor delays in accessing our service". Frankly, I'd be more open to forgiving them if they would just be upfront about it.
    – joeqwerty
    Jan 26, 2010 at 20:49
1

Tell them the truth, and divulge as much as you feel they need to know, without getting too technical.

Improve this answer
0

Sorry if this sounds harsh I would tell them that there was a general network problem and get on with fixing the problem rather than asking ethics questions on here - that stuff can come later once the problem's fixed.

Improve this answer
2
  • @Chopper3 - We found a solution to the problem and are implementing the fix now.
    – Patrick R
    Jan 26, 2010 at 17:00
  • Cool, I'd just hate to find that clients were suffering because a sysadmin wasn't focussed on their problem :)
    – Chopper3
    Jan 26, 2010 at 17:33
0

My company operates on several "core values", and one of them is "Bad News First - No Surprises." IMO, tell them it was a DDOS, as much as you can without being overly technical.

Improve this answer
0

Everyone wants honesty, and to be spoken to like a mature adult.

However, there are no mature adults. If customers learn that your site went down due to a DDOS attack, and you explained very clearly to them what you did to remedy the situation, they'll thank you for being straight forward with them and then they'll start looking for a vendor who hasn't done whatever it takes to be a target of DDOS attacks.

Now I know that you did nothing to deserve an attack, and so do you. These things happen. Your customers probably know it too. But some guy at some company who made the decision to spend money on you is now in the predicament of having to explain to his superior why he made such a bad decision.

Observe:

Statement 1: "Sorry I'm late, I had some car trouble."

Statement 2: "Sorry I'm late, somebody slashed my tires."

Which one of these statements do you want to hear from a person you're giving your money to? This is why corporations never tell the whole truth even though we demand they do.

Tell them you had a big network outage and that you really worked your butt off to get it worked out. This is what happened, and it's easy for anyone to understand, regardless of technical prowess. I'm not recommending that you lie. Just don't offer too many details unless you're pressed to do so.

(note: if you had been hacked and user data was at risk, that's when I would switch over to a more up-front tell-all policy)

Improve this answer
3
  • @Boden - I see your point about balancing too much and too little info. And thankfully no data was at risk in this case.
    – Patrick R
    Jan 26, 2010 at 18:16
  • @Boden: You make some good points and unfortunately there will be customers who will leave as a result of this type of event. IMHO though, most customers are reasonable and will respond favorably to an honest, upfront explanation. We have several customers who respond negatively to every effort on our part to satisfy them. That's just an unfortunate aspect of doing business these days. We focus on giving our very best to every customer and if we can't make a select few happy, we can at least rest assured that it was not due to a lack of effort on our part.
    – joeqwerty
    Jan 26, 2010 at 18:19
  • Certainly. You know your customers best, and which ones you can be completely open with. However, for customers that you don't have a strong relationship with, it's all about impressions. The more information you offer, the more questions you're creating. From the customer's point of view it doesn't matter if the problem was somebody tripping over a network cable or a DDOS attack, as long as you fix the problem quickly and offer up some something to apologize for the inconvenience.
    – Boden
    Jan 26, 2010 at 18:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .