I'm a moderate web developer. I haven't managed any high traffic websites. Generally, I observe that only high traffic websites are down for maintenance. stackoverflow.com will also go down for maintenance.

I always wonder. What kind of maintenance do they do? I mean, the process is automated.

user request --> web server --> server side programs --- > Database server.

What is there to maintain?

  • There's always something... – Justin Ethier Aug 9 '10 at 18:31
  • Data model change, update/correct data based on those changes... – OMG Ponies Aug 9 '10 at 18:32
  • 11
    Yes, fully automated. Sysadmins don't really do anything. We just collect salaries. – John Gardeniers Aug 9 '10 at 18:45
  • 3
    Monkeys need food/water and sometimes sleep, hamster-wheels need oiling, etc. – jscott Aug 9 '10 at 18:46
  • 2
    @claws, every job looks easier when someone else is doing it but consider that while very few programmers are sysadmins, many sysadmins are also programmers, so we see both sides and generally show more respect. As for that automation, remember it's an admin that designed and implemented it. – John Gardeniers Aug 9 '10 at 23:05

Usually the highest traffic sites don't go down for maintenance. They're designed so they don't have to. (Depending on the site, that can be very tricky. It's not just a case of running multiple servers, although obviously that's the starting point.)

However, usually "Site down for maintenance" means any of:

  • Web application software upgrade (adding new features etc)
  • Hardware change (e.g. moving to a different data centre; during the switchover)
  • Something's gone terribly wrong and they're trying to fix it (e.g. there's been a power outage at the data centre; change the DNS entry to point to a static "site is down" page elsewhere until the power comes back)
| improve this answer | |
  • 1
    In that case, why do they regularly take it down for maintenance? Software/Hardware changes are not regular. – claws Aug 9 '10 at 18:34
  • 1
    With financial sites, there's often a lot of data to move, and at end-of-day, they want an atomic change (plus things need time to be calculated and updated) – Matt Simmons Aug 9 '10 at 18:46
  • 2
    Claws - My own bank is a small US-based credit union - their site is down every night for a few hours. It's a small bank, they don't have a big budget, so maybe they took everything down so that they can do offline backups, or do a big data load. Who knows? There's a lot of reasons. – mfinni Aug 9 '10 at 18:48
  • 1
    If they need to take offline backups tehy are IDIOTS. Every database systm orth using supportrs transactionally consistent online backups - for the last 20 years or so. – TomTom Dec 6 '11 at 5:29
  • 2
    @TomTom For some banks, "the last 20 years or so" in our world are their distant future. – ceejayoz May 27 '15 at 1:35

They may want to run updates (or fixes) on many of the different pieces of software running on the server, including (but not limited to):

  • The operating system
  • The webserver software iteself
  • Any scripting frameworks
  • Databases
  • Etc

Beyond that, they could also be doing hardware maintenance, such as adding a new hard drive, upgrading a motherboard, putting in faster RAM, or swapping out network cards. There's plenty of things, both hardware and software, that can be upgraded or modified, really.

Now if they have a backup server (or a cluster or something of the sort), this can be transparent, but if it's literally one box serving the pages...well, it pretty much has to go down.

| improve this answer | |

Since you're coming from a coding background, I'll base my analogy there. Imagine that being a sysadmin is just like programming, except you'll be called on to code in a different language every couple of hours. And sometimes it's Pascal.

Truly, though it could mean anything. Sometimes a mouse chews its way into a warm place. Or a single point of failure makes itself known. Eliminating downtime is what we pursue ... like writing code that works perfectly on the first compile.

| improve this answer | |

Liken a single server to a running vehicle. If you turn off the vehicle, your 'server' is down.

There are some things you can do while the car is running - add fuel, oil, washer fluid, clean the windshield, change gears, etc.

However, you can't replace the fuel line in the car while it's running - liken fuel to data; you don't want to lose any, or you'll have unhappy customers.

These downtimes vary based on the level of administrator skill and the complexity of changes. On larger, high traffic sites - the only way this could feasibly happen is if there's a major architecture change; something that, no matter how many servers and redundancies you have, the architecture needs to change all at once.

This is rare for very large systems - I liken it to replacing the fuel line on a running vehicle: for many, it's not feasible to do (or worth the effort/risk) at certain skill and resource levels. However, for places that have the skills and resources, they can perform a fuel line replacement on a running vehicle. Liken that to architecture migration; they do it a lot more complex.

| improve this answer | |

Could be upgrade of servers, frameworks, databases Moving to a new datacenter and shutting the old servers own so that nobody can connect Patching of operating systems or software that runs on those servers

basically anything that could make the site unavailable for a certain amount of time

| improve this answer | |

Regular maintenance items would be things like rebuilding caches, upgrading software and/or templates, doing some data trawling for statistics, various routine maintenance tasks like backups, (which work better on quiet systems) and a variety of other expensive, infrequent tasks.

Some tasks just require pouring over a lot of data, and it's not really efficient to do after each change. Recommendation databases are one thing that comes to mind, as you don't need up to the second data, and it's rather expensive to calculate common purchase patterns across many different users. This is an N^2 complexity problem with some algorithms, and tends to take both a lot of data trawling, and lots of memory.

Financial institutions may use the down time to calculate and make interest payments to accounts, or close outstanding transactions and calculate reconciliation balances. This data in theory should never change after reconciliation, so it makes sense to write it to WORM storage at this point.

Backups are a major item that's often done during downtime because high Disk I/O tends to bring even very powerful servers to their knees, and taking the site offline can help speed the backup process. I remember one organization I was at, where they had a very large customer RAID array, and the backup team kept complaining because their backup window for this one customer typically extended 22-24, and at one point 26 hours. A small amount of quiet time can decrease that window substantially.

| improve this answer | |

Defrag the disk arrays. Its faster and safer to defrag servers when they are offline, allowing the CPU and disks to focus on that task rather than running 1000 websites. Its better to tell people to come back later, than to give them a poor user experience.

If its a windows server, you can crash it by running defrag while memory usage is over 50%. This is because at this point windows starts to rev up the page file. I learned this the hard way.

| improve this answer | |
  • 8
    I have no idea what you're talking about... I don't know anyone who defrags arrays or SANs, or even DAS or anything these days. Additionally, I've also never heard of crashing because of page file related activity. I suspect your server might have had bigger issues... – Mark Henderson Dec 6 '11 at 5:16
  • What Mark said -- I think you're running on some very outdated (hell, positively ANCIENT) information there... – voretaq7 Dec 6 '11 at 21:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.