7

I have a server hosting a web site and other services that needs to be reinstalled. I would like to relocate these services to another server temporarily, with as little downtime as possible. Both servers are in the same data center, and can be on the same network switch.

What is the best technique for moving these services with minimal downtime? The site is database-driven, so ideally I want a "railroad switch" event, where I can ensure all traffic is moved to the new server at once. I don't want to have a situation where the old database gets updates after I've migrated the data to the new one.

Two things I have considered:

Change the DNS to point to the temporary service. The major issue here is that I don't control the propagation time for DNS, and other servers can hold on to the cached results for a while, leaving the site "down" for users that get the old address.

Is there a way to fix that problem with Apache + redirects? I suspect not, since name-based virtual hosting breaks without the domain name, which I can't use because it's stale.

Bind the old IP address to the new server and (temporarily) assign the old server a different IP during reinstallation. I can leave DNS alone in this case.

Are there any other simple solutions I am overlooking?

10 Answers 10

4

It sounds like you might best be served with a relatively simple solution ... because you can tolerate a bit of downtime. I would avoid fooling with DNS, because you have little control over the propagation/caching delays.

1- build temp server
2- bring down services on primary server
3- move/copy key data from primary server to temp server
4- change primary server to another IP address
5- change temp server to primary IP address, bring up
6- fix primary server (on different IP)
7- bring down services on temp server
8- move/copy key data from temp server to primary server
9- turn off temp server
10- change primary server back to primary IP address, bring up

The only downtime is when the data is moved between servers, and will vary depending on how the data is moved.

Note: if you have a firewall and are doing NAT, changing the NAT between primary and temp is a good alternative to swapping IP addresses and will reduce the downtime.

2

Providing there are no other services bound to the IP then go with switching that over. It doesn't take long and you can be absolutely sure that traffic is going to the correct destination.

Just be aware of neighbouring machine's ARP caches. It's good practice to use arping -s after the change.

1

Does your server have a public IP on the server itself. If there is a NAT mapping, you can just change the NAT mapping to have the same public IP point to the new internal IP of the new server.

I would think it is often better to have a short maintenance page and test than zero downtime myself.

1

If you have Lan speed connectivity between the 2 systems and full access, using drbd (drbd.org) may be a good option to get the data sync'd between the systems before a cutover and back.

Setup DRBD and let it sync
Shut down db & web server
Switch drbd on original machine to secondary
Switch drbd on second machine to primary
Change original server IP
Add old IP to new server
Bring up db and web server on secondary system

Flip them around when the original system is rebuilt

The option to use database replication is good also if your "data" is primarily in the db

Waiting for DNS propagation even with a low TTL will provide 'inconsistent' results

1

I wrote about how to move web server to another on my blog. Includes a lot, including database issues.

http://mysqlbarbeque.blogspot.com/2009/03/how-to-move-your-web-server-with-no.html

1

Migrating a server.. What a pain.

Luckily, you have everything in the same data centre.

It really, however, depends on what apps etc you've got, and making sure that you've configured all of them on the new box.

Generally, within my workplace, we don't use IP addresses in configurations, we use DNS names. But these DNS names are only ever referred to in /etc/hosts

This means that if we need to change an IP address for something, we just change the hosts file, and everything points at the new location.

Again though, it really depends on how you want to do the switchover - gradual or not. You at least need to make sure that the data sources etc etc are exactly the same on both machines, so, create a replicated slave for a database, etc etc. Basically, when you pull the plug on one machine, the other machine should be able to take over instantly, and in the same state. This is a good reason to keep servers seperate. So don't have one server running all your mail, your databases, your webapps etc. Make sure that you don't have a single point of failure.

Here's how we did a recent server move (from one datacentre to another)

  1. Weeks before, change the TTL on the DNS for the domain(s)
  2. Setup new server, make sure it has exactly the same code
  3. Setup Master<->Master replicated slave in new datacentre
  4. Point everything at old datacentre while testing (SSH tunnels are your friend!)
  5. "Flip the switch" - run a script that changes everything on the all the servers (old and new) to point at the new servers (make sure that you do this all at the same time, or you could get replication issues!)
  6. Migrate DNS records - point the DNS at the new server
  7. Monitor - watch the old servers until traffic tails off.
  8. Decomission old servers

While this isn't exactly a full "blow by blow" of everything we did. It gives a good overview.

0

If you own the domain in question you can to have the TTL for that DNS entry changed to a lower value by your DNS admin(say 3-5 minutes). Let this new setting propagate out over the internet for a few days before you make your actual DNS IP change. This should ensure that any cached DNS entries get updated quickly after your change.

2
  • My main concern here is that I've heard anecdotal stories of large providers ignoring TTL and caching for days or even weeks, but perhaps I shouldn't worry about users on such networks. Jun 1, 2009 at 20:10
  • Disregard AOL at your peril.
    – chaos
    Jun 1, 2009 at 21:16
0

You're right that DNS cutover is completely unreliable. What I prefer to do is, at the same time as the DNS is changed, switch the database configuration of the old site so that it connects to the new server's database. Ta-da, all your updates going to one place.

The site will presumably run slower for people who connect to the old server, of course, but that'll only last until they get in sync.

-1

Use replication on your database servers. That will solve your problem with database updates on both servers in middle time.

-1

Ideally you set the TTL of the DNS host record way down a couple of days ahead of the scheduled time, but if you have no control over that (or can't work with someone on that) then that's shot.

If not, the only thing really is to build the new server until it's ready for production and then schedule a few minutes of downtime while you switch out the boxes...

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .