Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm about to start developing a web app. It will be hosted on dedicated servers in the cloud when its time to go live. Currently my development environment is my shared hosting account. I'm assuming it would be more difficult (and pointless) to mimic a shared hosting type of environment because of its heavy customization, restrictions and intended purpose (serve hundreds of sites instead of one).

Considering that I will have complete/root control of the production server(s), should I set up my development environment on a physical or virtual server instead? If so, do I simply convert/image/import my VirtualBox/VMware virtual dev machine (or P2V) to the virtual machine in the cloud or would I have to manually mimic the environment (i.e., install the same exact OS, software, libraries, components, patches, etc.) of the dev machine (and copy/SVN the site & DB over)? If not, what is the best practice for setting up dev servers for easy live transition?

Thanks in advance

share|improve this question
    
What's your development language and framework? There are some nice tools that are language specific ... –  Joseph Kern Mar 30 '11 at 18:46
    
PHP/MySQL. Using CodeIgniter for now –  crashintoty Mar 30 '11 at 19:18

4 Answers 4

up vote 1 down vote accepted

Using your shared account for dev work is probably the worst way to go, as it's unlikely to be anywhere near the same configuration as the final live server(s)e. I suggest setting up a local machine to be as close a clone of your final target configuration as possible. In regard to physical vs virtual, if virtual provides adequate performance for your needs there is no reason to go physical.

If you create a local clone of your target machine, when the time comes it's literally only a matter of copying the files over, ensuring of course that permissions are set correctly on the target. Then do a dump of your database(s) and load them on the target. It just doesn't get much easier than that.

share|improve this answer
    
Exactly what I figured! Thanks! –  crashintoty Mar 31 '11 at 22:14

First and foremost, keep a running log of any modifications that you do to your server, this is called a runbook. Some things to manually keep track of:

  1. Version levels of important packages. Language, libraries, and database packages.
  2. Locations of all important directories and paths.
  3. List any external dependancies.

Second, use scripts to define all of your environmental variables and system setup. Failing this, at least keep notes.

Third, keep as much (if not everything) under revision control, including your runbook, and environmental scripts. You should be able to deploy a new instance of your server from just your SVN repository.

Fourth, keep backups, keep, really, really good backups. Test them.

The rest is very specific to your deployment. Setting up on a virtual or physical server is entirely dependent on your IO (input/output) load across your network, disk, and memory. This can only be quantified over time, so graph as much as you can.

Above all, make things modular and repeatable.

(Thanks @bittrance for the comment!)

Here's a few examples of "use a script to setup your environmental variables."

share|improve this answer
1  
I thought that was a changelog. I thought a runbook was documentation for normal operational procedures (to keep things running.) –  mfinni Mar 30 '11 at 19:07
1  
I disagree with this post. Do not rely on backups for anything that is not user/app data. Backups imply that you treat the state of your box as a black box a la "whatever is in there works". In essence, the only documentation that can be easily verified is executable code. Do not document env variable changes: write scripts that makes sure that they are set to their proper values and make sure that those scripts are installed/run every time. But, yes, do revision control everything. –  Bittrance Mar 30 '11 at 19:11
    
Well, a changelog is a development term, and is used specifically for a change in the codebase. A runbook is server specific. In this case, his changelog is in partly in SVN. I wanted to be very specific about system documentation as opposed to "just keep good a changelog", which might not mean to him what it means to me. –  Joseph Kern Mar 30 '11 at 19:17
    
@bittrance, hmmm ... I concur, scripts are better than notes. –  Joseph Kern Mar 30 '11 at 19:18
    
I'm not that advanced yet for scripts so I'll stick with notes for now –  crashintoty Mar 30 '11 at 19:28

Under the assumption that we are talking whole machines (whether physical or virtual) for the production environment (rather than, say Google App Engine), my best practice is thus:

Take the extra effort to create proper packaging for your software. For example with Java development targeted for RedHat, use/write Maven tools that create RPMs. For your database, for all changes and include them in the RPM and either have them apply automatically or create a small tool that applies them for you. Essentially, you are done when a server can be installed/upgraded with a single command. Yes, this means including config files tailored to that system.

This is a bit of extra work, but it is so worth it in the long run.

Once you have properly packaged your software, the question of dev/staging/demo servers isn't so important anymore, because you can always build another. And with proper installers, you know exactly what changes you need to do to a machine to get it working properly.

Don't bother with migrating virtual machines around: you would need to change lots of configuration parameters on migration anyway.

If you tell us a bit more of OS/dev tools concerned, I may have some more specific tips for you.

EDIT: PHP/MySQL. Assuming this is LAMP.

Create a deb/rpm with your PHP code that depends on the various Apache, mysql, mod_ssl, php packages that you need. Look at other PHP applications packages on that platform for inspiration. Typically, you don't want to have a package dependency on a database, because the db may be on a different box.

Your package should prolly contain two SQL scripts (that comes from your source repo): one that initializes a new database, including grant statements, loading stored procedures, creating indices, etc and one that loads some demo data that can be used on test systems to get up and running quickly. Also, successive versions of your packages may contain patch scripts that upgrade the database.

Most modern Linux distributions frown on overwriting files from other packages, so try to avoid it if possible.

EDIT2: How to build a Debian package from just a directory tree.

You need a directory tree that looks like the installation you want to make (lets say in a dir called build), and you need some control files:

controlfiles/control:

Package: coolapp
Version: 1.0.0-2
Architecture: i386
Maintainer: Skunkworks Dept <bittrance@example.com>
Depends: apache2 (>= 2.2.16)
Section: contrib/libs
Priority: optional
Description: My Cool App
 It's going to revolutionize, yo!

controlfiles/conffiles:

etc/coolapp/settings.conf

Given these two, use this script called mkdeb:

#!/bin/bash

CONTROLDIR=$1
shift
SOURCEDIR=$1
shift
INCLUDES=$*
BUILD=/var/tmp/mkdeb

CONTROLFILE=$CONTROLDIR/control
PKGNAME=$(grep -E '^Package:' $CONTROLFILE | awk '{ print $2 }')
PKGVERSION=$(grep -E '^Version:' $CONTROLFILE | awk '{ print $2 }')
PKGARCH=$(grep -E '^Architecture:' $CONTROLFILE | awk '{ print $2 }')

FILENAME=$PWD/${PKGNAME}_${PKGVERSION}_${PKGARCH}.deb

rm -rf $BUILD && install -d $BUILD
tar -C $SOURCEDIR -zcf $BUILD/data.tar.gz $INCLUDES
# Proper debian packages have md5 sums for all their files
(cd $SOURCEDIR && find $INCLUDES -type f | xargs md5sum > $BUILD/md5sums)
metadata=
for f in control prerm postrm preinst postinst templates conffiles ; do
    if [ -f $CONTROLDIR/$f ] ; then
        cp $CONTROLDIR/$f $BUILD
        chmod a+x $BUILD/$f
        metadata="$metadata $f"
    fi
done

# Metadata and stuff
tar -C $BUILD -zcf $BUILD/control.tar.gz md5sums $metadata
echo 2.0 > $BUILD/debian-binary
(cd $BUILD && ar rc $FILENAME debian-binary control.tar.gz data.tar.gz)

Like so:

mkdeb ./controlfiles ./build .
share|improve this answer
    
I imagine I'd use Ubuntu 10.10 64-bit server, apache, PHP/MySQL... that's all I can think of for now (I'm new at this) –  crashintoty Mar 30 '11 at 19:25
    
@crashintoty: Ubuntu is good. Debian is generally speaking more accomodating in this sense than RedHat. For example, with Apache, you can stick stuff in /etc/apache2/sites-available and do a2ensite to get it active. –  Bittrance Mar 30 '11 at 19:37
    
Looking at your original response; is this method better than simply backing up and restoring /etc/apache2, /var/www and mysql databases? I'd imagine that's all I need since those are the only things I've touched so far (I just set up a virtual dev server). –  crashintoty Mar 31 '11 at 22:21
  1. Keep the underlying distro of dev and live the same. For instance, if live is going to be ubuntu 10.10 ec2 instances, than dev should be an ubuntu 10.10 environment, even if that means running a virtual machine on your (mac) laptop. Even better is to have it be the same as live, such that even network connectivity issues are equalized.

  2. Configure those two servers/environments using a config management (puppet/chef) tool so that its self documenting and reproducible. These work just as well, if anything easier, on a single server. You could start out bone dumb simple with one giant recipe that just has some if-dev/else-live hacks. Keep these recipes in your version control system along with your code.

  3. keep your code somewhere separate in version control, could be local like a trac/svnserve setup or could be hosted like github. Just separate, somewhere that is not at all involved in end-user service. Write a very simple script to deploy your application from that version control system. It can be just a shell script with 3 or 4 lines of bash mashed together, start simple, keep it in version control itself, and iterate as needed.

If you do all that you'll have a dev environment that matches the live environment in a way that you can prove it with a diff. You also get an simple deployment workflow that makes it easy to iterate super fast, and you'll have all the groundwork in place to build matching servers and scale out when needed.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.