Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I need some ideas on how our work flow compares to some others. We have a single developer that works on our marketing website and she knows how to code and that's about it (no source control experience, no command line.... nothing). In the past the developer has always edited everything directly on the server using Eclipse. Meaning, if she fat-fingered something and saved it, it was viewable to the world.

So after dozens of "mishaps", I've been tasked with creating a new workflow that's more appropriate than just editing the files on the live server. Here's what I've got so far..

  1. Subversion server with a marketing repository

  2. A "Production" VM with the main marketing site that is up on the webs as we speak.

  3. A "Staging" VM that allows her to see changes in an identical environment before we upload any changes to "Production"

  4. A script that allows her to sync the latest SVN tag to either of those servers.

To keep the integrity of those two servers, I'd like to make a third "Sandbox" server. It'd be another VM, copied from the "Staging" server. With this server she can experiment with code, try new things and not worry about making the Staging or the Production servers FUBAR.

When I proposed this plan, I got a lot of grumbling and moaning from multiple parties about how much work this creates for her. The flow would be:

Edit Code in new branch > sync to SANDBOX. Repeat until satisfied. Merge changes to Trunk, create a new a tag from trunk and sync to STAGING. Do a QA check. If everything looks ok, sync to PRODUCTION. (I'm very open to new ideas for this process)

As could be expect from someone who's never used source control in their life and is used to editing the file, pushing ctrl+s to save, and then refreshing the browser, this is a nightmare. I'd love to find some middle ground, but having a hard time doing so if we're to keep the Sandbox/Staging/Production set up.

What has worked for you guys?

share|improve this question
    
Eek. Having worked with developers that didn't see the benefit of version control in the past, I can sympathize. I'd say simplify the process to the bare minimum (maybe just leave out the "tag to staging" step, and have the staging server just live off of trunk? wait to tag until the production push, as part of the push process?), and automate as much as possible (have the staging server automatically keep itself up to date with the latest tag or the trunk), to cut down on the "extra work". Then.. train. An open-minded developer will quickly see how much work it saves them in the long run. –  Shane Madden Jan 16 '12 at 23:14
    
That sounds fairly reasonable. Maybe one of the thing I'll need to give up in this compromise is an unadulterated staging server. I really didn't want to have her 'experimenting' on the staging server, but only having the two servers would definitely be more appealing to them. –  Safado Jan 16 '12 at 23:29
    
For the Close Vote, I think that this question has real merit. This is a situation that many companies (especially smaller ones) have to deal with. –  gWaldo Jan 17 '12 at 2:49
add comment

3 Answers

up vote 2 down vote accepted

Your plan is largely sound.

Source Control:

Make it mandatory for the developer's job. By all means, send that developer to training or a local dev group for a beatdown. Convince your mutual boss of the need. (If you have to, use the 'turned malicious' or 'left something out' scenarios, or the plain-simple 'webserver got hacked and defaced' scenario. Although I would recommend git for any new projects, hosted on either a private server or a paid-for github account. ThinkLikeAGit, Roger Dudler's "Git - The Simple Guide", and "Git for Ages 4 and Up (direct to video) are all awesome.

I prefer that Master be a main development branch, a Staging Branch to be deployed for testing, and a Production branch (or Tags on the Staging branch with version numbers on the deployed commits). I also like the idea of branches for features. Ask 100 developers how they like their source-control, and you'll get about 65 answers...

I strongly dislike and advise against Master being the "Production" branch. One newbie mistake (which EVERYONE makes) is committing to the wrong branch. Don't let Production be the default branch.

Development Server:

Yes, a place for the dev to go crazy. Do whatever you want; this is your home, sandbox, whatever. Development should happen at the local machine, but this is an environment that will look like production, but the flexibility to go crazy. VM with snapshots taken at the base-state (and whenever Prod gets updated) are recommended.

Staging Server:

Once the dev is happy with her code, it gets deployed to the Staging Server. This is a server that exactly matches production with the exception of the URL. (Her code should be using relative paths, and not the site's actual name.)

Here's the kicker: Once she's satisfied that the site is ready for Production, she stops. She doesn't deploy to Prod (yet.) A Hither-to-unknown QA process now happens. Somebody with an eye for mistakes (and who has at least an overview understanding of what this new code should be doing) reviews the site in Staging. (You, the creator, are blind to many of your own mistakes, making you unqualified to validate your own work.) They test the site, ensuring that the problems that are being fixed, and that nothing new is broken.

I cannot stress this enough: The Developer does not do QA! She should ensure that her site doesn't have any obvious brokenness before passing it to QA, but she does not approve it for Production. (First, she has demonstrated a lack of understanding of best-practices, and mistakes have shown that best-practices exist for valid reasons.)

The person who is doing QA has their professional reputation at stake; If a mistake is pushed to Prod, it's their approval that made it happen.

If QA finds mistakes, they are identified to the developer to fix. The process beings anew, complete with QA review and sign-off.

Personally I'm a fan of Devs doing their own deploys. They get the instant feedback of knowing that the task is completed, and by not making someone else do the deploy, they feel a greater sense of ownership if something goes wrong. (Saying "it works on my machine" should be a capital offense.)

The most important part is that what is in the repo gets deployed. There are no 'one-off' cowboy-coding changes.

Production:

Once the QA sign-off has happened, the code can finally be pushed to Prod. It's up to you and your org who does the deploy; it can be the coder or it can be the QA person. If the deploy doesn't execute completely, it goes back to the developer.

In Conclusion:

The way things are going now obviously doesn't work. Push to make something like this mandatory company policy (if you agree with it, of course), and make employment conditional on it's adherence. It may make more steps for your poor developer, but for the company it's less work, especially in having mistakes go live. Not to mention the safety net of having known-good states of your code that live outside of the Production webserver.

Optional Bonus Points

I don't know what you're doing for deploys, but they should be one-step or no-steps.

An example of a One-Step Deploy is a script that you run on or against the Prod server (and should be the Staging servers as well, and Devl, too) that pull down the appropriate Branch from your repo and restart the needed web services.

An example of a No-Step Deploy is a cron that watches the appropriate Branch and automatically updates the code when the Branch is updated.

share|improve this answer
    
I'm still working on the development part of this plan, but everything else is in place and she's adjusting rather well to the new work flow. We've had a few "told you so" instances over the last week where I've been able to say "NOW do you see the value that this provides?" –  Safado Jan 20 '12 at 21:09
    
That makes me smile. I'm glad that it's working out for you. Version Control should be the first part of every programming book, IMHO. –  gWaldo Jan 21 '12 at 0:21
add comment

If she were to work directly on the SANDBOX server, but make the document root itself a working copy of the 'marketing' repository, then the only extra thing she has to worry about during her edit-refresh cycle is an occasional checkin. She'll soon be wondering how she did without VC. The value of tags and branches might be harder to get across. Make the deployment to STAGING and PRODUCTION as easy as possible by scripting them.

share|improve this answer
    
One caveat is that we're using Expression Engine to run the website (kind of like Wordpress for those that aren't familiar) that has a couple of configuration files spread throughout it's file structure. In those config files you have to have the Domain specified. So I would have to set it to something like "sandbox.mydomain.com" and just accept that the checked in copy is always "dirty" and exclude all config files form syncing to the staging and production servers. –  Safado Jan 16 '12 at 23:05
    
I actually found a blog that walks through making EE more multi-environment friendly. So I think it'll work out rather well to have the sandbox on her own machine with the doc root being what she has checked out. –  Safado Jan 17 '12 at 19:55
add comment

Part of what is missing here is fundamentally how to manage and execute a release, and/or a formal change control mechanism. I'm not suggesting that there be an actual application, but there should be a process that includes:

  • Communicating the change (date/time of change and what is changing)
  • Risk or potential for impact (low/medium/high)
  • The time window of the change
  • The time window that the system would be unavailable
  • Instructions that may be used by an independent person to execute the change, if necessary
  • Instructions for backing out the change, if necessary
  • Who is executing the change and their contact information
  • Who is validating the change and their contact information
  • The deadline for validation, after which the change should be backed out if validation is not complete

Ideally someone besides the developer or person executing the change would perform the validation, and a document or checklist describing the validation.

The time window would be agreed-upon in advance, that typically precludes doing something at an obviously bad time. (Who pushes code on Fridays?)

I would be careful to not impede the process. If they want to push code every day, the system should not be a barrier to getting things done.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.