2

Problem:

Deploying, and managing this site has become a nightmare. Deployment, going from development to QA, and pre production, takes a couple of days. We have lots of errors, and it involves a lot of manual labor. We often have config files that have been updated or changed incorrectly, and then the entire site on that web server fails.

The environment:

  • Plenty of web front end servers (windows 2003, IIS6)
  • Plenty of MemCached Servers (Linux)
  • Multiple database servers (SQL 2005)
  • Approx 5 million unique per month

More Detail:

I am one of the developers that are responsible for developing the system. Currently there is zero automation, zero automated testing, and the deploy process involves logging into the servers, with terminal server, and manually changing IIS settings, updating config files, and similar horror's. Disaster recovery documentation leaves a lot to be desired.

I would like to enable the team that handles the operations of the servers, to successfully deploy and maintain the application that I am working on. So for partly very selfish reasons, I want them to succeed.

I would like some direction; on what are specific questions I can ask them, or suggestions that I can make, on how as developers we can create things to help them be more successful.

Currently there is a bit of mudslinging going on, and I figure the only way to really fix that is to ignore the mudslinging, and focus on the problem. And the problem currently is that it is VERY labor intensive to maintain, troubleshoot and deploy the site.

Thanks Rihan

Improve this question

7 Answers 7

Reset to default
5

  • Write code that checks the sanity of any configuration it gets given and reports problems both directly to the person doing the configuration, and in the logs.

  • Make a clear separation between configuration values that are environment-specific and those which should be carried unchanged up your Dev, Test, Acceptance, Prod (DTAP) street.

  • Use version management software (e.g. subversion) to track configuration changes in all your environments.

  • Manage the configuration of your DTAP environments like a megalomaniac.

  • Just after you deliver a release, and before you start coding again, arrange to have the Production environment copied over all the other environments including development. When you communicate this to your fellow devs, see which of them turn white, and ask them which assets they couldn't replace at will from source control.

  • Most importantly!!! You're probably reading this thinking, who is this idiot? That's impossible! We couldn't do all that. Dead right - of course you can't - not now. (If you weren't already in trouble, you wouldn't be asking the question). So separate vision from implementation. Share that vision with the other stakeholders. Decide together which capabilities you should have, and make a plan that starts getting you there. Each cycle or release or whatever, make sure you move another step closer to the vision. Set realistic targets, and meet them. (Of course, you're allowed to change your vision based on experience as you go along.)

Improve this answer
5
  • With version managment software, we are using SVN, do you suggest updating SVN, and doing the deploy via svn? Currently our msbuild script does create all of the required enviroment settings, builds the dll, and package it all up neatly- Do you suggest to move that process to the server level? Thanks for your time in giving a very structured well thought out answer)
    – Rihan Meij
    Jun 2, 2009 at 14:14
  • Actually - my answer was aimed at the generic case, rather than specifically your technology. Obviously, creating a .msi is also quite an attractive approach. The most important thing is being very clear which parts of the configuration are part of your delivery (and therefore must go through test), and which belong to the server admins. You can use SVN as a means of moving things up the environments (I have done this.) but also as a means of managing manual changes in a given environment. Perhaps you could create a branch for local changes in a given environment.
    – Dominic Cronin
    Jun 2, 2009 at 15:39
  • How strict should one be on the way that servers are being built? The only way that I know to ensure constancy, across different machines, is using an automated build process / image as also pointed out by Alex. How high do you rate this as getting to a stable site? How would you go about, if the system administrators did not feel the need to automate this process, and feel that installing the OS by hand is sufficient?
    – Rihan Meij
    Jun 5, 2009 at 10:18
  • Dominic, what you said in your last point, made me realize you dont walk with your head in the clouds. I have worked before with a sys admin that was a security freak, and the funny thing is, we never had downtime. So I tend to feel that knowing what happens on your system and controling it like a megalomaniac is how you get a stable enviroment
    – Rihan Meij
    Jun 5, 2009 at 10:24
  • Just re-reading this a few years on..... these days, I'd use git instead of svn.
    – Dominic Cronin
    Mar 12, 2015 at 22:15
2

I would say that apart from keeping the server settings and config files the same, you should have test environments to deploy to in your office. If your developers test that the code they developed works on an environment thats very much like a live environment, then they would feel more confident. More confidence means they necessarily develop faster (as this is something we have learnt from our SCRUM teams) and far less mudslinging and feeling of not being in control.

So basically what I am saying is, just give the developers lots of machines to play and test with and you will see positive results.

Improve this answer
2
  • Hear hear. If the only way to test is to deploy to the live environment and see what happens, there are serious risks.
    – Alex J
    Jun 1, 2009 at 11:34
  • If you were in my shoes, and given the scenario, would you take this as your first priority. We do the deployment in our SCRUM setup. We do have a QA enviroment that is virtual, would you then just make a copy of the virtual enviroment, do the deploy, if it works great, if it does not, delete the state, and back to the drawing board?
    – Rihan Meij
    Jun 5, 2009 at 10:21
1

The most important step is a simple way to make the configuration of all servers across your infrastructure as consistent as possible. A Standard Operating Environment (SOE), if you will:

  • All IIS instances have identical configuration.
  • All web servers have the same copy of the site on them.

Once you have this, it should become significantly more possible to implement automated rollouts, testing, and so on.

If the IIS configuration is identical across all servers, you can update the config file IIS uses on one master server and run a script to push it out to all webservers, test it, and reload IIS. You can do the same for your actual website files.

Improve this answer
1

My guess is that if you had the option to standardize the servers you are going to deploy to, you would have done that by now, so I am going to skip that suggestion. What you are going to need to do is build scripts/MSIs etc. to automate the deployment process for the operations team. It will take some time to setup, but removing the Ops team's manual intervention from the process is the only way to make it run smoothly. Now I am not insulting their intelligence, far from it. The problem is that they don't know how the code works. They don't know what a config file with a bad value will look like. They depend on you knowing that to help them and the reason why automation is key to solving your problem.

My suggestion would be to setup virtual machines locally which mimic the various servers, so you can test over and over again how the install is going to work. If your IT department is anything like mine, getting new servers will be difficult, as it involves budgeting and resource allocation etc. Virtual machines remove a large part of the headache as you don't need new hardware, and input from anyone will be minimal as you'll only need to know how the servers are configured and then you can probably do the rest yourself. VMs also allow you to quickly reset the machine to it's original state and try the install process again.

Improve this answer
2
  • Kevin how far would you go with this "virtual" enviroment setup. If we have load balancers setup, would you included that? How about multiple SQL servers? What do you guys measure, in terms of a good deployment. Opening up the site in a browser, to say that the site code release has been sucessfully dont feel right to me.
    – Rihan Meij
    Jun 5, 2009 at 10:28
  • @Rihan Mei I would just have virtual servers which mimic the ones which host the code. We have it set up so that all the code is deployed through MSIs, and if there are any problems, the MSI roles back the installation. Since we use it to deploy to everything (Test servers etc.) we know the code and process we deploy has been checked for errors, so all we have to worry about when we deploy to production is if it actually installs.
    – kemiller2002
    Jun 9, 2009 at 11:04
1

In my infrastructure, we've got a staging environment called preview where the data goes before it hits production. The application code in the staging level is exactly the same as the production code, so any data problems will show themselves there.

We've also got a test environment for new code, which operates on old, known-good data (which we can mangle when needed to test how the code fails).

Generally speaking, this segregation allows us to have a pretty solid production environment, and we don't see too many unexpected things crop up.

I should note that I don't deal too much with that, since I'm not a programmer. I just sync the databases and refresh the files when they ask me to ;-)

Improve this answer
0

First and foremost CONSISTENCY. Make sure that all of your servers are exactly the same. All W2003 web servers should look like the rest. All DB servers, All Linux server. I mean same. Same OS version, same patches, same directory structure... Everything!

Second comes staging. You should have a separate server of each type, that can be used to test your deployment. This is where you make all your configuration mistakes. You would then copy your system from these deployment servers to production.

Third the "Separation of Concern" concept should be applied to your config files. If different modules have different configuration items, put them in their own config file. If there is machine specific configuration, move IT to its own config file. This way you can deploy the affected config files from the deployment server along with the components themselves.

Improve this answer
0

One word: Backups

Make sure they have a good, centralized, redundant, and consistent backups in place.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .