Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Problem: If I spin up a server1 today, all installed packages will be based on current upstream versions. If I spin up server2 tomorrow, again, all installed packages will be based on upstream versions, except this time, within the 24 hour difference between server1 and server2, upstream versions on some packages may have changed. So we end up with server1 and server2 out of sync with respect to installed package versions.

My goal is to keep all servers across the infrastructure at the same "snapshot" version, but still support a regularly scheduled update process.

Example. server1 is snapped today. As part of the snap process I run the following commands:

yum update --downloadonly --downloaddir=/path/to/my/repo/base/v1

yum update --enablerepo=mybaserepo

What this will do is pull down all the rpms based on current upstream versions (snapshot). This allows me to host the rpms from "mybaserepo", so when I update other servers, they can get the rpms directly from my repo, without having to worry about upstream rpms being no longer available.

I was thinking of implementing a monthly upgrade cycle, so v1 was snapshotted on 5.1.2013 and it's now 6.1.13. What I was thinking as a process cycle was this:

  • Update all other servers to snapshot version 1
  • Update snapshot server to version 2 using yum commands given above

And this cycle repeats monthly. Im my mind this strategy solves two problems:

  1. Constantly updating servers to ensure upstream patches are brought in on a consistent basis
  2. Not pushing out new package versions without first testing (for 30 days) on the snapshot server

Granted, there is no absolute single answer, as there are many ways to skin a cat. What I am looking for is a commonly implemented, industry accepted practice which directly answer the problem of updating servers, while still ensuring the same versions across all servers within an enterprise environment.

share|improve this question

2 Answers 2

  1. Make your own repo (rsync an official centos mirror)
  2. Update the repo only when you want to pull in new updates for all your servers
  3. Point your servers only at your local repo.

You can use yum --downloadonly, but you still need to create a repo. The work is already done for you if you simply mirror an existing public repo and only update it when you want to.

share|improve this answer
    
Ya this only answers part of the question. I'll post more tomorrow about a strategy I am working on to solve the rest. –  Mike Purcell Jul 24 '13 at 5:05

yoonix's answer is good way to invest in being proactive about updates, which we all know is what we should be doing. But it's also good to keep a handle on how things actually are. :) One thing I might recommend is a cronjob that would do something like this code sketch (untested):

#!/bin/bash

email="monitoring@example.com"
sshuser="monitoringuser"
servers=$(cat my-servers.txt)
when=$(date '+%Y%m%d-%H%M%S')

for server in servers; do 
    ssh $sshuser@$server 'rpm -qa |sort' > server-$server-rpmqa-$when.txt
done

diffreport=diff-rpmqa-$when.txt

diff -u server-*-rpmqa-$when.txt > $diffreport

mail -a $diffreport -s "Package diff report for $when" $email

You can see how it would also be useful to run 'rpm -Va' periodically (maybe once a week?) and send that around, to check for trojans, configuration drift, etc.

Of course, these kinds of scripts will produce "reports" that are only really readable by highly technical people, so you could consider using a tool instead. As you're probably expecting by now, my company, Metafor Software, makes just such a tool! :)

share|improve this answer
    
Thanks for the response, I will def look into this reporting capability. –  Mike Purcell Jul 24 '13 at 19:58

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.