86
votes

Through reasons that don't warrant exhaustive discussion, I find myself in charge of 10 servers:

  • A domain controller-~500 hosts/~350 users
  • IIS web server-This is where we make our money
  • SQL server-The crown jewels
  • Exchange server
  • Linux box for data entry
  • AV server
  • Backup server
  • A few others tossed around

The company where I work believes everybody is replaceable and therefore believes they can pay a minimum wage for any position. The IT manager and Sysadmin quit recently and I think I was the only person who did not take a big step backward when the call went out for volunteers. This also explains why someone with my background is in this position. That is the reality, as much as I wish it otherwise.

What are the things I should be doing to keep those systems running? There is no written procedure left behind and I crammed the A+ and Network+ certs in the last two months but that leaves me with some theory and no practical experience.

I am in the process of teaching myself powershell but from here to there is a long way. I have no scripting or programming experience.

What tasks should I be performing? What practices should I implement?

I understand I am probably hosed but a lifeline to get me through would be helpful.

26
  • 68
    After reading that, I think your first step is to find a nice cushion to put in the corner. You're going to have many hours rocking on it. With a box of kleenex to wipe the tears. You might also want a pad of paper to start updating your resume while you're at it. My condolences on the spot you're in... Dec 9, 2011 at 15:53
  • 86
    Honestly, this is an entirely serious suggestion. Find another job asap. I've been in IT for 7 years, 4 of them in Systems Administration and I'd find managing all of that by myself to be a handful and I know these applications inside and out. For someone with no background, it's suicide. If you're to be held accountable for their well-being, you'll probably end up losing your job over it. Get your resume out ASAP, or ask to be relieved of these duties.
    – MDMarra
    Dec 9, 2011 at 15:57
  • 13
    serverfault.com/questions/9766/…
    – Zoredache
    Dec 9, 2011 at 16:03
  • 10
    I am sure this will read as self-serving since I work at a managed service provider, but you may want to see if you can acquire an emergency budget/funds for outsourced help. Find a vendor in advance that can handle your needs. I would have a short list of paid help should it be needed. Given the climate, I doubt this will be possible but certain something I would try to do in your spot. Good luck. Dec 9, 2011 at 16:31
  • 15
    @Bart Silverstrim- I am on my third pallet of kleenex Dec 9, 2011 at 16:53

13 Answers 13

56
votes

Honestly, I would find another job, unless your current task is just to keep everything running until they hire a new SysAdmin. You are being setup for failure. You are doing the job of at least two people if this is all hosted locally and nothing is documented.

Don't worry about the scripting or programming anything just yet. Get a handle on keeping everything running.

Are you in charge of the corporate firewall too?

The quick and dirty daily tasks I see you needing to do are (in no particular order):

  • check the nightly backups
  • check the exchange queues to make sure they are processing
  • check the SQL backups
  • check the AV server for alerts if anything failed to scan or update
2
  • I'll also add to learn the simple IISRESET command to potentially fix problems on your IIS server.
    – NinjaBomb
    Dec 9, 2011 at 20:08
  • 5
    The line "IIS web server-This is where we make our money" says it all. A lot will depend on the programmers you are surrounded with. But given "everybody is replaceable and therefore believes they can pay a minimum wage for any position", I think jumping ship is the sensible thing to do.
    – DutchUncle
    Dec 9, 2011 at 22:37
51
votes

First off, look at Zoredache's link to the Beginning Sysadmin question. It's a lot of info but will get you where you need to go. That being said, it's a lot of data dumping.

Second, seriously, look for a new job. If your employer regards everyone as interchangeable widgets you're better off not being there anyway. But it's a bad economy, and that might not be an option.

In your specific case, I would:

  1. Find a local IT firm you can escalate labor to if something breaks and you're over your head.
  2. Make sure your maintenance contracts with your software vendors are up to date. Chances are most of your software packages (including whatever's on the IIS server that's making you money) were purchased rather than written internally. Make sure you have support with them - if anything breaks, you'll be calling them first.
  3. Make sure you have good backups - both data and system state. If one of your servers toasts, you'll need to be able to restore it quickly. This may actually be the most important priority.
  4. Do a hardware audit and make sure your servers are under warranty. If they're out, point out to management that there will be significant downtime if they die, and you only make money when they're up.
  5. Pray
  6. Perform a software audit, hope you find a lot of out of compliance software, report it to the Business Software Alliance (if you're in the US), and use the money you make from the finders fee to tide you over until you find a new job.

...6 may be optional.

4
  • 27
    Whereas 5 is a must Dec 9, 2011 at 16:24
  • 3
    #3 and #6 are musts too
    – Bigbio2002
    Dec 9, 2011 at 18:10
  • 5
    Nope, in his case, just 5 is a must.
    – surfasb
    Dec 9, 2011 at 20:05
  • Given the suggestion for 6, 3 seems optional too. Actually, it sounds like 5 is probably just as effective as 3 at the OP's place of work.
    – fluffy
    Dec 10, 2011 at 2:59
21
votes

Find a new job. Let that company fail. They clearly need a lesson.

Your stepping up to help will quickly become unappreciated and chances are, won't lead to anything else but more stress and abuse.

1
  • My company took similar lesson while they wanted to hire cheap Solaris Sysadmin by hiring guy with some Linux administration experience and... I found that I've got better knowledge of system administration that the guy (I am a programmer and he was sacked three after three months...). Dec 10, 2011 at 10:55
12
votes

Start drinking? :-)

Besides that, look at getting a spare box or two that can be used to set up non-production servers. That will give you the opportunity to learn how to deal with some of this software in a slightly less dangerous environment than "if I break it we all stop working and making money".

Even if you run them inside virtualization software like VirtualBox, it's something that gives you hands on with the software without jeopardizing the production work.

Finally, if you do have to touch the production servers before you're more comfortable with them, document every step you take. Make sure you always know how you got to the screen you're on and how to get back.

1
  • 9
    +1 for "Document every step you take." I would also add notes documenting the "why" as well as the "how." This will be helpful when backtracking (to say nothing of its importance in an unhealthy organization).
    – Roy Tinker
    Dec 9, 2011 at 17:46
12
votes

Honestly, find a mentor!

Not knowing where you are, it's tough to recommend a forum, but meetups are pretty common with IT folk so google your local area for a Sysadmin (or specific techs) meetup. You'll get a whole bunch of new contacts out of this too; which might help in unexpected ways.

Update:

I'd also highly recommend you start writing a blog about this (pseudonym required IMO)!

1
  • +1 for find a mentor. Two mentors, actually: a career mentor, yes, but also a paid outside contractor to consult when you have questions or problems. If your boss won't approve paying an hourly rate in the neighborhood of US$100-200/hour to bring someone in when you absolutely need help, run like hell.
    – Skyhawk
    Dec 9, 2011 at 22:04
11
votes

So, you will be learning on production servers and this is not your choice. In this case, you should not be blamed on any service interruption. It is the company responsibility to find a replacement for the sysadmin.

You can learn many things, but it will take time and it may hurt the running services. You have to be extremely careful before doing any change and you should do it on a development/testing server first.

There are many related questions on this site such as: SysAdmin & Developer: Responsibilities, https://serverfault.com/questions/51006/how-did-you-become-a-sysadmin, and What's the difference between sysadmin and an IT Consultant?. I hope these will give you an idea what should be done as a sysadmin if you could not get out of this ASAP.

4
  • 21
    "Should not" and "will not" be blamed are two entirely different things. Dec 9, 2011 at 16:21
  • 3
    What @Driftpeasant said -- Invariably when the fecal material hits the rotary air displacement mechanism whoever is in the SA chair gets blamed, even if you made them sign papers holding you harmless.
    – voretaq7
    Dec 9, 2011 at 21:47
  • 1
    I'd pay extra for a fan named that ...
    – thinice
    Dec 9, 2011 at 22:16
  • Based on what the OP has already told us about his company, I'm fairly certain that anything that goes wrong with the server will result in the blame being placed on him, regardless of whether or not it should be.
    – Wipqozn
    Dec 9, 2011 at 23:03
11
votes

Protect your jewels!

Brent Ozar has an excellent "Oh crap, I just inherited a database server" script and even a training video now. He calls it the "60 minute database blitz" but I think my title is implied.

Remember this

Your backups are important but nowhere near as important as your restores.

What's that mean? Yes you need to take backups to have a chance of unbuggering a situation but if you don't test those backups, all you're doing is burning disk space.

7
votes

If for some reason you decide you actually WANT to do sysadmin work, I heartily recommend Tom Limoncelli's Practice of System and Network Admistration. He provides excellent examples of how a well-operated organization might be run and if you get an opportunity to suggest changes, it's tough to go wrong with his recommendations. Book link to Amazon: http://amzn.to/sNUmsr

7
votes

You never want to be in a position where you are forced to learn on production systems. Learning involves making mistakes, and mistakes on production systems cost your company money. They may not care about anything else (which is clearly indicated by the very fact you are in this position to begin with) but they will care about this.

  1. If it is at all an option find another job. This will not end well. And honestly do you really want to work at a company that treats everyone as replaceable?
  2. Request an emergency budget so you can find a consulting company to keep things running and act as a resource while you get up to speed. If this is denied, make sure you have it in writing somewhere for CYA purposes.
  3. Read any documentation left behind by the previous IT staff.
  4. Perform an inventory of both physical hardware and devices. Figure out what is in warranty and what isn't.

Your goal should be to find another job and leave without anything breaking. If something does break (and it probably will) be prepared to walk out. This means you should be extremely careful regarding CYA procedures. You need to assume that no on at the company has your back - make sure you're not in a position where you need to rely on someone's support.

While you're looking for a new job setup a environment to practice and learn with and do your best to familiarize yourself with the current infrastructure.

And for what it's worth... my degree is in Philosophy. You can succeed in this field if you want to, but not at this company. Good luck.

0
6
votes

Reading everybody else's comment I would say:

  1. Take this opportunity to learn
  2. Tell your management without sugar coating that they are in a dire situation
  3. Take this opportunity to learn more. After all sysadmin / DBA related jobs are well paid.
  4. If you do get fired, collect some money from them and find a new job with your newly acquired skills.

I love the write a blog comment, you have pretty good writing skills. Make sure to put a link where we can find it.

1
vote

There are quite a bit of scattered answers here. Here are some of my thoughts.

Some people say you should do it and others say to completely avoid it. My question to you is are you ready to devote hours on and working on these servers? Not only that, but are you also willing to learn things completely new?

Server administration isn't tough once you grasp it, but I've ruined servers by running the wrong commands, not following the "standards" (like taking daily backups), and et cetera. Every system admin will fail sometime, hey, it's how we learn, but you have to be prepared on how to handle it. It's a good amount of work and the biggest question here is if you're willing to take on the job.

Sure you can learn something new to put on your resumé, but if you are not willing to learn or devote yourself to you job, you're just going to be another person who "hates their job".

1
vote

I will be frank, this is a handfull, even for more experienced guys, also learning most of the skills in server administration is learning from mistakes. I doubt you want to make mistakes on live-systems.

The danger of you not having any or few experience in server administration also makes it dangerous for your company. You are the victim of ignorant managers it seems. The only way to wake them up would probably be a sudden disaster, where disaster recovery is nescesary and nobody will be around who has the knowledge to make it run again and their crucial business processes stop working.

I'm not implying you should create a disaster, but just go and find another job, because you don't want to be held accountable when things go bad.

0
votes

Seize the day! This is a great opportunity. Get a subscription to the Safari Tech Books database and read up on all of the how to books and certification books for the systems you administer. Start by focusing on backups and restore procedures. Get yourself test servers to play around with before laying hands on the production models. Test servers can be cheap desktop units, they don't have to be expensive or durable like production ones. Make yourself a goal to be successful in this position for two years and then go get a new job with a good company that is willing to pay you for your great knowledge and experience. Also, earn certifications for all the technology you are responsible for. You are poised to have a lot of fun and earn a lot of money (way more than a degree in history would get you).

1
  • 3
    One important question for our history major to ask: "Is this fun for me?" If not, that might impact the feasibility of an enthusiastic approach...
    – Skyhawk
    Dec 10, 2011 at 1:41

Not the answer you're looking for? Browse other questions tagged .