What is the one thing you wish you knew when you first started being a system administrator?

Question

For a new system administrator it is important to learn from those that have experience.

What is the thing you know now you wish you knew when you first started?

Alternately, what is the piece of advice you would give to a new system administrator?

Answer 1

1. SLOW DOWN... we're in a hurry.

2. If you do your job right NOBODY will notice a thing.

3. It's never going to be 9-5 or a 40 hour work week.

4. You can never have too many backups. Test them!

5. That "impossible" scenario you didn't bother planning for WILL HAPPEN.

6. Hackers (actually script kiddies) hack things because they can, not because they targeted YOU specifically. It's not personal, you were just there. So don't ask "Why me?"

7. Document everything! Even if it's just for your own sanity. A private Wiki goes a looooong way. If you can't bother to do that then at least keep a "never ending" text document nicely formatted on your computer... then back that up too! Just because you know something "inside and out" today doesn't mean you'll remember what the hell you did 6 months from now.

8. If something goes wrong in the evening and you think you might be in for a long night... WRITE YOUR PLAN DOWN. You'd be surprised how "mush brained" you're going to be at 3 a.m. and suddenly you're going to say, "Now, did I actually do X or was I just thinking I needed to do X next? Oh crap!" (This WILL happen to you, especially if your recovery process takes a few hours.)

9. The weakest aspect of any computer/network is almost always going to be the HUMAN ELEMENT. It doesn't matter how secure you make the computer/program/network. Some moron is always going to try and use "bob" and "bob" as his username and password then write it on a post-it note and stick it to his monitor... which happens to be facing an outside window... which just happens to be outside of a bus stop. (I wish I was making this up) ;-)

10. Relax! I can almost GUARANTEE you that no matter what bad thing(s) happen to you someone, somewhere is having a WORSE day than you. Be happy you're not THAT guy and stay GROUNDED. If you can stay CALM when everyone around you is freaking out... you'll find that after a short time THEY calm down too. Don't participate in mass hysteria. ;-)

BONUS TIP: Wives/Girlfriends are NOT stupid. I once pulled up a mail server log with "tail -f maillog.log" for my significant other to watch a dictionary attack on a mail server. I explained to her that this level of attack is "normal" and is almost constant. I then explained that when my phone goes off at 3 a.m. it's because we're facing something about 10x's BIGGER. You'd be surprised how sympathetic/understanding they can be when they can actually SEE the crap we have to deal with daily. SHARE THE EXPERIENCE!

Answer 2

Exercise.

I'm not fat, but wow am I out of shape. 10 years behind a computer fly by and it's caught up to me.

Nature never intended me to sit around my whole life and it's hard to get into a routine if you never had one.

Answer 3

The Followup

After you're done fixing a problem. Wait a few days, or a week, and visit or call the user.

Hi, this is <insert your name> I helped you with your problem concerning <insert the problem>. I wanted to followup and see if everything was still working.

I was horrible at these until I started making appointments in my calendar. I would schedule a follow up (unknown to the user) for a later date after the problem was fixed).

You can call if pressed for time, but the preferred method is visiting in person. Why?

If you do your job right NOBODY will notice a thing.

People will notice you; It's brand recognition baby!

Get out of your cubical, meet some people, let them know who you are and what you do.

Answer 4

Don't try to be irreplaceable. Spread your knowledge and documentation. You don't want to be constantly called on the phone on holidays just because no one else knows about the systems you manage.

Don't just learn specific apps and systems and OS'es. Learn the standards and protocols and conventions behind them, so you don't get completely lost if you need to move from one OS to the next for example. Keep up with the knowledge and your skills.

Also, take care of yourself, take breaks and do some exercise when not working. Get a good chair and table. Don't break your back and neck on your first year of work.

Answer 5

Do not be ashamed to ask for help,

Do not think that all you can do alone and now

Answer 6

It's never going to be 9 to 5.
I've always known it, I wish someone would tell my wife!

Answer 7

• Time Management
• Get a help desk or trouble ticket system between you and your users
• Did I mention Time Management? Well allocate chunks of time for project work, and separate time for trouble tickets, or else you'll spend all your time fixing tickets (which will make users happy) and none on projects (which will make your manager unhappy).

Answer 8

Be proactive, not reactive. Or, analyze the reactive measures you take / might take - and design proactive ways to make sure you don't have to.

Ergonomics are about more than buying a bumpy keyboard. It's never too late to start, but a couple of minor back injuries mean I don't have the kind of leeway I used to. I wish I'd researched this stuff early on.

• Learn how to lift; avoid back injury
• Learn about chair/monitor/desk height, and how minor misadjustments can lead to big health problems & pain
• Learn about carpal tunnel, and how to protect your hands/wrists/arms from repetitive stress injuries

You can make this easier on yourself with the exercise tips mentioned above, but don't make the mistake of thinking that these kinds of injuries only happen to old people, sick people, other people...

Answer 9

That it's only work.

Answer 10

Most important thing: Keep and Test Backups

Secondly,

All problems are able to be solved technically, however the human variable is what makes finding a solution challenging.

Answer 11

It doesn't matter how much you know and how best you can show it off. If you don't have patience you will burn out and hate your job.

Answer 12

The one thing I learned is: don't expect appreciation if you do a good job. Sysadmins generally go unnoticed until something goes wrong.

Answer 13

1. Not all problems have a technological solution, sometimes a properly written and vetted policy is the right answer.
2. Configuration management and standards always work best when implemented from the deployment stage.
3. It is never too early in the development process to ask for a security review.
4. Users are almost always uninterested in the system configuration, they just want their apps to work.
5. Users/developers always want to be involved in new service designs. Be sure to include them in your requirements analysis.
6. Always take the time to research a standard configuration and/or look for pre-existing scripts or software packages.
7. Document everything. Always assume someone else will have to manage your product. Being the only person that can fix a problem is job security, but it also means you cannot take vacations. Live by the HBAB (hit by a bus) principle.
8. Take vacations. If you do your job right, somebody else can cover.
9. How you interact with your customers and coworkers defines how well they will work with you. We all read and laugh at the BOFH, but let's face it. He's an asshole and nobody likes him.
10. As adopilot said, don't be afraid to ask for help or advice. Know where your expertise ends. We're all adults, no person worth working with will hold it against you.

Answer 14

One thing is dealing with people. They're impatient. When they call you they're already flustered. IF they're upset, its not at you - its because the technology that's meant to solve their problems seems to them to be causing more (and quite often, it does). They'll most likely vent to you: But dont take that as a personal criticism.

One thing that took me a while to learn was how to communicate and disconnect emotionally. Deal with the situation, and treat the person you're helping with respect and kindness. You wont get thanks (often), and you wont see any 'gratitude'. But at the end of the day, they'll remember you, and will call on you because you're polite and friendly. That's the number one thing I'd tell anyone.

Because, if you're polite, but dont know the answer / how to fix it, they'll at least feel better that you didnt upset them further.

Answer 15

Don't worry that much about your problem. Worrying will not get you too far.

Instead try to solve the problem, smile and remember that the sun is going to rise up at dawn tomorrow. That will save you a couple of years of your life and that's all that matters in the end :)

Answer 16

Simple is normally better, but simple just for the sake of simplicity is normally bad.

Dealing with management can be the single biggest challenge.

Things can come back to bite you in the ass.

Answer 17

Stick to your decisions until someone shows you a policy that counteracts it, or until you're over-ridden by management.

I can't tell you how many times (recently) I had to pull our security policy to explain things, like why your dog's name isn't a good password for a production system.

Answer 18

Remembering that humans are invariably stupid. That includes yourself.

Answer 19

I realize many computer geeks have ADD (like I do) and love to hyperfocus on a problem. That's excellent. My stubborness has kept me working on problems that almost everyone else gives up on.

But,

This doesn't apply at 3 AM. If you are paged to respond to a problem and you find it isn't a simple reboot or misconfiguration, STOP. Take notes. If you tore down a configuration or a machine to fix something, STOP.

Make a list, even in your head, of what you can triage to keep the problem from getting any worse. If that means you have to back out of the changes you did just to get to status quo ante, do it. Think in terms of, "I can restore AR in 20 minutes but the other machines will take longer. One of them I can't bring up now, so I'll just do the AR db and have to hold off on everything else."

And go home at 4 AM if need be. Or sleep.

You may have to tell the boss, "I came in, worked on it, the problem wasn't what I thought it was so I had to back out. Let me regroup for 20 minutes and see what I can find out."

Because it might not be on your end anyway. It might be something you need to hand off to networking, development or your vendor's tech support.

If you made a change and the change did not fix anything, ALWAYS back out of the change! I hate getting something working and then finding two dozen unrelated things I did that didn't fix it. If you need to make those changes after all, identify the original problem first!

Point is, I love solving problems and I love sticking with a problem and not letting it go until I have a resolution. But I hate deathmarches, 3 AM calls that extend to 8 PM and firefighting in general.

Answer 20

When I started, having knowledge of Microsoft Active Directory Roles and implementation and migration was the key for a quick start.

But to roll your way you can have as a guideline:

• Try to refer to RFCs
• Practice a good information mining for all the setups and projects you're after, and use a test environnment
• Make backups! A good backup strategy is very important.
• Document your work but not only for you.
• Learn TCP/IP
• Learn one or many scripting languages (PowerShell/Python.)
• Practice Unix/Linux.

Answer 21

• Don't worry so much about checking your backups. Test your restores and have a bare metal restore procedure.

• Before pressing enter, reread that command.

• Do some design work before implementation. You wouldn't start programming a large system without designing it - don't make the same mistake as a sysadmin.

• Sometimes a system can be less than perfect. Learn to recognize when that is the case.

• Don't overengineer redundancy when there's a weak link upstream.

• Use revision control - it's important even when there is one admin.

• Ask for help, even if it's nothing more than a second set of eyes.

Answer 22

• Don't test things out on a production system in the middle of the day. Even if you are pretty sure that the thing you want to do will not impact the normal operations of the server. Even if management will not spring for dev/test servers that mirror production, at the VERY least set up a close facsimile in a VM and test what you are doing (be it loading a service pack, kicking off a full SharePoint content crawl in the middle of a day, database maintenance...) before you go and screw up a heavily utilized server that was running just fine before you touched it.
• Same as above, but that tip also applies at 5 pm on a Friday evening. That simple thing you decide to do before walking out the door will take down the whole server and you'll spend the whole weekend troubleshooting.
• Servers are not like your home computer... you don't have to install everything that is in Windows Update just because it is there, and rebooting to fix random problems is a last resort (not a standard part of the troubleshooting process)
• Time invested in planning and testing up front will save you many headaches down the line trying to fix what you didn't setup correctly at the beginning.
• YOU are responsible for making sure that you keep your knowledge and training current.
• Learn the business side of the house as well. There are MBA's without a clue how IT systems work, and gearheads that don't understand what their company does to generate revenue. Understand both, and you are worth your weight in gold.
• This last one goes against the core of my being, but... while you can skip reading the documentation on your VCR or vacuum, you really need to read it as a Sysadmin

Answer 23

Be as paranoid as you can afford - double-check what you're about to do, check backups before re-installing a machine, and always have a fallback plan.

Be friendly and helpful - the people around that you're doing stuff for, are your "customer", and they deserve proper service.

Answer 24

Remember to take time off. Depending on your personality, weekends and the occasional extra Friday isn't always enough to unwind and de-stress.

Answer 25

How important the service outage voicemails you make are, and how they affect users. For example, if you phrase something in a less-than-ideal way in a company-wide voice mail about service outages, the CEO/President and other various VPs could come after you and give flac if it is voiding your SA.

Answer 26

I wish I knew back then how easy plumbing is!

Answer 27

Buy stock in the companies behind Mountain Dew and NoDoz.

Answer 28

Unless you work in an enormous company, with segregated departments for telco, network, application, operations, security, etc, you're pretty much going to be in charge of all of it. I'm currently working at a company of about 130 employees, and I am in charge of the following systems:

• Entire network, including 2 servers, ~75 computers, 3 racks of various networking equipment (one for each level), 20 or so printers, 5 copiers, you get the point
• a Nortel Meridian PBX system
• our building security system, which includes access control
• wireless across the entire premises
• an enormous database
• help desk responsibilities
• phasing out old PCs with newer ones

Basically, anything with a transistor in it (short of shop equipment), I am responsible for maintaining. It's not bad, considering we pay for support contracts on most of the proprietary equipment (security system, PBX, payroll processing software, etc) and warranties on the hardware. However, when there needs to be 3 different techs from 3 different companies in on the same day, dealing with 3 different systems, I tend to spread myself a little thin.

Answer 29

One of my computer science professors gave me this priceless advice:

Users harangue you on performance, but they hang you on reliability.

Always remember that.

Answer 30

Some of these have already been mentioned, but...

1. Pick up a (physical) hobby that involves using your body differently from being in front of the keyboard (I find martial arts and juggling to be good all-rounders).
2. Food is important. You cannot run on only caffeine and sugar. Without food, your brain will not work at an optimum.
3. Shit always break in an unexpected way. Learn to trouble-shoot in a methodical fashion.
4. MAC addresses are only statistically unique.
5. Automate anything that can be automated, the third time you have to do it (or fourth, or fifth; first and second time are probably just happenstance)
6. Remember what I said about juggling? It's a good way of clearing your brain once you get stuck.
7. Read. A lot. Possibly even relating to your job. Reading is the primary way you have of getting accurate information of the state of your network (OK, bias, I've mostly herded networks and/or unix machines) and the faster you can read, the less time will elapse from looking at things until you have a grasp of them.
8. User account creation and deletion must be part of the HR discipline (not that HR do it, just taht they need to tell the sysadmin(s) as people join or leave the company). As a collorary, always change user passwords with the user unable to observe the changes.