Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We're about to get our first sysadmin to look after a multitude of SQL Servers which have previously been awkwardly looked after by a mixture of the developers and IT support. It's long overdue, and we've been trying to persuade the higher-ups to agree to one for years.
Well, finally they did but the salary we were able to offer wasn't exactly inspiring to say the least. Nevertheless, we've snagged one somehow.
What I'd like to know is what are the early signs to look out for that a new sysadmin doesn't really know what they're doing or what dangerous habits to look for, with particular focus on SQL Server. I'm a little nervous that our bargain-basement hunting may not work out too well, which has been the case with other roles.

Any thoughts, please?

share

locked by Tom O'Connor Jul 19 '13 at 8:18

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

5  
I first read your sql-server tag to mean that sql-server was one of the warning signs... –  Mikeage Dec 7 '09 at 14:27

16 Answers 16

up vote 123 down vote accepted

Take this first part with a grain of salt, because it is perhaps influenced by my having worked as a contractor for so many years.

Consider looking at a contractor if your ability to pay is such that you can't attract top talent in a full-time capacity. If you're paying too little and asking for too much you're going to either get poorly skilled employees, employees who have glaring defects that may not be skill-related (poor interpersonal skills, substance abuse issues, etc), or you'll end up with a "revolving door" position where employees work for awhile and leave for better pay.

If your company is hung-up on both paying too little and needing someone for a given period of time, as opposed to fulfill a given set of tasks, then you're probably in a hopeless scenario. Likewise, if the tasks will keep a full-time employee busy and the company is planning on paying too little, then it's also hopeless. You will get what you pay for in the long run, one way or another.

My guess is that you don't really have a full-time need, and the company could probably spend the planned salary, or less, on a contractor who would do everything you need.

A contractor is much easier to "get rid of" if the relationship is a "bad fit". A contractor can typically be much more flexible than a full-time employee re: work logistics (weekends, evenings, etc). A good contractor is going to treat your company's needs with a very high degree of skill and care because they know how easily your company can sever the relationship and look elsewhere.


This is going to sound really trite, but more than any of the other items below, pay attention to your sysadmin's ability to communicate with others. Basic writing and speaking skills are important, and do a lot to indicate the state of the mental processes occurring "behind the scenes". A sysadmin's work should involve communication with other IT and non-IT employees, and an ability to communicate effectively is essential. Having an ability to form analogies and communicate abstract concepts is certainly nice "icing on the cake", but if your sysadmin can't even write complete sentences or speak complete thoughts then it's hopeless already.

There are points in everybody else's answers that ring true to me re: a "bad fit" (be it an employee or a contractor). I've been the guy who helps companies bridge the gap between firing a bad sysadmin and hiring a replacement, and I've seen a number of bad scenarios play out. (Being the person who is changing passwords, looking for "back doors", etc, while the sysadmin is over in the CEO's office being fired is fun work, but stressful, too.)

Some "IT specific" nasty attitudes I've seen (cribbing from some parts of other posters' answers here, unashamedly) in dysfunctional situations include:

  • Rip everything out and start over: It's one thing to identify something that's a "ticking time bomb" and take care of it, but often in IT I run into (often immature and entry-level) sysadmins who seek to "build an empire" in their image, and obsess over removing old infrastructure for the sake of installing new. It's one thing to make a business case supported by facts and ROI projections, but I've seen this particular dysfunction as being nothing more than a strong personal drive to replace systems for the sake of replacement.

  • I can't tell you that: These are the sysadmins who, while taking a strong personal ownership stake in their work, go too far and become overly possessive, secretive, and paranoid. The computers belong to the business, not the sysadmin. Failure to document work, disclose passwords, or be open about how systems work (or fail) isn't a good sign. I've heard some sysadmins cite "security" as a reason to be secretive, but security by obscurity isn't security. I've also heard sysadmins with this attitude say things like "Yeah, but if I give the passwords to so-and-so they'll just screw it up." Usually, this is accompanied by a veiled or overt statement of fear for being blamed if something goes wrong subsequent to the disclosure. If a business is so unstable that this fear is justified the sysadmin would do better to leave and find another job than to play games with secrecy.

  • Blame somebody / everybody / anybody else: These are the sysadmins that constantly cite third parties, their predecessor, or the users experiencing issues as the cause of problems. Certainly, there are issues caused by all these factors, but a pattern of consistent and repeated finger-pointing is a bad sign. We've all had to deal with hardware errata, software bugs, and users creating problems for themselves. Being able to identify one of these sources as a root cause to an issue doesn't make it finger-pointing. Being unwilling to investigate an issue and identify a root cause, though, combined with the reaction of vaguely waving hands and saying "It's gotta be that buggy Windows / Linux / Cisco router / etc..." is cause for concern.

  • Power trip: These are the sysadmins who delight and setting up roadblocks for users because of a personal agenda or a perceived business agenda. Again, it's one thing to place limitations on users for justifiable business reasons. It's quite another, though, to be the "preventer of IT services" simply for the mad power rush of being able to control others. I've seen this particular dysfunction extend into really nasty things like "e-stalking" of employees by reading their email, covertly performing screen / session captures, listening to phone calls, and just generally being a "creepy" person to others.

  • Policies don't apply to me: Often combined with the "power trip" attitude, these are the sysadmins who refuse to be subject to the IT policies that they, themselves, otherwise enforce or dictate. While it can be benign and harmless, I've seen this cause nasty situations like threatened sexual harassment litigation (a sysadmin surfing and prominently displaying work-inappropriate content). Sysadmins are in trusted positions, and need to maintain an attitude of professionalism. Part of that attitude means playing by the same rules and being accountable like everyone else. Just because we have the ability to perform activities "off the record" w/ our elevated access permissions and rights doesn't mean that we should do it.

  • Can't admit weakness: It takes a strong person to say "I don't know the answer to that, but I can find it for you." Everyone has gaps in their knowledge and experience. This particular dysfunction often results in situations where a sysadmin ends up vastly over their head. It's important to take calculated risks in career development, and it can be said that great personal growth occurs when people "bite off more than they can chew" and succeed. On the other hand, great expense (or outright failure) for a business could easily occur when a sysadmin decides to tackle important issues like disaster recovery or IT security and fails for lack of ability. Managers who unreasonably disallow their employees access to third-party resources / training / support can help to create this kind of culture. No one should be penalized for admitting that they don't know how to do something while expressing a willingness to help find the right answer (or, even better, learning how to do it themselves).

  • These are my toys: This is the sysadmin who treats the business IT infrastructure as an exciting toy. It's one thing to identify a particularly interesting technology that happens to fulfill a business need well, but it's quite another to influence a business to spend money on technology for the unstated purpose of being something fun to play with. I've seen situations where sysadmins became enamored with a given technology and decides to bring that technology in to solve a problem not because it's suited to the business need, but because it's something they'd like to play with. I've seen this happen all kinds of things: fiber optics, virtualization, SAN gear, wireless networking, etc. Management should keep this in check as much as possible, but non-technical managers may always understand if a given technology really is something the business needs or not.

  • I've always done it this way: This is the sysadmin who is dead set in their ways. Usually, I've found this combined with an attitude of "I don't want to learn about new things", too. Our field is changing. Some of the work that we did 10 years ago is automated today, and some of it remains the "same old, same old". Everything about our industry is constantly being revised, updated, and refreshed. Best practices change more slowly, but even they change too. It's unreasonable to expect that every sysadmin will keep up with the "cutting edge" of technology, but it's also unacceptable for a sysadmin to languish in years-old technology showing no sign of interest in updating skills. If a business is a growing concern, its IT operation should be forward-looking. (Obviously, there's a balance here, too. You can tip the scales too far and end up in a "these are my toys" scenario, as well...)

  • No understanding of business: Business "does IT" because it helps in doing business efficiently. Any other use of IT in business is counter-productive. Too often I've seen sysadmins who are unaware of basic concepts of accounting and business (revenue less expenses equals profit, etc). I would never expect a sysadmin to be an expert in accounting, but I would expect them to understand the basic way in which a business incurs expenses for the purpose of turning a profit. In poor economic times, especially, it's nice to have your sysadmin understand where the money comes from and why the business makes the decisions it does related to where the money goes. A sysadmin who believes that IT stands apart from the "business" part of the business isn't an asset.

  • No desire for continuity: In today's occupational culture, it's should be assumed that we'll all work for a variety of employers. Our job today isn't, statistically, going to be our job forever. A good sysadmin should prepare documentation not because "they might get hit by a bus", but because their eventual replacement will need it. An unwillingness to prepare documentation because of perceived "job security" reeks, to me, of an individual who has no desire for upward mobility. I don't work for a single employer anymore, but if I did I'd be planning for what I was going to be doing next, and keeping documentation up to date so that my replacement will have better time of it (just like I'd like from my predecessor at my next job).

share
38  
It was hard reading this. Every point you make is valid, and each one forced me to self-evaluate. –  Nic Nov 14 '09 at 20:37
    
Very good points. Some of those problems are infectious as well. Even if a "bad admin" finally moves on or is moved on or loses their credibility, their finger-pointing and "tear it all out" critiques can been like weeds taking root in the minds of management and whomever is left on a team. Kind of similar to the "say it 2 times and it must be true" syndrome. –  damorg Nov 14 '09 at 21:44
1  
Evan ... you need a blog ... Never mind I just subscribed to your user feed. –  Joseph Kern Nov 15 '09 at 0:31
3  
@Nic: Writing it definitely made for some self-evaluation, too. Having just a little bit of many of these attitudes isn't necessarily bad, so long as it's kept in check and appropriate to the situation. I present a little bit different "face" to each of my contractor Customers, and it's interesting to see how I tailor my attitudes depending on the Customer's needs and the dynamics associated with interacting with my contacts there. In any case, I really do try to "play it down the middle", and I try not to fall deeply into any one of these categories. –  Evan Anderson Nov 15 '09 at 2:28
1  
@Joseph: I've found it impossible to keep up a blog. I'm often at a loss for topic ideas, and I seem to take way too long to write anything for that format. When Server Fault provides the inspiration, though, I'm happy to run on at the mouth (keyboard?) for entirely too long. The interaction on Server Fault is a lot more fun that I think I'd ever get from a blog, too. –  Evan Anderson Nov 15 '09 at 2:36

I'd say that the two most important things to look for in a good sysadmin is structure in their work and a thirst for knowledge - so an absence of one or both of these would be an early warning sign.

Almost nobody will be able to walk in a do everything on day one but if you have the time for them to pick things up then don't focus on a lack of any particular skill/experience, if they're a good sysadmin they'll be researching the bits they don't know within minutes of walking in the door and will be up to speed quickly.

They should also be interested in what 'reference/test' systems/tools they have - this will show that they want to try new things without risking the production environment, they may want TOO much of this kit but better that they want it all than none at all.

Oh and consider using http://jobs.serverfault.com/ to find someone ok ;)

share

Documentation of work. And some more documentation of work.

Edit: That came out wrong, but you get the idea. That's what a good sysadmin does, so that you can check in on his/her work.

share

When a problem occurs, in either production or a testing environment, does this person investigate the root cause, or assume it was a one-time incident?

Since this person won't have all the answers, does he or she have the interpersonal skills and modesty to seek help from others?

As @Chopper3 said, a thirst for knowledge.

share

As mentioned by Chopper3, evidence of a structured, disciplined approach and willingness to learn are good signs.

On the flip-side, early signs of poor skillset or "fit" can include lack of patience with questions, unwillingness to explain technical reasoning, constant and aggressive defensiveness, never-ending finger-pointing at colleagues and/or predecessors (if there's merit to this, there's no reason to flog it to death over and over).

Also, a desire to "rip it all out" or redo everything "the right way" are tendencies to watch.

A certain amount of "well I would have done it this way" can be natural but unless there is also an assessment of the current environment and its weaknesses, and a reasonable plan to correct those problems and meet whatever other requirements there might be, and lots of discussion, I'd be wary.

share

Openness. You want to be able to see what he's doing and how he is doing it.

I would say the number one symptom of a trainwreck in progress is if the guy locks down everything and forbids anyone else to have access to the systems.

He may give all sorts of "Security" related warnings about allowing other people to have access and accounts and root privs on other machines, but often that is a smokescreen for someone who wants to look important and put your junk in a vice. It's easy to manage access in such a way as to allow access but maintain the security and accountability of a system.

Oddly, people do better work when they know it is going to be seen by someone else...

share
7  
On the flip side, this person should show some concern regarding security, stability, etc. By nature sysadmins are possesive of the systems they manage. Taking a strong security stance and showing strong tendencies to take ownership shouldn't be a cause for alarm unless it hinders the work, culture, or goals of the company. –  joeqwerty Nov 14 '09 at 14:11
1  
Security is simply one means to an end, not an end in an of itself. Security is necessary to maintain reliability, stability, and auditability of a system. Time you spend locking down a system so that it has only one suid binary could also be spent automating the build system. For many, security is a fun game to play, but such a game isn't really of interest to most enterprises. Also, many people lock a system down to exclude eyes that would audit the admin's work, and that's when things can really go off the rails... –  chris Nov 14 '09 at 15:26
3  
Meh. The new admin may be tasked with getting your servers compliant with some regulation or other. A good first step, after gaining a thorough understanding of the system, might be to lock developers out of production systems they are used to having access to. As long as someone else still has access (say, the IT Department and the Software Development lead) this is valid, IMO. –  Kyle Hodgson Nov 14 '09 at 15:32
    
Chris, you just described my immediate predecessor, who somehow managed to last four months before the company woke up to him. It's took me months to find and rectify everything he did. –  John Gardeniers Nov 14 '09 at 21:30
    
It's surprising how much damage can be done in just a few months and how long it can take to set things right. Just think though...sometimes an admin like that can stay entrenched for years. –  damorg Nov 14 '09 at 21:37

Chopper3 and damorg make very good points. In addition I would stress giving the new sysadmin time to settle in and get comfortable in both the position and the company. There's a human aspect that needs to be considered as it's generally akward and nerve wracking being the "new guy". They'll need time to "figure out" what you've got, how it's configured, etc., etc. and they'll need time to start feeling comfortable with the people and culture of the company. Don't be hasty to evaluate or make judgements about skill or personality traits that you see in them that might actually be a result of nervousness, etc.

share

Early signs of a bad sysadmin....

  1. Sleeps in Server room
  2. Comes out of the server room saying 'Please tell me we have good backups!'

Will add more as I think of them.

share
    
@Crankyadmin: ROFL. –  joeqwerty Nov 14 '09 at 15:30
1  
3. Has a username that includes unpleasant character traits. =) –  Wesley Nov 25 '09 at 17:39

Watch the questions they ask. It sounds like you have a fairly complex system (multitudes of SQL Servers), so if it were me the first thing I would be doing is bothering anyone and everyone who will talk to me about what they all do, who depends on them and why, and taking copious notes. I would be doing this in as close a proximity to a white board as possible.

An attempt to find, and test the backups should be made. If there are performance problems, I'd be running the profiler and perfmon (or similar tools) to try to figure out what queries are causing them. I'd be reviewing the hardware to make sure the multitudes of SQL servers have at least a hardware mirror each.

Checking that there's some sort of monitoring system, and implementing one if there isn't. Nagios and cacti/rrdtool/mrtg come to mind.

Above all, if you see someone starting to take action to change your actual SQL servers (with the exception of measurement) before they have a thorough understanding... well, this is more inexperience than lack of skill, but it would scare me.

share

There will be some growing pains, just like braces are painful but they gradually pull teeth into a good and correct alignment. The admin will need to get settled in and then there will be some adjustments as he pulls things into proper alignment.

Biggest sign of a good or bad admin is how changes occur. Does he engage users in the discussion about WHAT and WHY things need to change? Is there a reason he wants to lock certain people out of a particular system? Like anything in a business you have to have a reason and a vaporous "for security" doesn't cut it. What are the risks of leaving it as it is? Why is what s/he wants to do better?'

If users feel involved in the process, and have a chance to explain why things are as they and can explore the alternatives they will be much more inclined to assist and can offer insights as historical reasons for odd things in the system. I find myself consulting often with some of our finance people who have been at my company for decades about why the heck is something setup so weird.

Actually this kinda goes for any position even outside of IT.

share

Some excellent answers so far; I'd like to add:

Being afraid of hard and/or dirty work. One shouldn't go out of one's way to invite hard and/or dirty work on oneself, of course, but when some nasty job needs doing it's a good sign if the person shows a willingness to roll up their sleeves and muck in.

Failure to realise that the reason they do their job is for the customers. Ultimately this is what it's all about; people need to be able to come in each morining, log on, and get their stuff. An admin who doesn't keep this in the back of their mind is failing in their job.

Letting themselves get out of touch with the people. It's easy to get suckered into thinking that you're on some ivory tower, and that you don't need to deal with users or answer calls. Users are a valuable and important source of feedback, and an opportunity to learn if something you've put in place is working well or not. Arranging to spend some time every month working with the helpdesk is great.

Being too much of a "by the book" person. OK, there are a lot of perfectly good and documented ways of doing things, so this one is most definitely NOT a case of one extreme or the other. I mean the type of person who clings to their MCSE manuals and treats everything in there as if it was the One, True and Only way.

Failure to take a proactive approach. A good admin will always be anticipating potential sources of trouble and dealing with them before they become a problem. A bad admin will just sit back and coast along letting things slowly disintegrate around them until the dreaded day when something collapses during office hours at a critical time.

Being a technology evangelist. I mean the type of person who would try to force in their favourite OS, apps or platforms irrespectively. You say you have SQL Server (which means you're a Windows house), so be on the lookout for someone who is constantly extoling the virtues of Linux or Lotus Domino, for example.

Forgetting to cover the basic stuff. It's a fairly huge field, and in order to be good at the complex fiddly stuff one needs a decent grounding in the basics. A good person will be almost immediately asking you about things like your backup strategy, your central documentaion repository, if you have a standard PC image, when the last time you had your firewall health-checked was, and so on. These are the things that keep you ticking over from day to day, and are equally as important as anything else.

share
1  
The first paragraph made me smile. Once I had an employer who wanted me to dive into those dusty dark corners to replace network cables, but also wanted me to look cool: wear a tie and a jacket :-) –  Anonymous Dec 8 '09 at 14:49

There has been some excellent answers already, so I won't repeat any of it but will add that while this will not necessarily indicate a bad sysadmin, someone who works for peanuts can be expected to be, or at least very quickly become, dissatisfied with the job. That person will inevitably be thinking as much about the next job, and how to get it, as the one he/she currently has. Can you realistically expect someone to give their all in such a circumstance? With that in mind, make sure the documentation is up to scratch.

share

Inability to prioritize, and multi-task.

share

sounds to me like your gut instinct has already told you that you've made a bad hire & your looking for evidence to re-enforce that instinct.

Here are some bad SQL Server habits IMHO may be a sign of an inexperienced DBA.

  • rebooting server or re-starting the sql server service to 'fix' problems
  • adding additional transaction log files on different drives, because the current log is running out of disk space
  • shrinking the transaction log is a regular part of maintenance to control log size (shirking data files is even worse)
  • using the open table option in SSMS (triple bonus points for using against production)
  • being entirely reliant on the SSMS GUI for performing backup and restore
  • not understanding the difference between a sql server login & a sql server user
share
    
"using SSMS GUI for performing backup and restore" is a bit too sweeping of a statement. "being entirely reliant on the SSMS GUI for performing backup and restore" might be more realistic? –  Wesley Dec 7 '09 at 15:24
1  
suggestion noted! –  Nick Kavadias Dec 7 '09 at 15:36
    
So far the signs are cautiously promising, although he's not had to do much yet other than sit in meetings and learn the setup... –  CodeByMoonlight Dec 7 '09 at 17:13

Time management.

Schedules activities around a work plan. Knows maintenance needs to happen in down time. Manages backups. Tests backups/restore. Has an active recovery plan - it's not a matter of if but when hardware will fail. The should be inteseted in knowing if things break or getting out of hand before you notice. Thinks that nagios or solarwind is a must in knowing if systems are alive or dead.

Documentation.

Should work with a ticket system. Puts in tickets on behalf of users not able willing to Do so to track issues.

Attitude.

Totally open to help the business. There is no can't , open. Says I can do this if you give me these resources.

share

I'd like to add something, its a type of admin. Usually entry level and inexperienced.

I call them shotgun upgrader

Every now and then in the update cycles systems that worked stops and several hours, somtimes days, are lost. The shotgun upgrader has struck again. A good sysadmin should know the dependencies that are needed for your production system to run and and not break it everytime an upgrade has a potential to do this. I caught one in the act once.

He was in the process of doing an "unattended" dist upgrade of one of our debian systems. aptitude -y dist-upgrade > /dev/null 2>&1 (Its horrible, dont ever try it, it most likely wont boot again)

I asked, what are you doing? He replied redirecting to /dev/null, it clogs up the screen!

share
    
+1 for aptitude -y dist-upgrade > /dev/null 2>&1 –  Hubert Kario Oct 30 '10 at 13:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.