Are there any well known anti-patterns in the field of system administration? [closed]

Question

I know a few common patterns that seem to bedevil nearly every project at some point in its life cycle:

1. Inability to take outages
2. Third party components locking out upgrades
3. Non uniform environments
4. Lack of monitoring and alerting
5. Missing redundancy
6. Lack of Capacity
7. Poor Change Management
8. Too liberal or tight access policies
9. Organizational changes adversely blur infrastructure ownership

I was hoping there is some well articulated library of these anti-patterns summarized in a book or web site. I am almost positive that many organizations are learning through trial by fire methods. If not let's start one.

Answer 1

Leaving automatable tasks to be automated until doing them manually takes up enough of the time that they cannot be automated, because doing the tasks manually eats all the time.

Conversely, premature automatisation. There's absolutely no need spending 3N hours automating a one-shot task that takes N hours to do manually (even if it's more fun automating than slogging through things by hand).

Answer 2

A. not testing restore - a backup can be verfied and ok, but how to restore?

How long it takes, what it takes? You have to know to do that in a stressed situation...

B. no configuration management, no uniformity - just a change here and there and I think I've tuned some here...

Who knows how to replicate a well done server if all quirks are not written down and there are no identical configurations in the shop? What if you succeed to restore data, but not configuration, apps?

C. no monitoring - having no idea how and what boxes are doing

This is twofold: a) you have to monitor for alarms to react in time before you run out of some resource or strange behavior and b) you have to monitor long-term trend to manage capacity (disk, CPU, RAM, network, ...).

D. no redundancy in your cfg - what happens when XX dies

This means planning ahead what you want of your sysadmin.

For me these are most important.

Answer 3

The most killing pattern is when the system administration department (or the whole IT) becomes a passive participant in the company. That is, they are viewed as a self-service where everyone comes with already formed ideas how things should be done, which takes exclusively user needs into consideration and not the needs of the complete IT ecosystem as a whole.

The second most killng pattern is when the system administration department turns into a bunch of button pushers, i.e. all software/tools are bought or developed and installed by third party and system administration get an official training and a manual and then only follow operating manuals and escalate to the vendor everything that is not explicitly in the manual. This situation may be very comfortable for (some if not most) system administrators but this is a disaster waiting to happen when the fact that no one really knows how the whole system actually works will bring it to the ground (think subtle interactions between components and the blame game between vendors).

Answer 4

1) over-promising and under-delivering (i.e. keeping user expectations realistic)

2) Not verifying backups until they are needed.

edit: I intended number 2 to include the restoration of files / data

Answer 5

Not monitoring AD account usage patterns like last login time > 30 days

(We have to do this for auditing reasons, but the results are pretty shocking)

Answer 6

• Keeping key information in one person's head / inbox / documents folder. If it's important, like vendor contact details, license keys, setup instructions, it must be available to everyone in the department who has authority and may need to access it, and in a standard place.

• Asking the person who knows about something to document it. This sounds good because they are the person who has the knowledge, but it's actually bad because they cannot easily tell what the important knowledge is. Better to have someone new deal with it, asking the knowledgable person any information they need and having them document as they do it.

• Unclear documentation. Anyone can fix a mid priority problem during the day with the whole IT department available to talk to. It's another matter to fix a high priority problem late at night when you are almost alone and have no clue why the system is setup how it is or why it doesn't match what the documentation says.

• Not tracking passwords well. So you quickly need an account, make one with a random password and then 18 months later it's still in use and nobody knows the password or which services will break if it is changed.

• Not buying vendor support for key systems because it's "too expensive".

• Inappropriate priorities. The IT people should be guided by management - an agreement of which projects are priorities, or in an emergency which systems are required first should be in place. If IT is trying to fix the business system, management is demanding email and users are demanding order processing it's a recipe for a mess.

• Inappropriate solutions - it's very easy for IT to get stuck in the mindset of "to fix it, the IT system must be working how it was earlier", when it may be more appropriate to have a management-IT agreement to "try for 2 hours, if it's not fixed then stop even if it's looking promising, and move to recovering from a backup."

• Copies of test files everywhere. You don't want to open a folder which runs a business system or website and see "website-new/, website-current/, website-copy/, website-testing/, website-test-dave/, website-use-this-one/, website-from-feb/, etc). Dev, production and testing should exist and should be separated with every department involved (IT, dev, project management, etc) knowing what should be where and agreed on how changes are approved. Also for config files.

• Change approval - even if you just have a verbal discussion first, don't go changing the way important stuff works without anyone else knowing. Up to you to decide what "important" covers for your situation.

• Bodged solutions left in place long term. I know you patched this server over to that network with old telephone wire just quickly so you could address an urgent problem. I know you don't have time to redo it properly. Make the time.

• Poor relations with the rest of the company. IT is a service which helps the rest of the company do their work. If they need huge files fast, make it happen. If you need managerial approval to buy hardware, get it. If you can't get it, communicate clearly that the huge files cannot move fast because management have prioritised some other expenditure. If you need archiving for legal reasons but have no budget then you need to fit archiving into your system as best as you can.