35
votes

As a developer, I spend most of my time thinking about the code, UI, data structures and so on, but (I bravely admit) I don't tend to consider the implications of my app for sysadmins and DBAs until it's time to deploy the app.

Firstly, I'm sorry. Secondly, what do you wish I, and other developers you deal with, would do differently? What things would make life easier for you, cause fewer problems, encourage co-operation, and reduce crashes, performance issues and configuration nightmares?

21 Answers 21

34
votes
  1. Think about and build in security from day 0.
  2. Use version control for everything: source code, documentation, configuration, etc.
  3. Documentation, documentation, documentation.
  4. Clean installation and de-installation, using platform-native packaging
  5. Separate configuration data from libraries and executables
  6. Support for running different versions in parallel for testing and migration
  7. Robust, configurable logging
  8. Lightweight, accurate, secure monitoring
  9. Application checkpointing and backup
  10. How does your application react to problems: out of memory, file system full, network down, missing/corrupted configuration files, time skew?
  11. Always have separate development, testing and production environments. With all the free VM sofware, there's no excuse!

Remember that your application probably has more states than up or down. Draw a state diagram. Most applications have states like:

  • down
  • initializing
  • recovery
  • up-but-not-accepting-work
  • waiting
  • checkpointing
  • processing
  • finishing up
  • shutting down
  • down

Think about what happens if the system crashes during each state. How will a sysadmin monitor and control state transitions?

2
  • 4
    Wow. That state diagram idea is AWESOME. I nominate it for coolest answer snippet of the day!
    – quux
    Jun 5, 2009 at 2:14
  • Just a nitpick: Security is more of a design problem. You must first define what "secure" means in your context (what should users be able to do, what is secret etc.). Otherwise there's little developers can do...
    – sleske
    Sep 27, 2009 at 21:31
17
votes

Distinguish the "user" from the SA.

A "user" needs to know how to use your software. Users don't care about things like how to install your software.

The SA doesn't care how to use your software, but needs to know some critical details about how to install the software.

Write documentation separately for each of those roles, including the information relevant for each of them.

1
  • 3
    I think it's worthy to mention that admins are supposed to be instant experts with each facet of their network, as well as the workstations and applications that run on them. When we have the finance people ask us how to do an update on their payroll software it's easy, when we have logistics ask us why they can't do their order, it is up to us to already know exactly what goes into their process of ordering. I think documentation about how each system actually talks to eachother is easily forgotten, leaving us admins looking "stupid" because we don't know the intricate details of their work
    – bobby
    Jun 1, 2009 at 20:11
9
votes

One of my wishes is inclusion of proper messages in exceptions and error codes. It's completely opaque to someone who has not developed the application what a JimmyNotAtHomeException: it's late! means.

But a message such as Unable to find jimmy - initial manual call_mother procedure is very helpful.

2
  • 2
    I agree. Please, have multiple log levels and document what goes into the log! Jun 1, 2009 at 12:58
  • Unfortunately, for some companies cryptic error messages are part of their business model to sell you support contracts. They don't really want you to understand.
    – knweiss
    Jul 15, 2009 at 21:39
8
votes

Communication, communication, communication. Every problem between a sysadmin and a dev can pretty much always be tracked back to a lack of communication. If prior to a project the sysadmins (or a representative thereof) and the devs got together and had a nice framework discussion, SOOOOOOOOOO many problems could be avoided down the line. I can't tell you how many things get fouled up because you develop everyone on one box in development only to watch it go down in flames in prod because the app then gets separated in to App server + DB server + interface server, etc. Kudos for bringing this topic up.

8
votes

Engage us early in the project. Like real real early, at functional spec stage.

Somebody else mentioned having to manually install on every PC, but the same applies for config and config changes. If you choose to store something like connect strings client-side, and need to update them on a regular basis, we will probably want to kill you.

Choose technology that can be properly centrally managed and configured, for the very same reason. Make sure that it integrates well with whatever central management tools we use.

Always test using the lowest common denominator. That means as a non-Admin, on the most primitive OS, applicaion suite and browser plaform in common use. We don't like having a required browser upgrade for all of our users landed on us at the last possible minute.

Don't jump to blame us when things go wrong. In my old job every time an app broke the developers would instantly point the finger at us. "You installed a new patch, you won't upgrade browsers, your security is too tight" or whatever. This generates a destructive atmosphere. We're on the same side, really, and we want to work with you to fix it, but we can't in those kind of circumstances.

1
  • Whish I could upvote twice :-).
    – sleske
    Sep 27, 2009 at 21:33
7
votes

Don't be elitist.

"Don't waste my time, buddy. You're just a dogsbody sysadmin; I write the software and you merely service it. So just shut up with your petty concerns and do as I say, OK?"

A developer actually said those words to me once(1). In email. CC'd to a large distribution group. The implication was clear: as a developer, he was lord and master of the entire software universe; and I was simply a day-laborer hired to deal with tasks too trivial for him to waste his valuable time on. Of course this is a nearly worst-case example, but you know, I've heard strong and weak echoes of that comment from a number of developers both before and since(2).

You may make more money than me (but don't assume it!). But it takes a team to build, operate, and maintain the systems that our users rely on. Ultimately we all serve them.

I get that your job and your skills are different than mine. I respect your abilities. I hope you'll answer my questions even when they seem elementary and stupid to you. I'll cheerfully return this courtesy!

I'm not on a mad power-trip, as so many bad (or simply uncaring) devs have said and thought and posted in various forums. But my concerns are different than yours, and my questions and suggestions are not in service of my ego. Indeed my job is to make you look better, by keeping your apps in tip-top running condition, available and responsive to all the users. To do it, I have to keep the rest of the network and systems also running in tip-top shape.

I'm fully aware that you have run into stupid, power-mad, and/or just plain lazy admins in the past. I'm trying not to be one and not to look like one. If you leave room for this possibility, and acknowledge it when you see it, I'm pretty sure you'll get what you need while those other assholes are still fuming about what an asshole their sysadmin is.


(1)He was also insisting that his program (a tool for writing and managing software requirements) needed domain admin privileges to install and run. It was a major security risk.

(2)I have also worked with many awesome developers who could teach when necessary, and learn when necessary.

2
  • Goodness me, what an idiot. It's bad enough saying it, but CCing it around the place is disgraceful
    – harriyott
    Jun 5, 2009 at 19:27
  • Agreed. That developer should really have been thoroughly chewed out by their superior (who was hopefully CCed as well ;-)).
    – sleske
    Sep 27, 2009 at 21:37
6
votes

Respect that the sysadmins have a job to do, and let them do their job. A lot of companies have incompetent sysadmins and this is often not realistic. But I have seen arrogant developers ignore the advice of systems groups even after the sysadmins have proven their competence.

Discuss the design of a new system with sysadmins. There is often valuable insight. Developers often look at discussions with sysadmins and giving initial requirements as "premature optimization". I actually saw the head of a development group say that it was a waste of his time do discuss requirements for new database servers with sysadmins and DBAs, even to the extent of describing whether it is a write-intensive or read-intensive load, or how much storage would be needed.

Discuss performance problems with sysadmins. Honestly only sysadmins are capable of properly interpreting performance metrics on systems. I've seen developers decide that Linux always leaks memory because the free memory reported by "free" always decreases, even after the 10th time the output of "free" is explained.

Don't draw conclusions without discussing it with sysadmins. I've seen developers get stuck on theories such as "databases are always diskbound" (they didn't know that iostat even existed), "RAID 5 is faster for transactional workloads" (based on a recollection of one database system that was moved from one hardware platform to another - it was a read-intensive workload, the RAID5 solution had more and faster drives spread across more controllers. But they forgot these details and only remembered the conclusion.)

Don't design a solution to a systems problem without discussing it with sysadmins. I worked in one pathological environment where developers would design a solution and come asking for small implementation assistance. The members of the Unix group besides myself, the head of the Unix group, and his boss, wanted to treat developers as "customers", not as co-workers trying to make an overall infrastructure function. The customer being always right meant not questioning what they were doing or why. I was the only one who would insist on having the problem described so that a correct solution could be determined. Don't act in ways that create pathological environments such as this. It doesn't result in a net benefit - instead, systems managers will act defensively and everyone will suffer.

You're not in school anymore. These are real-world systems and they do not act in an ideal manner. For example, not everything has zero latency. When a sysadmin warns you that clustering solutions are only for political purposes, and the added complexity of the system decreases overall reliability, take it seriously. You have to design for real-world failure modes, for example when you lose the server you're talking to via TCP, the connection will probably not be reset for you. Sysadmins understand real-world failure modes.

Either listen to what your sysadmin tells you, or complain to management that your sysadmins are incompetent and need to be fired. Ignoring your sysadmin makes no sense.

Consider how you will deploy your application. Realize that discussing this with your sysadmins makes sense. If you have 100 servers that are identical, differing only based on a single configuration file, you may want to consider storing the master copies of these config files in a central location. Realize how much better off everyone is if your application is easy to re-deploy. If there is a problem with a system, would you rather re-deploy in under a minute to a spare, or wait for ages while the broken one is repaired? If you can re-deploy your application, the OS can be upgraded more easily and safely. You might care about this in the future.

If you have a problem that you think might be due to the OS, it makes sense to immediately call the sysadmin to check it out. But after a cursory investigation reveals nothing, you have a duty to explain the problem.

Understand that there is a difference between "responding slowly" and "not responding at all".

3
votes

Configure and lay things out in a predictable way with predictable ways of changing them for the OS(en) you're developing for. This means everything. For instance: OpenLDAP has a weird way of doing loglevels; certain IMAP servers don't even have configuration files but must have options compiled in; some packages want their stuff to be in one particular directory path, which inevitably will break the conventions of a particular operating system. These things all stand out as warts on my usual configs.

It's a general rule, but don't assume you're special, and therefore blessed to break the general conventions of how software packages generally work on your platform, unless there is an abundantly good reason inherent to your software that requires it. "I strongly feel this should be so" isn't good enough to break everyone's usual setup; it has to be a reason connected to the function your software is trying to perform.

3
votes

When there are inter-server communications involved with the app, please include at least one sysadmin in the design phase. Also, clearly document dependencies on other services: SQL, SMTP, HTTP, etc... Don't make us guess what your doing or we can't help you when something's not working like you expected.

3
votes

Please make it possible to deploy your software on dozens or hundreds of systems in an automated fashion. If an organization needs a software package, sysadmins almost certainly don't have time to install it manually on every box. If a file needs to have licensing information, documenting how to provide it would be a great benefit.

Adobe has historically had some installers that were a real pain to work with; please aim higher than that!

2
votes

Think about scaling from day one. Sysadmins can do wonders by throwing money/hardware at a performance problem but sometimes no amount of these will help - in particular think about locking - sometimes you can't buy yourself out of a locking problem. Thanks for asking though :)

Oh and try to be 64-bit where possible, and multi-threaded too :)

2
votes

Design for operations.

2
votes

Beyond everything else here...

  • Ask for a simulated production environment (a development server or VM with the same configuration as the live server) and then use it to test the release process. Then provide us with this release process, including a list of changes and the order in which they should be applied (i.e. 1. Enter maintenance mode, 2. apply SQL update, 3. update source to X version, 4. leave maintenance mode, 5. pray)
  • Make sure you have a maintenance mode that can kick users out to retain data integrity. You don't want us running a big system update across several servers with users logged in and executing transactions... that's a recipe for fail in most cases.
  • Use a development model that is somewhat standardized. For instance, all of our new applications at work (web group) use Zend Framework.
  • Provide us with changelogs that we can read, including a list of bugs fixed that we can look up to get an idea of the scope of changes. Sometimes we can identify potential problems just because we have a different point of view.
2
votes

Although unrealistic it would be helpful if developers were forced to work in a production sysadmin or dba role before being set loose on the world.

So many specialized applications I deal with have either no clue about installation in a managed environment, or commit atrocities of database design.

1
  • Fully agree. I'm a dev, but have worked as admin for a few months, and found it an extremely valuable experience. It really broadens your horizon.
    – sleske
    Sep 27, 2009 at 21:40
1
vote

1) The log file that logs the errors in details. or good error tracking system like ELMAH.

2) The detailed documentation for installation, implementation, and SA guide.

3) Plus the stuff mentioned above from other amazing SAs. :)

That's all I can think of right now.

1
vote

The book Release It! has a nice selection of horror stories about what can go wrong in production. Reading it can provide inspiration/ideas for designing with operations in mind. (It's written by Michael Nygard, who has worked on both the development and operations sides.)

1
vote
  • Don't develop without specs
  • Document (or ensure that those who do document do so accurately)
  • Don't be afraid to support the customer (as a higher tier level of support)
1
vote

In my experience, the thing that would make the most difference is if developers would think about deployment from day 1. As soon as you start conceiving of a new feature in the production/customer environment, start thinking about how it will be deployed into that environment, and how it will run.

Once they get into the development process, it's not too late, but it can take some time before they are able to shift their perspective that far; they don't realize how abstractly they are viewing the codebase until forced to confront it. In their mind, it's just a "component". Of particular interest is how it will be deployed into a pre-existing environment, running the previous (or older!) version of the software. Deployment discussions can have significant impact on how the architecture is adjusted to accommodate the new feature.

1
  • i agree with your comment. as a deployment engineer I deal with incredible amounts of complications during deployment that could be made simpler if the engineer only had my point of view.
    – djangofan
    Dec 1, 2009 at 18:48
0
votes

Something I like that I haven't seen yet is Configurable. If you have an app that uses any sort of config file, make everything configurable.
At my company, I wrote a simple app that will export a piece of out database. The next week I was rewriting it to allow a small part to be turned off. Every week since then I have had to rewrite parts and rebuild it just to change a small feature. I just barely added an xml config file, and now it is way easier to redeploy.
I love config files.

0
votes

I wish that developers would have better separation from QA tasks. I think that its nice when a developer is able to create some unit test cases for a project he/she is working on but it would be nice if those tests were passed off to QA. In fact, its nice when developers give a little help to QA engineers because it benefits DEV in the end.

0
votes

Make sure it's supportable: in addition to everything else mentioned here, look at this post on SO - https://stackoverflow.com/questions/205374/what-are-the-core-elements-to-include-in-support-documentation/

Not the answer you're looking for? Browse other questions tagged .