7

I have a server that host few websites, I haven't done any maintenance for about 2 month other than looking at a couple logs and compressing them.

Just wondering what are a regular maintenance you need to perform on your Linux server? Are there any scripts that let you do the maintenance?

1
  • 2
    My monitoring tools do the closest thing to "regular maintenance" that usually matters. And all they do is monitor. Apr 14, 2011 at 0:09

6 Answers 6

10

The most important thing is to keep your server monitored so that you know what's going on the server at every moment. It's important to keep track of disk usage, cpu usage, load, memory usage, traffic, etc. There are a lot of open source tools that can help you with this task. This also helps you to plan future upgrades.

On the other hand, you need to perform regular and automatic backups and it's very important to test your backups once in a while to make sure they're working.

It's also important to check for updates, especially security ones, and install them when needed.

Everything that can be done in an automatic way should be done in an automatic way, but do not forget to check your logs once in a while.

Hope this helps!

2
  • 3
    +1 for package/security updates.
    – EEAA
    Apr 14, 2011 at 1:41
  • +1 for updates. Do it like microsoft does - put in a specific time every month for upgrades. Check for out pof band highly important ones, but at the end of the day I have one weekend (sunday, my maintenance interval) every month to make sure I dont have any outdated stuff.
    – TomTom
    Apr 25, 2011 at 10:41
3

There are two useful classes of tool here:

  1. Monitoring Tools like Nagios let you keep track of usage and other stats.
  2. Configuration Management Tools like Puppet and Chef let you make statements about the expected state of a system (what packages it should have installed, what the contents of files should be, etc), and then run periodically to ensure that the system conforms.
3

Patch your OS to get the latest bugfixes and security updates. This is the single most important step you can take. Beyond that, you need to have monitoring in place to tell you the state of your system over time. I have to disagree with the point above that monitoring is not maintenance. You need to get a baseline understanding of your system and how it runs to see when things are not working or need maintenance in the first place. The only way to really do this is to graph everything. We use a combination of Nagios, cacti, and ganglia for this.

As far as scripts, you mentioned compressing logs, so look at logrotate. There is also logwatch.

1
  • 1
    Most linux based systems that are decently secured don't need to be patched/updated that often. As long as Apache/PHP, or whatever is public facing is up to date from security problems, he should be fine. Uptime in years is not uncommon in the linux world. It is not common to reboot your server every month on patch tuesday, thats windows.
    – Brian
    Apr 16, 2011 at 13:59
3

The previous ones to this question are all worthy answers, but they are all colored by large-scale thinking. If you have only a single host, providing web sites whose failure will not immediately kill kittens, you may want to think somewhat smaller. I would suggest the following:

Use logwatch or similar system that aggregates your logs and mails you a summary. Read the summary at least every other day. Use a tool that crunches your weblogs (e.g. analog) into a readable summary. If you want to be a bit more ambitious, use a cloud service to monitor your sites.

However, the essence of systems maintenance remains the same: kill each issue that occurs dead so that it can never occur again. This is an important point, because it means that there should be no "regular" maintenance.

Now for the bigger scale:

Among the first issues you have when you scale up is that something breaks and you don't know it. This must never occur again. Then you get a full-scale monitoring solution. Also, having seen the issue occur on one host, you want to prevent it occurring on any other host. That's when you get a configuration management system.

But you must not at this point be complacent. All your effort should go into permanent solutions.

1
  • +1 for preventing problems from happening again. Apr 25, 2011 at 10:30
1

Maintenance is done by monitoring. If you have an machine exposed to outside access you have to have some checking tools: rkhunter, chkrootkit are basics with mails sent daily .. smartd must check the hdds and send mail in case something start going wrong, tripwire for integrity checker, aide for IDS+ integrity checker, and a lot of others .. of course it is supposed that system is already hardened (at least with default SElinux targeted policy), would be great to do performance monitoring with nagios,zabbix and others .. (to keep on eye on out of ordinary events and status of the machine)

3
  • 1
    Maintenance != Monitoring. While monitoring is a crucial part of any type of systems administration, that's not what the OP asked about.
    – EEAA
    Apr 14, 2011 at 1:38
  • 1
    I disagree :) maintenance IS monitoring .. maintenance is the act the comes after the information (given by monitoring). At this point i will add (something that i thought is known) that automated scripts used by monitoring IS part of monitoring not of maintenance .. and with this ideas that means that maintenance is only done by human factor. With all this in mind i will repeat : with good monitoring there is no need of maintenance. A good professional will just have follow the email alerts ... MAYBE to look by himself in logs.(IMHO) Apr 14, 2011 at 18:21
  • Monitoring is very much maintenance. We have systems where the main cause of problems is disk space running out. (Databases don't like that much) We have to keep an eye on them, and throw alerts when they hit about 85% full. Then we can add another disk, and expand the volume groups.
    – Brian
    Apr 16, 2011 at 13:56
1

Marcos did a nice resume of common points of MAINTENANCE: monitoring, backups and updates.

I need to repeat (by people comments) that monitoring != maintenance.

I would like to add: cleanups of old users, old cron tasks, etc. Passwords renewals. Evolution of system configuration (looking at monitoring history) (and this includes evolution of monitoring and backups) with the life changes of the server. Security audits. Custom reports (i.e. detailed usage of services, shared resources, security reports). And...

Restore backups randomly (without needing to do it). Also always include a copy of the MBR (Master Boot Record) if you do not use an automated deployment system to restore the whole server.

I've seen firewalls survive 500 days of uptime (yes, that is a BAD maintenace, because there are about 10/15 kernel upgrades a year, so big uptimes, use to be equal to vulnerable kernel) without any human maintenance. It's just a good config on partitions, syslog and no major configuration changes.

You ask for scripts. Maybe there are generic pre-packaged scripts, but it is more about DIY (Do It Yourself). So first you learn or read how your system works (this includes distro specific REAMEs, man pages, policies and compilation options), then yo think how your system should behave, and then you write custom scripts (cron, custom commands, etc) to manage the combination of O.S. + desired behavior/results/reports/change-management/rollbacks/reactions/etc.

Also, available generic tools (monit, nagios, cacti, zabbix, etc) can be extended with your scripting skills and available third party plugins.

In example: do you need only to know "if partition is full", or do you need too to control disk temperature, free inodes, files and folders rights, checksums, S.M.A.R.T. status, bad sectors, filesystem checks, and estimated life cycle of physical hardware?

And as well as the backups, the monitoring system + customizations, should be validated where possible, simulating failures, attacks, etc.

Custom scripts are better, when you can validate them and clean bugs, on a pre-production server(s), environment, directory, database, etc.

1
  • 1
    Usually the detail and customization of your maintenance definition, use to go with the fails you have seen in your face. When I was a junior sysadmin I was happy with a RRD graph of disk space, until I had my first disk problem, or until I had my first inode issue, or until I had my first "too many files" issue, etc...
    – poisonbit
    Apr 16, 2011 at 19:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .