Monitor RAID status through terminal (CentOS 5)

Question

I am currently running CentOS 5 and looking for a terminal command that can allow me to monitor the status of the RAID set up (ie, if a drive is down) without having to get into the kernel. This is a live web server afterall.

update: specs are its a dell sc1435 with SAS 5i/R controller.

Answer 1

If you're using software RAID with a regular disk controller then use:

mdadm --detail <dev>

where is /dev/md0 for example. This will show you the current status. If a drive fails, you'll also see lots of nastiness in /var/log/messages.

Answer 2

it's raid dependent. for lsi [ it's in plenty of dell and hp servers ] you use tool called MegaCLI.

for 3ware cards - tw_cli

it usually comes with 'drivers' or documentation for your hardware.

Answer 3

If this is a software raid (mdadm) and you want to look at the current status you could simply do a cat /proc/mdstat. If you want to have something where the screen is refreshed every 10 seconds you could do a watch -n 10 cat /proc/mdstat.

Answer 4

Dell probably provide a tool to monitor it, but I can hazard a guess that it will be bloated and implemented with Java, like most lacklustre OEM utilities.

Fortunately enough it appears that the SC1435 is supported by the wonderful mpt-status utility. Just make sure that you have the following options enabled in your kernel:

CONFIG_FUSION=y
CONFIG_FUSION_SAS=y
CONFIG_FUSION_MAX_SGE=128
CONFIG_FUSION_CTL=y

You can then use mpt-status from the CLI to view the health of your RAID array.

I personally then use a simple Python script called from cron which periodically checks the status and issues us alerts by email. In a similar fashion to how mdadm behaves. You can of course specify how frequently you wish to check. Feel free to use it yourself:

#!/usr/bin/env python

# Copyright (c) 2009 Dan Carley <dan.carley@gmail.com>
#
# Permission to use, copy, modify, and distribute this software for any
# purpose with or without fee is hereby granted, provided that the above
# copyright notice and this permission notice appear in all copies.
#
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

"""
Report failures from hardware RAID controllers.

Requires the supporting utilities:
    mpt-status(8)   for MPT controllers.
    tw_cli(8)       for 3ware controllers.

Intended to be scheduled from crontab as follows:
    MAILTO="alert@your.domain"
    0   */3 *   *   *   /usr/local/sbin/hwraid_monitor.py options
"""

from re import search
from sys import exit
from os.path import isfile
from optparse import OptionParser
from subprocess import Popen, PIPE

def check_controller(type):
    ret = True

    if type == 'mpt':
        cmd = [ '/usr/sbin/mpt-status', '-s' ]
        array = {'regex': '^log_id$',
                 'pos': 2,
                 'string': 'OPTIMAL'}
        drive = {'regex': '^phys_id$',
                 'pos': 2,
                 'string': 'ONLINE'}
    elif type == 'tw':
        cmd = [ '/sbin/tw_cli', 'info' ]
        contr = {'regex': '^c\d+$'}
        array = {'regex': '^u\d+$',
                 'pos': 2,
                 'string': 'OK'}
        drive = {'regex': '^p\d+$',
                 'pos': 1,
                 'string': 'OK'}

    if not isfile(cmd[0]):
        print "%s: Utility not found" % cmd[0]
        return False

    if type == 'tw':
        controllers = []
        p = Popen(cmd, stdout=PIPE)
        o, e = p.communicate()
        if e:
            print e
        for c in o.split('\n'):
            c = c.split()
            if len(c) > 2 and search(contr['regex'], c[0]):
                controllers.append(c[0])
    elif type == 'mpt':
        controllers = ['']

    for c in controllers:
        p = Popen(cmd + [c], stdout=PIPE)
        o, e = p.communicate()
        if e:
            print e.split('\n')
        for v in o.split('\n'):
            v = v.split()
            if len(v) > 2:
                # Array check.
                if search(array['regex'], v[0]) and v[array['pos']] != array['string']:
                    print "Array failure: \n\t%s" % '\t'.join(v)
                    ret = False
                # Drive check.
                if search(drive['regex'], v[0]) and v[drive['pos']] != drive['string']:
                    print "Drive failure: \n\t%s" % '\t'.join(v)
                    ret = False

    return ret

def main():
    usage = "usage: %prog options"
    parser = OptionParser(usage=usage)
    parser.add_option("--mpt", action="store_true", default=False,
                      dest="mpt", help="MPT controller support.")
    parser.add_option("--tw", action="store_true", default=False,
                      dest="tw", help="3ware controller support.")
    (options, args) = parser.parse_args()

    if not options.mpt and not options.tw:
        parser.print_help()
        exit(2)

    fail = False

    if options.mpt:
        if not check_controller('mpt'):
            fail = True

    if options.tw:
        if not check_controller('tw'):
            fail = True

    if fail:
        exit(1)

if __name__ == "__main__":
    main()

Answer 5

mdadm offers detailed information about every aspect of the Linux software raid.

Answer 6

watch -n 10 cat /proc/mdstat this will setup on u r system