0

Since 2-3 weeks now, my main server is hanging for no apparent reasons. Was working without issues for more than 4 months in a row before that. Everytime, a simple reboot fix the problem.

Current setup :

  • HP DL380 G5, 2 x Xeon 4C 3GHz, 16GB memory, 6 x 146GB in RAID 0+1
  • Slackware 14.0

I leave the server with PuTTy openned and top running, when it hang (about 1 to 3 times a day), I see a high load, about more than 60, all web services (HTTP, DNS, SMTP, IMAP, POP3, etc) are unresponsive. When connecting with PuTTy, I'm able to log, but the prompt never appears, same thing on local prompt (keyboard + screen). Also, I've seen that green LEDs on drives are flashing simultaneously at a frequency of about 0.5Hz - 1Hz (normally they flash way faster and in random order).

I first suspected DDoS attacks, etc, added many fail2ban validations, external firewall TCP requests limitations, etc. After, I verified firmwares versions (including P400), upgraded all to latest versions, the problem still occurs. I've also synched root to another DL380 G5 (same hardware except 4 x 450GB drives) to replace the server, same issue again.

I verified using top, iostat, iotop still no clue. When load is high, there is almost no CPU usage (top) and no disk activity (iostat).

Now I'm wondering if it could be the CCISS driver that could have an issue in the version I'm using?

Here are some informations that may be useful :

Controler details :

root@hyperion:~# hpapucli

=> ctrl all show status

Smart Array P400 in Slot 1
Controller Status: OK
Cache Status: OK
Battery/Capacitor Status: OK

=> ctrl all show detail

Smart Array P400 in Slot 1
Bus Interface: PCI
Slot: 1
Serial Number: P61620G9SVM38V
Cache Serial Number: PA2270H9SVI198
RAID 6 (ADG) Status: Enabled
Controller Status: OK
Hardware Revision: D
Firmware Version: 6.86
Rebuild Priority: Medium
Expand Priority: Medium
Surface Scan Delay: 15 secs
Surface Scan Mode: Idle
Wait for Cache Room: Disabled
Surface Analysis Inconsistency Notification: Disabled
Post Prompt Timeout: 0 secs
Cache Board Present: True
Cache Status: OK
Cache Ratio: 25% Read / 75% Write
Drive Write Cache: Disabled
Total Cache Size: 512 MB
Total Cache Memory Available: 464 MB
No-Battery Write Cache: Disabled
Cache Backup Power Source: Batteries
Battery/Capacitor Count: 1
Battery/Capacitor Status: OK
SATA NCQ Supported: True

=> ctrl all show config

Smart Array P400 in Slot 1 (sn: P61620G9SVM38V)

array A (SAS, Unused Space: 0 MB)


logicaldrive 1 (838.3 GB, RAID 1+0, OK)

physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SAS, 450 GB, OK)
physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SAS, 450 GB, OK)
physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SAS, 450 GB, OK)
physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SAS, 450 GB, OK)

Driver details :

root@hyperion:~# modinfo cciss
filename: /lib/modules/3.2.29/kernel/drivers/block/cciss.ko
license: GPL
version: 3.6.26
description: Driver for HP Smart Array Controllers
author: Hewlett-Packard Company
srcversion: D553A90CDE37829B37A9C27
alias: pci:v0000103Cd00003230sv0000103Csd0000323Dbc*sc*i*
alias: pci:v0000103Cd00003230sv0000103Csd00003237bc*sc*i*
alias: pci:v0000103Cd00003238sv0000103Csd00003215bc*sc*i*
alias: pci:v0000103Cd00003238sv0000103Csd00003214bc*sc*i*
alias: pci:v0000103Cd00003238sv0000103Csd00003213bc*sc*i*
alias: pci:v0000103Cd00003238sv0000103Csd00003212bc*sc*i*
alias: pci:v0000103Cd00003238sv0000103Csd00003211bc*sc*i*
alias: pci:v0000103Cd00003230sv0000103Csd00003235bc*sc*i*
alias: pci:v0000103Cd00003230sv0000103Csd00003234bc*sc*i*
alias: pci:v0000103Cd00003230sv0000103Csd00003223bc*sc*i*
alias: pci:v0000103Cd00003220sv0000103Csd00003225bc*sc*i*
alias: pci:v00000E11d00000046sv00000E11sd0000409Dbc*sc*i*
alias: pci:v00000E11d00000046sv00000E11sd0000409Cbc*sc*i*
alias: pci:v00000E11d00000046sv00000E11sd0000409Bbc*sc*i*
alias: pci:v00000E11d00000046sv00000E11sd0000409Abc*sc*i*
alias: pci:v00000E11d00000046sv00000E11sd00004091bc*sc*i*
alias: pci:v00000E11d0000B178sv00000E11sd00004083bc*sc*i*
alias: pci:v00000E11d0000B178sv00000E11sd00004082bc*sc*i*
alias: pci:v00000E11d0000B178sv00000E11sd00004080bc*sc*i*
alias: pci:v00000E11d0000B060sv00000E11sd00004070bc*sc*i*
depends:
intree: Y
vermagic: 3.2.29 SMP mod_unload
parm: cciss_tape_cmds:number of commands to allocate for tape devices (default: 6) (int)
parm: cciss_simple_mode:Use 'simple mode' rather than 'performant mode' (int)

top output when hanging

top - 10:39:45 up 43 min,  2 users,  load average: 24.58, 7.14, 2.88
Tasks: 282 total,   1 running, 281 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32894436k total, 17964512k used, 14929924k free,    97732k buffers
Swap:        0k total,        0k used,        0k free, 10694424k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3928 root      20   0 37164 2988 2444 S    0  0.0   0:00.41 sshd
 4478 root      20   0 17608 1540 1060 R    0  0.0   0:07.62 top
    1 root      20   0  4316  696  600 S    0  0.0   0:00.98 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd
    3 root      20   0     0    0    0 S    0  0.0   0:00.01 ksoftirqd/0
    5 root      20   0     0    0    0 S    0  0.0   0:00.02 kworker/u:0
    6 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0
    7 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/1
    9 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/1
   11 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/2
   13 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/2
   14 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/3
   16 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/3
   17 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/4
   19 root      20   0     0    0    0 S    0  0.0   0:00.01 ksoftirqd/4
   20 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/5
   22 root      20   0     0    0    0 S    0  0.0   0:00.01 ksoftirqd/5
   23 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/6
   25 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/6
   26 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/7
   28 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/7
   29 root       0 -20     0    0    0 S    0  0.0   0:00.00 cpuset
   30 root       0 -20     0    0    0 S    0  0.0   0:00.00 khelper
   31 root      20   0     0    0    0 S    0  0.0   0:00.00 kdevtmpfs
   32 root       0 -20     0    0    0 S    0  0.0   0:00.00 netns
   33 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/u:1
  495 root      20   0     0    0    0 D    0  0.0   0:05.24 sync_supers
  497 root      20   0     0    0    0 S    0  0.0   0:00.00 bdi-default
  499 root       0 -20     0    0    0 S    0  0.0   0:00.00 kblockd
  654 root       0 -20     0    0    0 S    0  0.0   0:00.00 ata_sff
  661 root      20   0     0    0    0 S    0  0.0   0:00.00 khubd
  667 root       0 -20     0    0    0 S    0  0.0   0:00.00 md
  676 root      20   0     0    0    0 S    0  0.0   0:00.40 kworker/3:1
  677 root      20   0     0    0    0 S    0  0.0   0:00.12 kworker/4:1
  678 root      20   0     0    0    0 S    0  0.0   0:00.65 kworker/5:1
  679 root      20   0     0    0    0 S    0  0.0   0:00.16 kworker/6:1
  680 root      20   0     0    0    0 S    0  0.0   0:00.21 kworker/7:1
  774 root       0 -20     0    0    0 S    0  0.0   0:00.00 rpciod
  826 root      20   0     0    0    0 S    0  0.0   0:00.00 khungtaskd
  832 root      20   0     0    0    0 S    0  0.0   0:00.00 kswapd0

DL380 G6 with P410i migration

I also tried with another HP server by moving hard drives directly and changing /dev/cciss/c0d0* by /dev/sda* in /etc/fstab and /etc/lilo.conf, still the same issue.

Controler details :

Note : Yes the cache is disabled, I simply have no battery for that server now.

root@hyperion:~# modprobe sg
root@hyperion:~# hpacucli ctrl all show detail

Smart Array P410i in Slot 0 (Embedded)
   Bus Interface: PCI
   Slot: 0
   Serial Number: 50123456789ABCDE
   Cache Serial Number: PAAVP9VYBAU0
   RAID 6 (ADG) Status: Disabled
   Controller Status: OK
   Hardware Revision: C
   Firmware Version: 6.64
   Rebuild Priority: Medium
   Expand Priority: Medium
   Surface Scan Delay: 15 secs
   Surface Scan Mode: Idle
   Queue Depth: Automatic
   Monitor and Performance Delay: 60  min
   Elevator Sort: Enabled
   Degraded Performance Optimization: Disabled
   Inconsistency Repair Policy: Disabled
   Wait for Cache Room: Disabled
   Surface Analysis Inconsistency Notification: Disabled
   Post Prompt Timeout: 0 secs
   Cache Board Present: True
   Cache Status: OK
   Cache Ratio: 100% Read / 0% Write
   Drive Write Cache: Disabled
   Total Cache Size: 512 MB
   Total Cache Memory Available: 400 MB
   No-Battery Write Cache: Disabled
   Battery/Capacitor Count: 0
   SATA NCQ Supported: True

Driver details :

root@hyperion:~# modinfo hpsa
filename:       /lib/modules/3.2.29/kernel/drivers/scsi/hpsa.ko
license:        GPL
version:        2.0.2-1
description:    Driver for HP Smart Array Controller version 2.0.2-1
author:         Hewlett-Packard Company
srcversion:     624DA19A5286F6BDA1645F3
alias:          pci:v0000103Cd*sv*sd*bc01sc04i*
alias:          pci:v0000103Cd0000323Bsv0000103Csd00003356bc*sc*i*
alias:          pci:v0000103Cd0000323Bsv0000103Csd00003355bc*sc*i*
alias:          pci:v0000103Cd0000323Bsv0000103Csd00003354bc*sc*i*
alias:          pci:v0000103Cd0000323Bsv0000103Csd00003353bc*sc*i*
alias:          pci:v0000103Cd0000323Bsv0000103Csd00003352bc*sc*i*
alias:          pci:v0000103Cd0000323Bsv0000103Csd00003351bc*sc*i*
alias:          pci:v0000103Cd0000323Bsv0000103Csd00003350bc*sc*i*
alias:          pci:v0000103Cd0000323Asv0000103Csd00003233bc*sc*i*
alias:          pci:v0000103Cd0000323Asv0000103Csd0000324Bbc*sc*i*
alias:          pci:v0000103Cd0000323Asv0000103Csd0000324Abc*sc*i*
alias:          pci:v0000103Cd0000323Asv0000103Csd00003249bc*sc*i*
alias:          pci:v0000103Cd0000323Asv0000103Csd00003247bc*sc*i*
alias:          pci:v0000103Cd0000323Asv0000103Csd00003245bc*sc*i*
alias:          pci:v0000103Cd0000323Asv0000103Csd00003243bc*sc*i*
alias:          pci:v0000103Cd0000323Asv0000103Csd00003241bc*sc*i*
depends:
intree:         Y
vermagic:       3.2.29 SMP mod_unload
parm:           hpsa_allow_any:Allow hpsa driver to access unknown HP Smart Array hardware (int)
parm:           hpsa_simple_mode:Use 'simple mode' rather than 'performant mode' (int)

Possible cause

Yesterday by doing tests on different processes, I disabled postfix and the server stopped hanging. As soon as I started it again, the server hanged. Looks like there is a bad configuration or suspicious smtp requests are made.

2
  • do you have much logging turned on? Is write caching enabled on your RAID controller - I can't remember if the 400s support it or not, if not then you can fairly easily overload your disks with logging lots of small writes to different files when it's in write-through mode.
    – Adrien
    Jul 26 '16 at 3:15
  • There is not much logging, I mean normal... apache, postfix, etc. The cache is activated, 75% write of the 512MB cache. The server was working fine for periods up to 4 months, and since 2 days it hanged 15 times per day. Jul 26 '16 at 13:31
1

The HP ProLiant G5 server series is pretty old equipment and out of support from every reasonable perspective. This equipment went end-of-life in 2009.

However, if you don't mind being unsupported and the fact that the system is 4 generations and old, the server can still function.

For your situation, you're working with a bad revision of firmware on the RAID controller. I recommend you update the firmware of your RAID controller to the most current release (2012).

Normally, you could do this from within the operating system, but Slackware is totally unsupported by HP as well. If you can find a way to update firmware, this will likely resolve the issue.


enter image description here

2
  • I was thinking upgrading to latest version with the HP utility CD but I'll try to upgrade to this version tonight. Jul 26 '16 at 15:04
  • I've been able to upgrade P400 to 7.22 (not last version but previous) because I can't get Smart Start more recent than 10.10 (without valid warranty or support contract), and the CP0XXXXXX.scexe is running 32 bits only (havn't got time yet to try with a live CD etc...). Same issue... Also tried with a DL380 G6 including P410i upgraded with Smart Start 10.10... same issue. Jul 26 '16 at 22:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.