7

We're running an import of a database dump on an Oracle system - (RHEL 5.9, 2.6.18-348.6.1.el5). The import does not complete, eventually erroring-out with:

ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 248 logical extent 0 of file 280 in group 1 on disk 1 allocation unit 986
Errors in file /u01/app/oracle/diag/rdbms/dbprod/DBPROD/trace/DBPROD_lgwr_24520.trc:
ORA-00345: redo log write error block 509314 count 2023
ORA-00312: online log 1 thread 1: '+DATA/dbprod/redo01.log'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk

There are corresponding errors in the ring buffer and /var/log/messages:

Jun 12 18:54:42 db1-test kernel: megasas: build_ld_io  error, sge_count = 51
Jun 12 18:54:42 db1-test kernel: megasas: Err returned from build_and_issue_cmd
Jun 12 18:54:42 db1-test kernel: megasas: build_ld_io  error, sge_count = 51
Jun 12 18:54:42 db1-test kernel: megasas: Err returned from build_and_issue_cmd
Jun 12 18:54:42 db1-test kernel: megasas: build_ld_io  error, sge_count = 51
Jun 12 18:54:42 db1-test kernel: megasas: Err returned from build_and_issue_cmd
Jun 12 18:54:42 db1-test kernel: sd 0:2:1:0: timing out command, waited 360s
Jun 12 18:54:42 db1-test kernel: sd 0:2:1:0: Unhandled error code
Jun 12 18:54:42 db1-test kernel: sd 0:2:1:0: SCSI error: return code = 0x06000000
Jun 12 18:54:42 db1-test kernel: Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT,SUGGEST_OK

The drive array containing the import is a 10-disk SAS array in RAID 1+0 using 300GB 10k disks. The RAID controller is an LSI MegaRAID SAS 9260-8i. No disk or adapter errors are reported via MegaCLI.

  • Is this a hardware issue?
  • Is there any way to troubleshoot? The RAID controller status is fine. The disks and logical drives report healthy.
  • Is this a Linux OS or tuning issue? I'll try with different I/O schedulers to be sure. CFQ is default.

Edit:

Other schedulers have been tried with the same result. There is a third-party (Vormetric) filesystem encryption module running in this setup. Removing it allows the import to complete. So now I'm wondering if this is a deficiency in the module or if it is triggering a bad condition in the LSI driver.


During the import, we're hitting 14,000 write IOPS. enter image description here

In recent attempts, the system stalls entirely with the following on the console. enter image description here

Last top output before freeze. enter image description here

6
  • Just to be clear - the Raid controller says the drives are OK? What does LSI support say? Given that this is a VERY obvious "RAID / OS intergation" issue, they would be my first choice to ask. If the discs ARE broken, at least their tools should say so - that is the job of a raid controller.
    – TomTom
    Jun 13, 2013 at 18:37
  • This only seems to occur during the database import. Normal activity is unaffected.
    – ewwhite
    Jun 13, 2013 at 18:38
  • 2
    from the code in the megaraid driver, it seems to be overrunning some buffer while preparing a command to the logical drive. Try updating the controller's firmware, the driver and asking LSI support - in this order.
    – the-wabbit
    Jun 13, 2013 at 19:16
  • 2
    The error means that the driver got a request with more SG entries than supported by hardware; this may be caused by bugs in the block IO stack. A similar bug in RHEL5 involving dm was claimed to be fixed; there was also a possibly related bug in upstream Linux dm. Jun 13, 2013 at 19:16
  • @syneticon-dj Firmware was updated... I did get screenshots from watching logs/console in realtime, though. dmsetup was sitting at the top of the process resource utilization list.
    – ewwhite
    Jun 13, 2013 at 19:21

1 Answer 1

3

Ultimately Sergey is right - this is a driver problem. But let's check things out first:

First off you'll want to use the deadline I/O scheduler rather than CFQ. deadline, as its name implies, ensures that all IOPs complete in a timely manner.

Grab the events from the megaraid card:

megacli -adpeventlog -getevents -f /tmp/megaraid-$(date +%F_%T) -aALL

Check the SMART data on the disks (you will need to build a new smartmontools for this to work):

# megacli -pdlist -a0 |grep 'Device Id'
Device Id: 10
Device Id: 9

# smartctl -a /dev/sda -d megaraid,9
«…»
# smartctl -a /dev/sda -d megaraid,10
«…»

If everything looks OK, go ahead and try out the latest driver from LSI.


There is a third-party (Vormetric) filesystem encryption module running in this setup. Removing it allows the import to complete. So now I'm wondering if this is a deficiency in the module or if it is triggering a bad condition in the LSI driver.

The Voretric module is likely doing something incompatible, yes. I would start by talking with them about how their module is screwing up your system under high load.

2
  • I tried with deadline and noop. Both crashed or hung the system. I've left things at deadline. Firmware is current. Driver really shouldn't be changed due to other factors with the EL5.9 build.
    – ewwhite
    Jun 14, 2013 at 13:57
  • See edit about about the encryption filesystem layer being used.
    – ewwhite
    Jun 14, 2013 at 14:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .