Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We've got DB2 LUW running on a RHEL box. We had a crash of DB2 and IBM came back and said that a file that DB2 was trying to access (through open64()) unmounted or became invalid.

We have done nothing but restart the database and things seem to be running fine. Also, the file in question looks perfectly normal now:

$ cd /db/log/TEAMS/tmsinst/NODE0000/TEAMS/T0000000/
$ ls -l
total 557604
-rw------- 1 tmsinst tmsinst 570425344 Jan 14 10:24 C0000000.CAT
$ file C0000000.CAT 
C0000000.CAT: data
$ lsattr C0000000.CAT 
------------- C0000000.CAT
$ ls -l
total 557604
-rw------- 1 tmsinst tmsinst 570425344 Jan 14 10:24 C0000000.CAT

With those facts in hand (please correct me if I am mis-interpreting the data at hand) what could cause a file system to 'spontaneously unmount or become invalid for a short time'?

What should my next step be?

This is on Dell hardware and we ran their diagnostic tools against the hardware and it came back clean.

share|improve this question
    
Is the mounted partition a local disk or some kind of network mount? –  Scott Pack Jan 14 '11 at 18:38
    
They are local. –  Ichorus Jan 14 '11 at 21:09
add comment

1 Answer

up vote 1 down vote accepted

My guess would be underlying hardware issue, for example a drive disconnecting and reconnecting to the bus. Examing /var/log/messages (and run dmesg) and look for unusual scsi or sata messages about disconnects, etc.

share|improve this answer
    
That is also the first thing we thought...no relevant messages in dmesg or the log messages. Also, no manual umounts in history. –  Ichorus Jan 14 '11 at 18:24
    
Because of this answer I took a second look at dmesg: There were a lot of similar messages about oom killer getting invoked (on a box with 96 GB RAM no less) so I missed this: –  Ichorus Jan 14 '11 at 18:38
    
dsm_sa_datamgr3 invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 –  Ichorus Jan 14 '11 at 18:39
    
all Trace: [<ffffffff800c6301>] out_of_memory+0x8e/0x2f3 [<ffffffff8000f487>] __alloc_pages+0x245/0x2ce [<ffffffff80012e03>] __do_page_cache_readahead+0x96/0x179 [<ffffffff80013765>] filemap_nopage+0x14c/0x360 [<ffffffff8000898c>] __handle_mm_fault+0x1fa/0xf99 [<ffffffff800c591a>] generic_file_read+0xac/0xc5 [<ffffffff80066b25>] do_page_fault+0x4cb/0x830 [<ffffffff80063ad5>] mutex_lock+0xd/0x1d [<ffffffff880a7509>] :sd_mod:scsi_disk_put+0x2e/0x3f [<ffffffff8002ac33>] iput+0x4b/0x84 [<ffffffff8000d3a4>] dput+0x2c/0x114 [<ffffffff8005dde9>] error_exit+0x0/0x84 –  Ichorus Jan 14 '11 at 18:40
    
I wonder if oom killer just happened to kill the file read process and the failure of that read is what showed up in the db2diag log? –  Ichorus Jan 14 '11 at 18:41
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.