Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am running OEL 6.4 (a RHEL clone), and rsyncing large files over ssh nightly. During the rsync, I regularly (most nights) get a kernel panic:

Message from syslogd@cheshire at Mar 24 00:39:01 ...
 kernel:Oops: 0000 [#1] SMP

Message from syslogd@cheshire at Mar 24 00:39:01 ...
 kernel:Stack:

Message from syslogd@cheshire at Mar 24 00:39:01 ...
 kernel:Call Trace:

Message from syslogd@cheshire at Mar 24 00:39:01 ...
 kernel:Code: 00 55 48 89 e5 48 83 ec 10 48 89 1c 24 4c 89 64 24 08 66 66 66 66 90 41 89 d4 48 89 f3 e8 cf 23 fe ff 41 83 fc 01 48 89 c2 75 1a

Message from syslogd@cheshire at Mar 24 00:39:01 ...
 kernel:CR2: 0000000000000000

Message from syslogd@cheshire at Mar 24 00:39:01 ...
 kernel:Kernel panic - not syncing: Fatal exception

Is there any information in the above that might give me a clue to the reason or is it all generic? There appears to be nothing unusual in /var/log/messages at the point of the crash.


edit:

I should have mentioned I'm using ocfs2 (local not clustered). The files being transferred are the backing files for VMs and they are not in use at the time of the transfer: they are 'reflink' copies taken purely for the purpose of the rsync. The OS is up to date with patches.

share|improve this question
    
I've seen a similar thing on servers where there's been a bug in the file system driver. What file system are you using? Are you using the latest version of your kernel/drivers? –  Jenny D Mar 25 '13 at 8:12
    
great question, sorry I didn't mention it I'll edit the Q –  Jack Douglas Mar 25 '13 at 9:36
    
@JennyD, I switched to btrfs and had the same issue. Then I switched to LVM snapshots and the crashes have gone away. Still no idea why unfortunately. –  Jack Douglas May 6 '13 at 8:19
add comment

1 Answer

A kernel panic may be induced by several factors, for instance

  • ssh daemon
  • conflict between files being updated / another service using them
  • NIC interface ...

To help identify the source of the problem I'd try a few rsync --dry-run (which doesn't copy anything).

Besides, I read a while ago that there could be some problem with FS mounted with noatime option, relatime being better.

Also I'd try rsync with the --delay-updates option, that minimizes the actual files update timespan.

This is what comes to mind right now, I'll update the answer if something else rings the bell..

share|improve this answer
    
Thanks for the suggestions I'm using OCFS2 (I should have mentioned that earlier, sorry) and atime (not noatime or relatime) –  Jack Douglas Mar 25 '13 at 9:47
    
I'm not quite sure what --delay-updates does but it isn't compatible with --inplace which I think I need to avoid doubling up storage requirements at the target end (some of the files being transferred are in the 100s of Gb) –  Jack Douglas Mar 26 '13 at 12:16
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.