0

as of last night we have had an IBM Server running 2003R2 BSOD a couple of times and I have been trawling the net for answers ever since. I've pinned down what triggers it and that is a BackupExec job which has ran without error for as long as I can remember every night up until last night. This is the main file server and as such has access to the IBM Tape Unit for its backups. No actual changes have been made to the File Sever - the only changes made have been some old Databases that were removed and actually live on a different server.

The reason I believe it is the BackupExec job is that I have had this happen twice and both times it occurred approx 2hrs 30mins into the backup job.

I have attached the minidump file to see if anyone can shed any light on what the actual cause might be. Until its resolved I'm not going to be able to backup the file system to tape. The minidump is as follows:

Microsoft (R) Windows Debugger Version 6.11.0001.404 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\WINDOWS\Minidump\Mini111909-02.dmp]
Mini Kernel Dump File: Only registers and stack trace are available

Symbol search path is: symsrv*symsrv.dll*C:\WINDOWS\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 

"nt" was not found in the image list.
Debugger will attempt to load "nt" at given base 00000000`00000000.

Please provide the full image name, including the extension (i.e. kernel32.dll)
for more reliable results.Base address and size overrides can be given as
.reload <image.ext>=<base>,<size>.
Unable to load image nt, Win32 error 0n2
Unable to add module at 00000000`00000000
Debugger can not determine kernel base address
Windows Server 2003 Kernel Version 3790 (Service Pack 2) MP (8 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Machine Name:
Kernel base = 0xfffff800`01000000 PsLoadedModuleList = 0xfffff800`011d4140
Debug session time: Thu Nov 19 13:16:06.265 2009 (GMT+8)
System Uptime: 0 days 10:45:23.816

"nt" was not found in the image list.
Debugger will attempt to load "nt" at given base 00000000`00000000.

Please provide the full image name, including the extension (i.e. kernel32.dll)
for more reliable results.Base address and size overrides can be given as
.reload <image.ext>=<base>,<size>.
Unable to load image nt, Win32 error 0n2
Unable to add module at 00000000`00000000
Debugger can not determine kernel base address
Loading Kernel Symbols

Loading User Symbols
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck E0E7E, {ffffffffc0000005, fffffade93956726, fffffade95c6c450, fffffade95c6be60}

***** Debugger could not find nt in module list, module list might be corrupt, error 0x80070057.

Probably caused by : Unknown_Image ( ANALYSIS_INCONCLUSIVE )

Followup: MachineOwner
---------

0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Unknown bugcheck code (e0e7e)
Unknown bugcheck description
Arguments:
Arg1: ffffffffc0000005
Arg2: fffffade93956726
Arg3: fffffade95c6c450
Arg4: fffffade95c6be60

Debugging Details:
------------------

***** Debugger could not find nt in module list, module list might be corrupt, error 0x80070057.


CUSTOMER_CRASH_COUNT:  2

DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP

BUGCHECK_STR:  0xE0E7E

CURRENT_IRQL:  0

LAST_CONTROL_TRANSFER:  from fffffade9bf68759 to fffffade9bf68b42

STACK_TEXT:  
fffff800`00074d68 fffffade`9bf68759 : fffff800`011b4940 fffffade`d3b54040 00000000`00000010 fffffade`d40fce60 : 0xfffffade`9bf68b42
fffff800`00074d70 fffff800`011b4940 : fffffade`d3b54040 00000000`00000010 fffffade`d40fce60 fffff800`011b2450 : 0xfffffade`9bf68759
fffff800`00074d78 fffffade`d3b54040 : 00000000`00000010 fffffade`d40fce60 fffff800`011b2450 fffff800`0103190c : 0xfffff800`011b4940
fffff800`00074d80 00000000`00000010 : fffffade`d40fce60 fffff800`011b2450 fffff800`0103190c fffff800`0006e000 : 0xfffffade`d3b54040
fffff800`00074d88 fffffade`d40fce60 : fffff800`011b2450 fffff800`0103190c fffff800`0006e000 fffff800`011b4901 : 0x10
fffff800`00074d90 fffff800`011b2450 : fffff800`0103190c fffff800`0006e000 fffff800`011b4901 fffff800`011b2450 : 0xfffffade`d40fce60
fffff800`00074d98 fffff800`0103190c : fffff800`0006e000 fffff800`011b4901 fffff800`011b2450 00000000`0005ffd4 : 0xfffff800`011b2450
fffff800`00074da0 fffff800`0006e000 : fffff800`011b4901 fffff800`011b2450 00000000`0005ffd4 00000000`0005ffd4 : 0xfffff800`0103190c
fffff800`00074da8 fffff800`011b4901 : fffff800`011b2450 00000000`0005ffd4 00000000`0005ffd4 fffff800`01067b5e : 0xfffff800`0006e000
fffff800`00074db0 fffff800`011b2450 : 00000000`0005ffd4 00000000`0005ffd4 fffff800`01067b5e fffff800`011b0180 : 0xfffff800`011b4901
fffff800`00074db8 00000000`0005ffd4 : 00000000`0005ffd4 fffff800`01067b5e fffff800`011b0180 fffff800`011b0180 : 0xfffff800`011b2450
fffff800`00074dc0 00000000`0005ffd4 : fffff800`01067b5e fffff800`011b0180 fffff800`011b0180 fffff800`011b4500 : 0x5ffd4
fffff800`00074dc8 fffff800`01067b5e : fffff800`011b0180 fffff800`011b0180 fffff800`011b4500 fffffade`d3b54040 : 0x5ffd4
fffff800`00074dd0 fffff800`011b0180 : fffff800`011b0180 fffff800`011b4500 fffffade`d3b54040 00000000`00000006 : 0xfffff800`01067b5e
fffff800`00074dd8 fffff800`011b0180 : fffff800`011b4500 fffffade`d3b54040 00000000`00000006 fffff800`014141d1 : 0xfffff800`011b0180
fffff800`00074de0 fffff800`011b4500 : fffffade`d3b54040 00000000`00000006 fffff800`014141d1 00000000`00000000 : 0xfffff800`011b0180
fffff800`00074de8 fffffade`d3b54040 : 00000000`00000006 fffff800`014141d1 00000000`00000000 00000000`00000000 : 0xfffff800`011b4500
fffff800`00074df0 00000000`00000006 : fffff800`014141d1 00000000`00000000 00000000`00000000 00000000`00000000 : 0xfffffade`d3b54040
fffff800`00074df8 fffff800`014141d1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x6
fffff800`00074e00 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0xfffff800`014141d1


STACK_COMMAND:  kb

SYMBOL_NAME:  ANALYSIS_INCONCLUSIVE

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: Unknown_Module

IMAGE_NAME:  Unknown_Image

DEBUG_FLR_IMAGE_TIMESTAMP:  0

BUCKET_ID:  CORRUPT_MODULELIST

Followup: MachineOwner
---------

0: kd> lmvm Unknown_Module
start             end                 module name

Any thoughts would be greatly appreciated!

1
  • Interestingly there has been a new development in this saga. A file system backup running on a remote agent (completely different server) has managed to reboot suddenly. This server didn't BSOD instead just rebooted out of nowhere. This other server just has a RAID 5 array where as the file system backup was taking place on an iSCSI connection to the SAN. There was no crash dump to report on for that reboot, and not really anything in the error log that gives any clues. But its easy to see that BackupExc is causing the issue.
    – pigeon
    Nov 20, 2009 at 6:19

4 Answers 4

0

thats funny. we got the exact problem since yesterday morning. today the second bluescreen. i think i can help to find the solution.

we are using two filecluster nodes. the first one got bluescreen during the backup yesterday morning at 5.45 am. the second node got the filecluster ressources while first one rebooted.

today morning at 3.45 am the second one rebooted caused by bluescreen.

were using backup exec 11d at sp3. nothing was changed, only dfsr config on monday. no patches by microsoft since oct09. no change on the backup server for a long long time. both servers running 2003 x64 ent r2 with sp2 in en language (although working in germany).

i can find no log information, only the bug code E0E7E.

the fact that two servers got the same problem, if they serve the file ressource during backup, there must be a problem primary with the backup.

we also use trendmicro officescan. no other cluster/server do have problems at all, only the filecluster.

im checking the antivirus now. i cannot believe that only a problem with the backup exists. there must be an action during the backup with a second software/process.

2
  • Thanks for the tip off you were spot on Mathias - I owe you a beer next time your in Australia. Seems like Trend has pushed out a new Scan Engine (ours updated the servers on Wendesday) which was the first day of the error. It seems to be only related with 2003 x64 R2 (at this stage - might be an issue with x64 Windows machines in general). Ran a backup job with the officescan client unloaded on the file server and sure enough it worked without issue. Now I just need to roll those clients back to an earlier version and prevent updates until Trend come out with a fix.
    – pigeon
    Nov 21, 2009 at 7:00
  • Hi Pigeon, i will take the offer, if i got once the possibility to come to australia. i solved the problem with the new enine. upgraded to officescan 10 sp1 and also installed all patches to veritas 11d (sp4 and hotfixes). take a look, maybe you can patch your 12.5, too. give me a hint if you got tickets to australia... :-)
    – Mathias
    Nov 21, 2009 at 20:13
0

What version of Backup Ecexc are you running. Are there any agents on remote servers? Are all Servcie packs and hot fixes installed?

1
  • This is BackupExec 12.5, there are approx 10 servers running the 12.5 remote agents. So its running the latest 12.5 as far as Backupexec is concerned The strange thing is that when backing up Exchange and the SQL server the jobs run without issue. The file server was patched mid October so there may be some security fixes not applied but its running SP2 for Windows 2003R2.
    – pigeon
    Nov 20, 2009 at 6:22
0

I had the exact same issue a few months back, tried to replace SCSI-cables, terminator and scsi card (one at the time). Nothing worked. Replaced all of them together, everything went smooth.

It was probably a short-circiut in the scsi card wich caused the active terminator (and the scsi card itself) to burn and die. Maybe you should try this aswell?

1
  • Thanks for the suggestion. I ran a repair job on BackupExec last night but we're still having the same issue. Might be starting to look hardware related.
    – pigeon
    Nov 20, 2009 at 6:18
0

i checked the local officescan client and see that a rollback folder was created at 11/18 at 700 am (GMT+1). what seems to be an update with the firewall/engine driver.

i enable trend debugging and we´ll see...

what do you use? mathias

1
  • Hi Mathias - we indeed do use Trend Office scan and it wouldn't be the first time that it was responsible for taking servers down. I think your onto something. Thats the exact bug code we have as you may have seen - little to no other information than that. I could indeed disable the officescan client for a backup run - which I might do today (Saturday here). In an attempt to just run an ntbackup on that secondary server that I mentioned rebooted during a backupexec run, the machine rebooted itself again. And this was a local ntbackup job backing up to a location on the same server.
    – pigeon
    Nov 20, 2009 at 22:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .