1

I even could not find where to start research ! There is no pattern between crashes .

2014-08-27T13:48:01.033Z cpu0:32957)MCE: 1118: cpu0: MCA error detected via CMCI (Gbl status=0x0): Restart IP: invalid, Error IP: invalid, MCE in progress: no.
2014-08-27T13:48:01.033Z cpu0:32957)MCE: 222: cpu0: bank0: status=0x9000004000010005: (VAL=1, OVFLW=0, UC=0, EN=1, PCC=0, S=0, AR=0), ECC=no, Addr:0x0 (invalid), Misc:0x0 (invalid)
2014-08-27T13:48:01.033Z cpu0:32957)MCE: 231: cpu0: bank0: MCA recoverable error (CE): "Internal Parity Error."
2014-08-27T13:48:01.033Z cpu1:33242)World: 8773: PRDA 0x418040400000 ss 0x0 ds 0x10b es 0x10b fs 0x0 gs 0x13b
2014-08-27T13:48:01.033Z cpu1:33242)World: 8775: TR 0x4020 GDT 0x4123876a1000 (0x402f) IDT 0x418014cf3000 (0xfff)
2014-08-27T13:48:01.033Z cpu1:33242)World: 8776: CR0 0x80010031 CR3 0x11e3e4000 CR4 0x42768
2014-08-27T13:48:01.039Z cpu1:33242)Backtrace for current CPU #1, worldID=33242, ebp=0x4119c0013dd0
2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013dd0:[0x418014c8cf99]PanicvPanicInt@vmkernel#nover+0x575 stack: 0x8, 0x4119c0013e40, 0x41
2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013e30:[0x418014c8d1dd]Panic_NoSave@vmkernel#nover+0x49 stack: 0x800000001, 0x5, 0xbe200000
2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013e90:[0x418014c63e75]IDTReturnPrepare@vmkernel#nover+0x2c5 stack: 0x67374e, 0x6738ce, 0xf
2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013f20:[0x418014c6475f]Int18_MachineCheck@vmkernel#nover+0x163 stack: 0xff8f3bb8, 0x1196300
2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013f30:[0x418014cf1064]gate_entry@vmkernel#nover+0x64 stack: 0x0, 0x13b, 0x0, 0x12a7495c, 0
2014-08-27T13:48:01.040Z cpu1:33242)[45m[33;1mVMware ESXi 5.5.0 [Releasebuild-1892794 x86_64][0m
NOT_REACHED bora/vmkernel/main/idt.c:1165
2014-08-27T13:48:01.040Z cpu1:33242)cr0=0x80010031 cr2=0x12a18be0 cr3=0x11e3e4000 cr4=0x42768
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:0 world:32949 name:"memMapKernel-0" (S)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:1 world:33242 name:"vobd" (U)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:2 world:32791 name:"CmdCompl-2" (S)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:3 world:33210 name:"vmsyslogd" (U)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:4 world:32783 name:"coalesceWorld-0" (S)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:5 world:53216 name:"vmm1:Analytics_VM" (V)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:6 world:33212 name:"vmsyslogd" (U)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:7 world:32784 name:"netCoalesce2World" (S)
2014-08-27T13:48:01.040Z cpu1:33242)@BlueScreen: NOT_REACHED bora/vmkernel/main/idt.c:1165
2014-08-27T13:48:01.040Z cpu1:33242)Code start: 0x418014c00000 VMK uptime: 2:02:57:31.656
2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013dd0:[0x418014c8cf99]PanicvPanicInt@vmkernel#nover+0x575 stack: 0x8
2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013e30:[0x418014c8d1dd]Panic_NoSave@vmkernel#nover+0x49 stack: 0x800000001
2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013e90:[0x418014c63e75]IDTReturnPrepare@vmkernel#nover+0x2c5 stack: 0x67374e
2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013f20:[0x418014c6475f]Int18_MachineCheck@vmkernel#nover+0x163 stack: 0xff8f3bb8
2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013f30:[0x418014cf1064]gate_entry@vmkernel#nover+0x64 stack: 0x0
2014-08-27T13:48:01.042Z cpu1:33242)base fs=0x0 gs=0x418040400000 Kgs=0x0
2014-08-27T13:48:01.042Z cpu1:33242)MC:PCPU0 B:0 S:0x9000004000010005 M:0x0 A:0x0 0
2014-08-27T13:48:01.043Z cpu1:33242)MC:PCPU1 B:8 S:0xbe2000000005110a M:0x9080000086 A:0x118be2600 5
MC:PCPU0: 1 hardware errors seen since boot (1 corrected by hardware)
MC:PCPU1: 1 hardware errors seen since boot (0 corrected by hardware)
2014-08-27T13:48:01.043Z cpu1:33242)PCPU fam:6 model:58 step:9 type:2 name:Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
2014-08-27T13:48:01.043Z cpu1:33242)vmkernel             0x0 .data 0x0 .bss 0x0
2014-08-27T13:48:01.043Z cpu1:33242)chardevs             0x418015171000 .data 0x417fc0000000 .bss 0x417fc0000400
2014-08-27T13:48:01.043Z cpu1:33242)user                 0x418015178000 .data 0x417fc0400000 .bss 0x417fc040e180
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_mgmt          0x41801522b000 .data 0x417fc0800000 .bss 0x417fc0800140
2014-08-27T13:48:01.043Z cpu1:33242)vprobe               0x418015231000 .data 0x417fc0c00000 .bss 0x417fc0c0b7c0
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_socket        0x41801526f000 .data 0x417fc1000000 .bss 0x417fc10005c0
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_vmkernel_shim 0x418015274000 .data 0x417fc1400000 .bss 0x417fc14080c0
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_vmkernel_shim 0x418015279000 .data 0x417fc1800000 .bss 0x417fc1808840
2014-08-27T13:48:01.043Z cpu1:33242)procfs               0x41801527e000 .data 0x417fc1c00000 .bss 0x417fc1c00240
2014-08-27T13:48:01.043Z cpu1:33242)vfat                 0x418015281000 .data 0x417fc2000000 .bss 0x417fc2002600
2014-08-27T13:48:01.043Z cpu1:33242)procMisc             0x41801528b000 .data 0x417fc2400000 .bss 0x417fc2400000
2014-08-27T13:48:01.043Z cpu1:33242)vmci                 0x41801528c000 .data 0x417fc2800000 .bss 0x417fc28057c0
2014-08-27T13:48:01.043Z cpu1:33242)iodm                 0x4180152af000 .data 0x417fc2c00000 .bss 0x417fc2c00138
2014-08-27T13:48:01.043Z cpu1:33242)vmkplexer            0x4180152b3000 .data 0x417fc3000000 .bss 0x417fc3000260
2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9           0x4180152b7000 .data 0x417fc3400000 .bss 0x417fc3408e80
2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9_2_0_0     0x41801533e000 .data 0x417fc3800000 .bss 0x417fc3807e84
2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9_2_1_0     0x418015341000 .data 0x417fc3c00000 .bss 0x417fc3c07f98
2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9_2_2_0     0x418015344000 .data 0x417fc4000000 .bss 0x417fc4008838
2014-08-27T13:48:01.043Z cpu1:33242)iscsi_trans          0x418015347000 .data 0x417fc4400000 .bss 0x417fc4401800
2014-08-27T13:48:01.043Z cpu1:33242)iscsi_trans_compat_shim 0x418015352000 .data 0x417fc4800000 .bss 0x417fc480096c
2014-08-27T13:48:01.043Z cpu1:33242)iscsi_trans_incompat_shim 0x418015353000 .data 0x417fc4c00000 .bss 0x417fc4c007e4
2014-08-27T13:48:01.043Z cpu1:33242)etherswitch          0x418015354000 .data 0x417fc5000000 .bss 0x417fc5013a00
2014-08-27T13:48:01.043Z cpu1:33242)netsched             0x418015389000 .data 0x417fc5400000 .bss 0x417fc5404800
2014-08-27T13:48:01.043Z cpu1:33242)cnic_register        0x41801539b000 .data 0x417fc5800000 .bss 0x417fc58001e0
2014-08-27T13:48:01.043Z cpu1:33242)e1000                0x41801539d000 .data 0x417fc5c00000 .bss 0x417fc5c01240
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_iscsi_shim 0x4180153c3000 .data 0x417fc6000000 .bss 0x417fc6000970
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_iscsi_shim 0x4180153c4000 .data 0x417fc6400000 .bss 0x417fc6400970
2014-08-27T13:48:01.043Z cpu1:33242)random               0x4180153c5000 .data 0x417fc6800000 .bss 0x417fc6800600
2014-08-27T13:48:01.043Z cpu1:33242)usb                  0x4180153c9000 .data 0x417fc6c00000 .bss 0x417fc6c01660
2014-08-27T13:48:01.043Z cpu1:33242)ehci-hcd             0x4180153eb000 .data 0x417fc7000000 .bss 0x417fc70002a0
2014-08-27T13:48:01.043Z cpu1:33242)hid                  0x4180153f6000 .data 0x417fc7400000 .bss 0x417fc74004e0
2014-08-27T13:48:01.043Z cpu1:33242)healthchk            0x4180153fb000 .data 0x417fc7800000 .bss 0x417fc7811e00
2014-08-27T13:48:01.043Z cpu1:33242)teamcheck            0x418015411000 .data 0x417fc7c00000 .bss 0x417fc7c12240
2014-08-27T13:48:01.043Z cpu1:33242)vlanmtucheck         0x418015424000 .data 0x417fc8000000 .bss 0x417fc8012000
2014-08-27T13:48:01.043Z cpu1:33242)heartbeat            0x418015439000 .data 0x417fc8400000 .bss 0x417fc8411f00
2014-08-27T13:48:01.043Z cpu1:33242)shaper               0x41801544a000 .data 0x417fc8800000 .bss 0x417fc8813e80
2014-08-27T13:48:01.043Z cpu1:33242)lldp                 0x41801545d000 .data 0x417fc8c00000 .bss 0x417fc8c00040
2014-08-27T13:48:01.043Z cpu1:33242)cdp                  0x418015462000 .data 0x417fc9000000 .bss 0x417fc9013400
2014-08-27T13:48:01.043Z cpu1:33242)ipfix                0x41801547e000 .data 0x417fc9400000 .bss 0x417fc9412540
2014-08-27T13:48:01.043Z cpu1:33242)tcpip4               0x418015492000 .data 0x417fc9800000 .bss 0x417fc9818180
2014-08-27T13:48:01.043Z cpu1:33242)dvsdev               0x418015598000 .data 0x417fc9c00000 .bss 0x417fc9c00030
2014-08-27T13:48:01.043Z cpu1:33242)dvfilter             0x41801559b000 .data 0x417fca000000 .bss 0x417fca000b00
2014-08-27T13:48:01.043Z cpu1:33242)lacp                 0x4180155bd000 .data 0x417fca400000 .bss 0x417fca400160
2014-08-27T13:48:01.043Z cpu1:33242)hbr_filter           0x4180155c7000 .data 0x417fca800000 .bss 0x417fca800300
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_dvfilter_shim 0x4180155f3000 .data 0x417fcac00000 .bss 0x417fcac009b0
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_dvfilter_shim 0x4180155f4000 .data 0x417fcb000000 .bss 0x417fcb000930
2014-08-27T13:48:01.043Z cpu1:33242)dvfilter-generic-fastpath 0x4180155f5000 .data 0x417fcb400000 .bss 0x417fcb412380
2014-08-27T13:48:01.043Z cpu1:33242)vmkstatelogger       0x41801560f000 .data 0x417fcb800000 .bss 0x417fcb803a00
2014-08-27T13:48:01.043Z cpu1:33242)esxfw                0x418015633000 .data 0x417fcbc00000 .bss 0x417fcbc12d00
2014-08-27T13:48:01.043Z cpu1:33242)dm                   0x418015648000 .data 0x417fcc000000 .bss 0x417fcc000000
2014-08-27T13:48:01.043Z cpu1:33242)nmp                  0x41801564a000 .data 0x417fcc400000 .bss 0x417fcc403e50
2014-08-27T13:48:01.043Z cpu1:33242)vmw_satp_local       0x41801566d000 .data 0x417fcc800000 .bss 0x417fcc800028
2014-08-27T13:48:01.043Z cpu1:33242)vmw_satp_default_aa  0x41801566f000 .data 0x417fccc00000 .bss 0x417fccc00000
2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_lib          0x418015670000 .data 0x417fcd000000 .bss 0x417fcd000290
2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_fixed        0x418015672000 .data 0x417fcd400000 .bss 0x417fcd400000
2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_rr           0x418015674000 .data 0x417fcd800000 .bss 0x417fcd800068
2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_mru          0x418015677000 .data 0x417fcdc00000 .bss 0x417fcdc00000
2014-08-27T13:48:01.043Z cpu1:33242)libata_92            0x418015679000 .data 0x417fce000000 .bss 0x417fce002660
2014-08-27T13:48:01.043Z cpu1:33242)libata_9_2_0_0       0x41801569b000 .data 0x417fce400000 .bss 0x417fce401750
2014-08-27T13:48:01.043Z cpu1:33242)libata_9_2_1_0       0x41801569c000 .data 0x417fce800000 .bss 0x417fce801750
2014-08-27T13:48:01.043Z cpu1:33242)usb-storage          0x41801569d000 .data 0x417fcec00000 .bss 0x417fcec04780
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_nmp_shim 0x4180156a9000 .data 0x417fcf000000 .bss 0x417fcf000ca8
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_nmp_shim 0x4180156aa000 .data 0x417fcf400000 .bss 0x417fcf400ca8
2014-08-27T13:48:01.043Z cpu1:33242)svmmirror            0x4180156ab000 .data 0x417fcf800000 .bss 0x417fcf8000c0
2014-08-27T13:48:01.043Z cpu1:33242)cbt                  0x4180156b7000 .data 0x417fcfc00000 .bss 0x417fcfc00080
2014-08-27T13:48:01.043Z cpu1:33242)migrate              0x4180156bb000 .data 0x417fd0000000 .bss 0x417fd0004d40
2014-08-27T13:48:01.043Z cpu1:33242)libfc_92             0x41801571a000 .data 0x417fd0400000 .bss 0x417fd0400b80
2014-08-27T13:48:01.043Z cpu1:33242)libfcoe_92           0x418015733000 .data 0x417fd0800000 .bss 0x417fd08001c0
2014-08-27T13:48:01.043Z cpu1:33242)libfc_9_2_0_0        0x418015739000 .data 0x417fd0c00000 .bss 0x417fd0c00868
2014-08-27T13:48:01.043Z cpu1:33242)libfcoe_9_2_0_0      0x41801573a000 .data 0x417fd1000000 .bss 0x417fd10001f4
2014-08-27T13:48:01.043Z cpu1:33242)libfc_9_2_1_0        0x41801573b000 .data 0x417fd1400000 .bss 0x417fd1400868
2014-08-27T13:48:01.043Z cpu1:33242)libfcoe_9_2_1_0      0x41801573c000 .data 0x417fd1800000 .bss 0x417fd18001f4
2014-08-27T13:48:01.043Z cpu1:33242)ahci                 0x41801573d000 .data 0x417fd1c00000 .bss 0x417fd1c00420
2014-08-27T13:48:01.043Z cpu1:33242)sunrpc               0x418015744000 .data 0x417fd2000000 .bss 0x417fd2002b80
2014-08-27T13:48:01.043Z cpu1:33242)nfsclient            0x418015753000 .data 0x417fd2400000 .bss 0x417fd2403940
2014-08-27T13:48:01.043Z cpu1:33242)vmkibft              0x41801576c000 .data 0x417fd2800000 .bss 0x417fd28037c0
2014-08-27T13:48:01.043Z cpu1:33242)lvmdriver            0x41801576f000 .data 0x417fd2c00000 .bss 0x417fd2c03380
2014-08-27T13:48:01.043Z cpu1:33242)deltadisk            0x418015783000 .data 0x417fd3000000 .bss 0x417fd3005c00
2014-08-27T13:48:01.043Z cpu1:33242)tracing              0x4180157ae000 .data 0x417fd3400000 .bss 0x417fd3405b40
2014-08-27T13:48:01.043Z cpu1:33242)rdt                  0x4180157b5000 .data 0x417fd3800000 .bss 0x417fd3804e00
2014-08-27T13:48:01.043Z cpu1:33242)vsanutil             0x4180157db000 .data 0x417fd3c00000 .bss 0x417fd3c069c0
2014-08-27T13:48:01.043Z cpu1:33242)lsomcommon           0x4180157fa000 .data 0x417fd4000000 .bss 0x417fd4001680
2014-08-27T13:48:01.043Z cpu1:33242)plog                 0x41801582e000 .data 0x417fd4400000 .bss 0x417fd44056c0
2014-08-27T13:48:01.043Z cpu1:33242)vmfs3                0x418015871000 .data 0x417fd4800000 .bss 0x417fd4803840
2014-08-27T13:48:01.043Z cpu1:33242)dvfg-igmp            0x4180158d9000 .data 0x417fd5a00000 .bss 0x417fd5a00208
2014-08-27T13:48:01.043Z cpu1:33242)cmmds_net            0x4180158df000 .data 0x417fd5e00000 .bss 0x417fd5e02f40
2014-08-27T13:48:01.043Z cpu1:33242)cmmds                0x4180158ec000 .data 0x417fd6200000 .bss 0x417fd6204d80
2014-08-27T13:48:01.043Z cpu1:33242)cmmds_resolver       0x418015921000 .data 0x417fd6600000 .bss 0x417fd6600140
2014-08-27T13:48:01.043Z cpu1:33242)vsan                 0x41801592d000 .data 0x417fd6a00000 .bss 0x417fd6a1c200
2014-08-27T13:48:01.043Z cpu1:33242)vmklink_mpi          0x418015a48000 .data 0x417fd6e00000 .bss 0x417fd6e02400
2014-08-27T13:48:01.043Z cpu1:33242)swapobj              0x418015a4d000 .data 0x417fd7200000 .bss 0x417fd7203010
2014-08-27T13:48:01.043Z cpu1:33242)osfs                 0x418015a55000 .data 0x417fd7600000 .bss 0x417fd7603380
2014-08-27T13:48:01.043Z cpu1:33242)vflash               0x418015a63000 .data 0x417fd7a00000 .bss 0x417fd7a03540
2014-08-27T13:48:01.043Z cpu1:33242)vfc                  0x418015a6e000 .data 0x417fd7e00000 .bss 0x417fd7e02ac0
Coredump to disk. 
2014-08-27T13:48:01.093Z cpu1:33242)Slot 1 of 1.
2014-08-27T13:48:01.093Z cpu1:33242)Dump: 2212: Using dump slot size 2684354560.
Improve this question

2 Answers 2

Reset to default
2

Error message looks like you have a hardware problem with CPU or RAM. Have you tried running memtest or if available, a built-in hardware diagnostic tool which you can choose during the server boot?

Improve this answer
2
  • Memtest shows no error . Is there any other bootable tool to try ?
    – Omid Kosari
    Aug 28, 2014 at 8:06
  • 1
    Proper servers usually have a built-in diagnostic tool selectable during server boot sequence. Have you checked out you don't have one? Those tend to test all the components. Of course they are not fool-proof but sometimes they succeed in finding the fault. Other than that, you might boot up any Linux live distro you like and let tools such as stress and cpuburn run for a longer period of time (at least overnight or a full weekend).
    – Janne Pikkarainen
    Aug 28, 2014 at 8:11
-1

This is definitely look like a hardware error.You can try tool like mcelog which just act as a translator for any machine check exception(mce).You haven't menrioned which version of OS you are using but in RedHat/Centos you can install it via yum

               yum install mcelog

and the message are logged in /var/log/mcelog

For more info you can refer to there main website http://www.mcelog.org/

Improve this answer
1
  • 2
    He speaks about ESXi right in the question title...
    – Sven
    Aug 28, 2014 at 8:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .