1

I have an Dell R510 running esxi 5.1 with 16GB Ram; 1 CPU (Xeon L5520 @ 2.27HGz; 4 core) running a single Server 2012 Machine. I came in this morning to this core dump

core dump. I held down the power button and rebooted. It came back to a "VMWARE Recovery" screen and after 10-20 minutes, I came back and it was back to its usual screen (Vmware info; ip address management info, etc), and the single server on the machine had booted itself back up. The issue is that ever since, it seems incredibly sluggish. The Server seems OK; the performance monitor never has anything maxed out. The most used it seems is network; I have only a single NIC in use (we have about 10 users total access data files on the server and it is running DNS, AD and DHCP services). I've replaced a switch in between thinking it may have been the culprit, but still have the same results. Sporadically, my RDP session to the server will be lost but eventually come back up.

Any ideas on where to look for the culprit for the slowdown? Any ideas on what I could try to increase performance? Is a single NIC enough for 10 users filesharing (we use Quicken files mostly, and office xls/word) nothing too intensive.

I tried once to add another NIC and "team" them, but it broke everything and I had a hard time getting back to change things since the "teamed" nic decided to use dhcp and I couldn't get to the dhcp server to find the address (since the server is the dhcp server).

Thanks for any ideas.


UPDATE

I have since taken the server off of the network and moved it to another spot. Since it's been off the network, it has been running fine (n.b. there was one more core dump same as in pic before I took off network). Could this be the result of an external mapped iSCSI datastore? I had 2 on the host and there were errors in vmkernel log about it not being able to log on, but at least some of the errors were known OK:https://kb.vmware.com/s/article/1031221 but not sure about others. Here are a couple of shots of the vmkernel errors: https://imgur.com/a/DOA3umn

Also, here is a pastebin of the latest from vmkernel.log. I also have syslog, usb, vmkeventd, vmksummary, vmkwarning, vprobed, and vpxa.log files I can upload. https://pastebin.com/rmp3k1G5

Update 2

I've left the machine running over the weekend, and as of now, it's still running just fine. No panic, no crash. Wondering if somehow the external NAS devices mapped as datastores via iSCSI could be the cuplrit? Especially concerning since I have them on a couple of other hosts as well...

  • Did you take the picture in such a low resolution? It's a bit difficult to read. Try re-uploading it in its original high resolution. Also you should plug into the server's DVI, DP or HDMI port if it has one, instead of the VGA port. If it only has a VGA port, use the monitor's controls to move the displayed image so that all of it is on screen. The missing line is probably not that important, but it might be. – Michael Hampton Jul 22 at 21:00
  • I didn't take the pic in low resolution, there's a size limit in the image upload feature here. Here's the original resolution, though: i.imgur.com/iNg30ng.jpg I did adjust the screen and the top line just says "VMWARE ESXI" and has a hardware ID or something, – stormdrain Jul 23 at 13:32
  • Really? I've uploaded the new image URL you gave, and it is at the original resolution. I've never heard of this size limit. That's weird. Anyway, now I can read it and like Don Zoomik I suspect you have failing hard drives or a failing controller. – Michael Hampton Jul 23 at 13:35
  • The limitation is within the serverfault imgur uploader. While making a post/question, you can add an image and it automatically uploads to imgur and generates the link and puts the link in the Question you are posting/asking. The limitation is in there.Likely so people aren't embedding 4000px images into their questions. – stormdrain Jul 23 at 14:05
  • 1
    Well, that's odd. I uploaded the too-large image, but by providing its URL. It looks like the uploader shrank it from 4.5MB to 709KB but also kept the original resolution. I imagine it complained because you uploaded it from your computer rather than from a URL. Mystery (probably) solved. – Michael Hampton Jul 23 at 14:41
5

Stack trace ends with megasas that means command to your MegaRAID (you probably have some RAID array) may have unexpectedly failed or it's a driver bug. I'd recommend updating MegaRAID firmware together with ESXi driver (and probably ESXi itself while you're at it). ESXi 5.1 is ancient but you can still get updates and async drivers from VMware site.

My gut feeling says that something is wrong with storage - this could also explain sluggishness, caused by very high disk latency (you should see it in PerfMon or Resource Monitor or vSphere Client monitoring). Check RAID and disk status in megacli/storcli or reboot to WebBIOS to check array and drive status.

| improve this answer | |
  • The strange thing is the Dell OpenManage page for the host didn't indicate any problems at all with the array or disks. All green across the board. I'm going to get the vmkernel logs to add to the question. Likely you guys are right, though. Thanks. – stormdrain Jul 23 at 14:11
  • Do you think this could be caused by an external mapped iscsi drive? Not internal Drive? I've disconnected the server from the network and it's been running fine since then... – stormdrain Jul 24 at 15:06
  • What does "external mapped" mean? Is it mounted by ESXi or Windows VM? If mounted by ESXi and used for Windows boot disk, it could be the culprit. If used for nor non-system drive, I'd doubt it. – Don Zoomik Jul 26 at 16:04
  • Sorry, mounted in esxi itself as a datastore (I had 2 external NAS devices mapped as datastores via iSCSI in esxi). Interestingly, it's been up and running now for 4 days without the panic since I've detached it from the main network (just have it in my office on a little Netgear switch with another machine I have Vsphere client on). – stormdrain Jul 27 at 13:20
  • 1
    Battery failure will hurt your write performance. I saw that you have RAID10 so it might not be that bad but worse still. No performance data to estimate. It's unlikely that iSCSI is the problem. If you're not sure, just remove the datastore and iSCSI configuration as you don't seem ot use it anyways. – Don Zoomik Jul 30 at 19:53
3

My first guess would be "storage".

How are disks and volumes configured? Which kind of RAID is in use?

I'm pretty sure you have a rebuilding or degraded disk array; that would explain the slowness.

| improve this answer | |
  • It is a raid 10 array. The strange thing is the Dell OpenManage page for the host didn't indicate any problems at all with the array or disks. All green across the board. I'm going to get the vmkernel logs to add to the question. Likely you guys are right, though. Thanks. – stormdrain Jul 23 at 14:10
  • Thanks. Seems likely correct it's storage. But could external iscsi datastores cause a panic like that? I've updated the question with some more shots including vmkernel errors: imgur.com/a/DOA3umn Thx – stormdrain Jul 24 at 15:12
  • What's the external iSCSI datastore used for? It's used by the host or by the VM? – Massimo Jul 24 at 15:20
  • Was just used to shuffle iso's around when setting up the server. Was not being used on an ongoing basis. – stormdrain Jul 24 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.