Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This question already has an answer here:

My VMWare ESXi 4 server appears to be under a Denial of Service attack. I am getting massive packet loss to the server (60+%) and am barely able to load any services on the VMs running on the host.

I have Cacti installed but cannot load it due to the attack. I can SSH in to the VMware host. Are there any commends I can run to either determine where the attack is coming from, or block all IP addresses except mine so that I can load Cacti again to troubleshoot?

I tried esxcli network firewall get but received: Unknown Object firewall in namespace network

All the VMs with network access are directly connected to the internet, that is, there is a virtual switch between the internet-facing VMs and the router.

EDIT: MDMarra had a great idea: disable the vswitch that the VMs are on. But I can't get the vSphere console to respond long enough to do this. Can this be done through SSH?

share|improve this question

marked as duplicate by Josh, Michael Hampton Jun 29 at 22:23

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

    
You have called your ISP, right? –  Michael Hampton Jul 27 '12 at 19:34
    
Yes, thanks @MichaelHampton. They told me it was a DoS / DDoS but were unable to assist beyond that. If I can't get this resolved soon I am calling them again. I was hoping maybe SF would be more helpful, because they were not! –  Josh Jul 27 '12 at 19:36

3 Answers 3

up vote 1 down vote accepted

The ISP was unable to determine the cause of the traffic, but what they were able to do was null-route all the IP addresses assigned to this server at the network switch. Then, one-by-one we removed the null routes, until we determined which IP addresses were being attacked. Once the target IPs were null-routed, the problem went away and I am able to access the server again.

I am now going to console in to the affected VMs and start tcpdump, and then remove the null-routes to those VMs. This will allow me to find the source IPs of the attack, which can be blocked by my ISP before traffic from them enters the core network.

share|improve this answer

I would say first and foremost would be to call your datacenter and see if they can block the offending IP with their equipment. Hopefully their hardware has the bandwidth to handle something like that, which will then at least allow yours to start functioning like normal.

share|improve this answer
1  
Thanks. Unfortunately they were unable to even tell me where the attack was coming from, much less block it, which I found patently absurd! –  Josh Jul 27 '12 at 19:31
    
Wow, that is unfortunate! –  Safado Jul 27 '12 at 19:43
    
Yeah, it's because they want to sell me a $500/mo DDoS prevention plan rather than helping a long time customer. It's time for a new ISP. –  Josh Jul 27 '12 at 19:48
    
@Josh, do you have console access to this machine? If not, you should make plans to do so going forward. –  Mike Pennington Jul 27 '12 at 19:54
    
Thanks Mike. I did when I performed the initial setup through a IP KVM, but I stopped that service after the install because I was able to do all I needed to through SSH / vSphere center... –  Josh Jul 27 '12 at 19:55

Sniff the wire and filter traffic to just that host. tcpdump / wireshark

share|improve this answer
    
There is no tcpdump on vmware ESXi, and I am getting so much packet loss that when I try to run it on the virtual machines, it just locks up. –  Josh Jul 27 '12 at 19:23
1  
you misunderstand. Configure a span port on your switch so you can monitor the traffic with a laptop. Span whatever Vlan your server is in. –  Paul Ackerman Jul 27 '12 at 19:24
    
Sorry for not being clear, I'm in the middle of trying to fix this ASAP as all my customers' sites are down. This is a managed server, I have no access to the switch. I will ask the datacenter if they can do that, however, thanks –  Josh Jul 27 '12 at 19:28
    
ahh. I see. In that case, can you run vm-support and get VMware to help you diagnose? –  Paul Ackerman Jul 28 '12 at 1:09
    
I don't know the specifics but the vm-support utility can diagnose various performance issues so it may provide insight into the network traffic. –  Paul Ackerman Jul 28 '12 at 1:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.