1

My VMWare ESXi 4 server appears to be under a Denial of Service attack. I am getting massive packet loss to the server (60+%) and am barely able to load any services on the VMs running on the host.

I have Cacti installed but cannot load it due to the attack. I can SSH in to the VMware host. Are there any commends I can run to either determine where the attack is coming from, or block all IP addresses except mine so that I can load Cacti again to troubleshoot?

I tried esxcli network firewall get but received: Unknown Object firewall in namespace network

All the VMs with network access are directly connected to the internet, that is, there is a virtual switch between the internet-facing VMs and the router.

EDIT: MDMarra had a great idea: disable the vswitch that the VMs are on. But I can't get the vSphere console to respond long enough to do this. Can this be done through SSH?

Improve this question
2
  • You have called your ISP, right?
    – Michael Hampton
    Jul 27, 2012 at 19:34
  • Yes, thanks @MichaelHampton. They told me it was a DoS / DDoS but were unable to assist beyond that. If I can't get this resolved soon I am calling them again. I was hoping maybe SF would be more helpful, because they were not!
    – Josh
    Jul 27, 2012 at 19:36

3 Answers 3

Reset to default
2

I would say first and foremost would be to call your datacenter and see if they can block the offending IP with their equipment. Hopefully their hardware has the bandwidth to handle something like that, which will then at least allow yours to start functioning like normal.

Improve this answer
9
  • 1
    Thanks. Unfortunately they were unable to even tell me where the attack was coming from, much less block it, which I found patently absurd!
    – Josh
    Jul 27, 2012 at 19:31
  • Wow, that is unfortunate!
    – Safado
    Jul 27, 2012 at 19:43
  • Yeah, it's because they want to sell me a $500/mo DDoS prevention plan rather than helping a long time customer. It's time for a new ISP.
    – Josh
    Jul 27, 2012 at 19:48
  • @Josh, do you have console access to this machine? If not, you should make plans to do so going forward.
    – Mike Pennington
    Jul 27, 2012 at 19:54
  • Thanks Mike. I did when I performed the initial setup through a IP KVM, but I stopped that service after the install because I was able to do all I needed to through SSH / vSphere center...
    – Josh
    Jul 27, 2012 at 19:55
1

The ISP was unable to determine the cause of the traffic, but what they were able to do was null-route all the IP addresses assigned to this server at the network switch. Then, one-by-one we removed the null routes, until we determined which IP addresses were being attacked. Once the target IPs were null-routed, the problem went away and I am able to access the server again.

I am now going to console in to the affected VMs and start tcpdump, and then remove the null-routes to those VMs. This will allow me to find the source IPs of the attack, which can be blocked by my ISP before traffic from them enters the core network.

Improve this answer
0

Sniff the wire and filter traffic to just that host. tcpdump / wireshark

Improve this answer
5
  • There is no tcpdump on vmware ESXi, and I am getting so much packet loss that when I try to run it on the virtual machines, it just locks up.
    – Josh
    Jul 27, 2012 at 19:23
  • 1
    you misunderstand. Configure a span port on your switch so you can monitor the traffic with a laptop. Span whatever Vlan your server is in.
    – Paul Ackerman
    Jul 27, 2012 at 19:24
  • Sorry for not being clear, I'm in the middle of trying to fix this ASAP as all my customers' sites are down. This is a managed server, I have no access to the switch. I will ask the datacenter if they can do that, however, thanks
    – Josh
    Jul 27, 2012 at 19:28
  • ahh. I see. In that case, can you run vm-support and get VMware to help you diagnose?
    – Paul Ackerman
    Jul 28, 2012 at 1:09
  • I don't know the specifics but the vm-support utility can diagnose various performance issues so it may provide insight into the network traffic.
    – Paul Ackerman
    Jul 28, 2012 at 1:36

Not the answer you're looking for? Browse other questions tagged .