5

Recently, we had some problems with one of our ESXi servers, caused by the NTP Server DRDoS Amplification Attack using ntpdc.

How do I configure the NTP server on ESXi to not be exposed to this DDoS attack?

Or, if I switch off the service, will that have any effect on my VMs?

  • 3
    Your ESXi server is directly accessible from the Internet, without any sort of firewall?! – Chris S Feb 6 '14 at 19:21
  • @fefe Can you please add some details on what problems you had on the ESXi host, and why you think that it was related to such an attack? People answering and commenting here (incl. me) might make false assumptions about what happened and give wrong or irrelevant answers as a result. So you better provide more details to get the best answers. – VFrontDe Feb 6 '14 at 20:01
2

The answer can be found in this blog post. All you need to do is disable the "monlist" command, which by the way was removed in ntpd 4.2.7 (our ESXi 5.1.0u2 servers are running 4.2.6p2).

  1. Access your server's console, either by enabling the local console or SSH.
  2. Edit /etc/ntp.conf by adding noquery to the first restrict line.
  3. Restart the NTP service with /etc/init.d/ntpd restart.
  4. Verify that the monlist command has been disabled:

    ntpdc -c monlist 1.2.3.4

  • The September 2014 version of ESXi 5.1.0u2 appears to fix this issue, as the monlist command now returns "***Server reports data not found" in all cases. – Eric3 Oct 8 '14 at 21:30
3

The question is not answerable - and you may not like the answers that can be given.

2 scenarios, you fail to mention any detail to filter out:

1 - you were used to amplify. In this case I wonder why the ESXi host was accessible from the internet. It should not be. It should not have a public IP. I do not run ESX but maintain a number of Hyper-V servers for customers and man, you would not find ANY of them on the internt. They live in an internal network, all access is done via VPN to that internal gateway. Yes, servers may have public IP addresses - virtual servers - but never the hosts. No need to. ALl traffic they ned from the internet goes through the firewall (via NAT), so they are protected. I consider this professional baseline in security.

2 - you were targetted. In this case - no way. That is like saying "what do I chane on my house so that people do not send me tons of packages I never ordered". by the time the traffic hits the ESX host it already overloads your bandwidth. BUt again, why is the host at all on the internet?

  • You are right and make sense what you say. The sever was not configured by myself and being newbie I do not have to musch expreience on ESXi. How would I configure my external switch to not be exposed on internet? – fefe Feb 6 '14 at 11:43
  • Not at all. This is not something a switch does, but a router. Standard 101 setup - interneal network, internal addresses, edge router with NAT for outbound traffic and a VPN server for accessing the internal network. – TomTom Feb 6 '14 at 11:46
  • oh boy is getting complicated the server is not in here – fefe Feb 6 '14 at 11:47
  • @fefe It's also possible that you don't filter ntp traffic, (we certainly didn't care about ntp traffic coming into or going out of our network before this) and that's how it's reaching your ESXi server. If that's the case, you could block ntp traffic at the firewall/router for your network, but this might cause problems for legitimate ntp traffic you need. Sounds like you lot might benefit from having a network engineer on staff over there. :) – HopelessN00b Feb 6 '14 at 14:55
  • THe problem is that if you read his answer investigative, they have a esxi server with a switch in front. No router, no firewall.... otherwise it would not be a problem to go to a nice vpn based setup anyway ;( – TomTom Feb 6 '14 at 15:04
1

How do I configure the NTP server on ESXi to not be exposed to this DDoS attack?

At the moment, you can't, really. The ntpd in ESXi isn't really configurable (at least in a VMware-supported fashion), so your options are really on or off.

Presumably, VMware will release a patch or update sometime soon to address the issue, (I don't see one that says it addresses the issue right now), and you could apply that when it comes out, but that won't help right now.

You could manually update your ESXi host to a newer ntpd client that isn't vulnerable (ESXi is "just" a customized Linux distro, after all), but I wouldn't do that and risk being in an unsupported configuration with VMware.

Of course, as you suggest in your question, you can also prevent the attack by turning the service off (for the time being).

Or, if I switch off the service, will that have any effect on my VMs?

That entirely depends on how your guest operating systems are configured for ntp. If they're configured to synchronize time with the host system, they'll start to lose time sync (the more time they spend at idle CPU usage, the worse time drift you'll see).

If they're configured to get time from a different ntp source, they won't have problems... unless, of course, they're also running vulnerable ntp clients, in which case, they'll probably be getting the DRDoS attack instead of the host.

If you're currently configuring your virtual guest OSes to get time from their host servers, now might be a good time to consider a different approach, which would depend entirely on your use case. Running Windows domains makes this easy - the PDC emulator gets ntp time from a reliable (external) ntp source, all other Domain Controllers get their time from the PDC emulator, and all clients get their time from their local Domain Controller. Of course, your use case may well be different or more complicated.

  • 2
    SO, why would anyone put an ESXi server onto the internet in the first place then? No need to have that one with a public IP on a public network, or? – TomTom Feb 6 '14 at 11:31
  • @TomTom just because there's no good reason to do something doesn't mean that it's not happening. Seriously, at least half the questions on this site are an example of someone doing something stupid with no reason for doing it, and then having problems as a result. In these parts, saying here's how you fix that problem you made is generally preferable to saying you're doing something stupid, so... well, that. (But no, I can't think of a good reason to put an ESXi host on a public-facing IP address, either. Maybe I just lack imagination.) – HopelessN00b Feb 6 '14 at 11:35
  • Sorry @HopelessN00b, you made some false claims that I need to correct: 1. The NTP daemon is configurable on ESXi by editing /etc/ntp.conf (See e.g. kb.vmware.com/kb/1005092) 2. ESXi is NOT a customized Linux distro. It does use many of the GNU utilities and packages that are also used on Linux and other Unix OSs, but it does not use the Linux kernel. And you cannot just install some Linux package in ESXi. 3. Turning the NTP service off is not the only option. You can secure it through modifications of /etc/ntp.conf and/or by configuring the ESXi internal firewall. – VFrontDe Feb 6 '14 at 18:55
0

First of all, you should not have your ESXi retrieving its time from an outer source to avoid this.

I recommend you to create a NTP Server internally which will themself be pooled by the ntp.org servers and protected correctly to mitigate NTPD DDOS attack.

  • As mentioned by TomTom, this only applies if he's the source of the attack, not the target. (He didn't say which he was, of course.) – HopelessN00b Feb 6 '14 at 11:32
  • Yep, of course. But even if it is the target esxi should always rely for an internal time source ;-) Best practices advice x) then the internal serveur should be configurated to restrict sources and client on its config file + have an update upper than the 4.47 version of ntp. – Dr I Feb 6 '14 at 11:39
  • Those best practice don't really apply here - see my answer. For one thing, the ESXi ntpd is not really configurable in a vendor-supported manner (it's on, or it's off), and VMware has yet to release a patch addressing this relatively recent vulnerability... so you can't upgrade to a safe version, and stay in a vendor-supported configuration, which is an important consideration for a professional IT environment. – HopelessN00b Feb 6 '14 at 11:44
  • I didn't mention to replace the ESXi binary ;-) just adviced to change the NTP targeted server and THEN on the NTP server to operate maintenance and update ;-) But I may not have express my thoughts so well. – Dr I Feb 6 '14 at 14:43
  • 1
    There is nothing "wrong" with configuring your VMWare hosts to use an external time server for their reference clock (an internal server is generally a better solution, but certainly not a mandatory one). The problem is allowing outside systems to talk to the NTP daemon on the ESXi hosts. – voretaq7 Feb 7 '14 at 22:05
0

I'm really puzzled by all the long but irrelevant answers here ...

The answer is easy: Enable the ESXi bultin firewall!

By default it is enabled and blocks incoming NTP traffic. So why the heck have you disabled it in the first place??

  • 1
    And what makes you say the firewall is disabled? There's no evidence of that in the post. Not that blocking incoming ntp traffic helps you much if you're participating in a DDoS by sending ntp traffic out, either... which could well be the case (the asker never specified which side of the problem he was on). – HopelessN00b Feb 6 '14 at 19:04
  • What inside ESXi could cause it to be the original source of such an attack?? Once it was made part of the attack it is of course also a source, but it can only be made part of it by being accessed from the outside first. – VFrontDe Feb 6 '14 at 19:29
  • For some reason "malware" springs to mind immediately. What does it really matter? The question (which is admittedly poorly asked, and bound to generate confusion) is about how to defend against a given DDoS attack. "Enable the firewall" is one valid answer, (it's on by default, so I don't particularly like it as an answer, by itself) but so is everything else I've seen in this thread. For all we know, this guy's running an ntp server (as a time source) on his ESXi host, and that's why it accepts incoming traffic. Point is that "why" or "what" don't really matter to the question at bar. – HopelessN00b Feb 6 '14 at 19:40
  • In ESXi the NTP daemon runs by default, but it is supposed to be used in client mode only. This is why it is allowed to query other external servers to synchronize time, but the ESXi firewall will by default not allow it to be queried itself from the outside. There is not even an option available to open the NTP port for incoming traffic, so the only option to use it as a time source for external partners is to disable the firewall completely. I cannot think of any other scenario that would allow it to become part of an NTP based DDOS attack. – VFrontDe Feb 6 '14 at 19:52
  • Just to let you know firewall was not disabled – fefe Feb 6 '14 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.