12

So, my apache server was slow, and I looked in the log files. Turned out they had grown to 12GB of accesses from tons and tons of different hosts trying to access /wpad.dat on one of my Vhosts.

Now, the virtual host in question is the "catch-all" vhost that is invoked when a browser doesn't supply a known hostname.

I am currently getting thousands of requests per minute to "/wpad.dat" and as far as Google can tell me, this is something that has something to do with proxy servers? But I don't use proxy servers, so why am I being literally bombarded by these requests.

I am getting more requests per minute for this non-existant file than I am getting normal requests. So my assumption is that I am under some form of attack. Funny thing is that it generally only occurs at night (here in Sweden) and not during the day.

A sample size of the latest 500 requests (i.e. half a minute) shows that it consists of 200 different hosts, and a small sample of those shows that they are all valid hosts (not TOR proxies) so is this some DNS servers being incorrectly configured? I do run a DNS server on the machine.

Please help! :)

EDIT The host they are accessing is "cluster.atlascms.se" so what they do is access http://cluster.atlascms.se/wpad.dat thousands of times per minute.

Now, cluster.atlascms.se is my DNS failover host. So all my clients point their subdomains to cluster.atlascms.se, which in turn points them to the current IP (master server of failover server).

As it seems - this means I am getting tons and tons of requests to cluster.arlascms.se - could that mean that my DNS is misconfigured?

  • 3
    You know, you could put up your own WPAD.DAT file and really have some fun with these people. >smile< Seriously, though, somebody has b0rked up a configuration somewhere horribly if they're pulling a WPAD.DAT from an untrusted source. You redirect all their browsers to a proxy you control and MiTM their traffic. – Evan Anderson May 23 '13 at 21:21
  • 3
    I would strongly be tempted to put up a wpad.dat that simply points the the local host. That should break things enough whoever is causing the problem might take the time to fix it. – Zoredache May 23 '13 at 21:23
  • 1
    @Sandman - The WPAD.DAT file needs to be Javascript to work. Have a look here: en.wikipedia.org/wiki/Web_Proxy_Autodiscovery_Protocol – Evan Anderson May 23 '13 at 21:28
  • 1
    BTW, it is extremely rude for you to discover a problem, and then try to dump your trash on someone else's lawn. So please don't set your wpad to point at someone elses systems. – Zoredache May 23 '13 at 21:36
  • 1
    The problem with your DNS setup is that any of your clients that use a wildcard dns entry to point all of their sub domains to cluster.atlascms.se will by default be pointing wpad.theirdomain.whatever there as well. Which means if they set their desktop hostname to something.theirdomain.whatever then it will lookup wpad.theirdomain.whatever, get your IP and repeatedly request wpad.dat thousands of times a day. – Justin Buser Aug 2 '14 at 6:35
9

It appears that your DNS zone eklundh.com has a wildcard record defined pointing to cluster.atlascms.se. This includes wpad.eklundh.com. I suggest you add a DNS record explicitly defining wpad.eklundh.com. to 127.0.0.1 or something.

  • 7
    I hate wildcard DNS records. They've never been anything but trouble. – Evan Anderson May 23 '13 at 22:03
  • Ok, I have now added a wpad subdomain to all my domains in my DNS that all point to 127.0.0.1 - I have to wait to see it propagate and see if this fixes the problem. – Sandman May 24 '13 at 5:13
  • Looking in my log files, the vast bulk of the requests aren't directed at a wpad subdomain, but to the IP or to cluster.atlascms.se... – Sandman May 24 '13 at 13:09
10

Machines will look for a WPAD.dat file hierarchically based on their own FQDN, if they are configured for proxy autodiscovery. So, if a windows PC is a member of a domain c.d.e.com, it will look for WPAD.dat in:

http://wpad.c.d.e.com/wpad.dat
http://wpad.d.e.com/wpad.dat
http://wpad.e.com/wpad.dat
http://wpad/wpad.dat

Chances are that somewhere, someone has a domain which is a subdomain of one of the ones you are hosting HTTP on, and hasn't properly configured or disabled proxy auto-discovery. In consequence, they are likely searching hierarchically.

It's possible that a virus may have caused them to do this; likely, if the machines making the query are extremely numerous and in diverse subnets, this is what is up.

If possible, avoid defining a DNS record for the wpad subdomain of anything you don't intend to use for proxy auto-discovery.

If this is not an option, you could consider using layer 7 filtering to find queries for wpad.dat and reject the packets with an ICMP message. This might actually be the most effective way to stop the traffic, unless the IPs are all from the same network and their technical contact in whois is responsive.

Things that will point a host at a particular location for wpad.dat include domain settings, the domain name option in DHCP replies, and an explicit setting in the web browser to load proxy information from some URL.

  • I do not have a wpad subdomain, but I do have a wildcard subdomain. I think the culprit may be my atlascms.se domain (for which I do not need a wildcard subdomain) which is the domain I use for my clients for their CNAME records. I have updated my DNS and I'll have to wait and see if this fixes things. – Sandman May 24 '13 at 5:15
  • The problem is that all these hundreds of thousands of requests I get from hundreds and hundreds of different hosts couldn't all possibly be thinking that atlascms.se is in their own subnet, surely? – Sandman May 24 '13 at 5:16
  • It has nothing at all do with subnets; it's all domains. – Falcon Momot May 24 '13 at 5:56
  • Yeah, sorry for the terminology confusion. – Sandman May 24 '13 at 7:03
  • It's entirely possible someone is trying to use your domain to host a malicious wpad.dat (and failing). There might be a virus out there setting your URL as the place to get wpad.dat from explicitly, or otherwise pointing hosts there, or there might be a misconfigured network out there. – Falcon Momot May 24 '13 at 7:09
4

The first thing I would do is to try to find out where these requests are going to, i.e. their destination. Apache doesn't log the hostname by default, so you can either use tcpdump to get a brief capture and inspect it for the Host: request header, or change your Apache log format to log it. I prefer to log it in the otherwise useless second field, for example:

LogFormat "%h %{Host}i %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined

Once you know who these mistaken requests are being addressed to, what to do next may become clear. For instance, it might turn out to be some big company example.se in which case you can go find their network admins and yell at them.

  • using server-status, I see the host they are "attacking", and it's not even a configured vhost (which is why it is logged by the catch-all vhost) - the host they ALL are connecting to is "atlas.eklundh.com" – Sandman May 23 '13 at 21:24
  • And also, all these requests comes from hundreds and hundreds of different hosts from all over the country and all over the ISP spectra, this is not coming from one source or one misconfiguredc company. I am wondering if I am doing something wrong with my eklundh.com domain here, whose DNS server I also run on the machine – Sandman May 23 '13 at 21:25
  • "atlas.eklundh.com" resolves to the same IP as "sandman.net". – Evan Anderson May 23 '13 at 21:26
  • 1
    You don't really need to configure systems to look for a wpad.dat. You just to have a valid DNS record like wpad.eklundh.com (you do have that record) and computers that have a FQDN that is set to something like *.eklundh.com` will automatically try to do the WPAD lookup. – Zoredache May 23 '13 at 21:49
  • 1
    The problem then becomes that any computer that thinks it's in the eklundh.com domain will see that wpad.eklundh.com is valid, and attempt to download a proxy server configuration from it. You can remove the DNS record, or reconfigure all of the computers. – Michael Hampton May 23 '13 at 21:52
0

Just FYI, ModSecurity will catch this and block it. There is a ruleset provided by Comodo. Here is a log entry. I stripped out account relevant data so it has in it just so as to use it as an example.

Apache-Error: [file ""] [] [] [client xxx.xxx.xxx.xxx] ModSecurity: Access denied with code 403 (phase 2). Matched phrase ".dat/" at TX:extension. [file ""] ["] [id "210730"] [rev "2"] [msg "COMODO WAF: URL file extension is restricted by policy"] [data ".dat"] [severity "CRITICAL"] [hostname "removed"] [uri "/wpad.dat"] [unique_id "WjFa06qDOW3DDPRieFmICgAAAEg"]

-1

Had this problem and fixed it by creating a wpad.dat file putting "this page left blank" page in it.

CPU went to nearly zero. Problem seems solved.

  • a syntactically wrong response, yet a success response code for a URI you clearly do NOT intend to serve is just wrong. – anx Nov 12 '17 at 15:10
  • But it solved the symptom. In this instance, it reduced the server load, go the site running again, and gave me time to redo the A-Records where the real problem lived. – Brian Tolman Nov 15 '17 at 21:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.