I work for a local government that bills for water usage and garbage collection. I received a call today from a customer that a "New York company" called him asking for his customer number and pin to give them access to his online account so they can "scrape" his water usage from the site. They are apparently collecting that information from several apartments for some reason that he couldn't recall. Of course this raises flags with me and I informed him to not give them this information. I also told him that if they call back to tell this company that they can contact us directly to get this information if they require it.

Now, if one of our customers received a call, I feel it's safe to assume that others have gotten the same or similar call and they may or may not have given this information. How can I check our logs to see if there is a bot hitting our site and screen-scraping our data? I also feel we should block that bot and prevent further attempts.

Note: The only information stored on the web server is the name and address of the customer, water usages and costs of the bills with a total amount due. They can also pay the bill. We don't store any account information online. So overall, the information on the web server could be considered public information (though the proper channels).

  • 1
    IANAL but... if the user has given them their logon details then that bot is in effect acting on behalf of the user, which I imagine would be seen as legitimate access. I'd run that by legal to ensure you're not creating a whole new bunch of problems for yourself. – John Gardeniers Jan 19 '12 at 3:50
  • Yeah - look at Mint.com. The users give their login details to the aggregator, which then logs in on their behalf. (And it sometimes has troubles - some of my CC accounts weren't because Mint wasn't capturing the challenge questions properly.) – mfinni Jan 19 '12 at 14:06
  • 1
    how do you know he was even called by the company at all, and "the company" isnt just someone wanting his login details for your site ? – Sirex Jan 19 '12 at 15:31
  • Why do you mind if they scrape this data? How is it going to harm the local government, or your citizens? I agree that it is worrying that the company might be stealing personal information your users. However, that doesn't seem to be what is concerning you - you seem to be concerned that somebody might be gathering gup data about water and garbage collection usage. The main reasons I can think somebody would do that are good. They're probably running a business that ultimately reduce water use or help recycle garbage. What's the bad use case you're worried about? – frabcus Jan 7 '13 at 11:45
  • @frabcus Because there is a process to getting that data. We can then run a query on the data they need/want and give them the data. – Mike Wills Jan 7 '13 at 14:39

There's no way to detect or block a well-written bot that's only scraping a small number of pages -- its behaviour can be indistinguishable from a genuine user.

  • This is exactly what I am afraid of. We could implement standard bot-blocking techniques, but who's to say they aren't paying some foreigners to log in manually and pull in information either. – Mike Wills Jan 18 '12 at 21:28
  • 2
    Well - and I'll forgive you for laughing - you'll just have to hope that users will keep their information to themselves if they want their information to be kept privately. Assuming you have a normal route for contacting customers, you could warn them about these calls through that. – Rob Moir Jan 19 '12 at 16:14
  1. You could block or rate-limit any single source IP that is accessing more than one account. As mentioned above, this would have to be able to know that more than account is being accessed and might not be trivial to implement. This could also block tenants in an apartment complex that have NAT-ted internet as a "utility", of course.

  2. You could implement a CAPTCHA.

  • 1
    +1 Good call on the captcha – squillman Jan 18 '12 at 21:47
  • 7
    I wanted to say "CAPTCHA", too, but it pains me. That's punishing the legitimate users of the site. – Evan Anderson Jan 18 '12 at 21:48
  • I think we would go a different route with that. We have people of all ages and technical ability use the site. I don't thin CAPTCHA is quite where we would go. – Mike Wills Jan 18 '12 at 22:21
  • 2
    If they've got the time to call everybody, I bet they've got the time to handle CAPTCHAs. – Jeff Ferland Jan 19 '12 at 15:51
  • 2
    You could make it so that the CAPTCHA appears as a rate limiting tool... i.e. "if the same IP address makes more than one request for <the resource you want to protect> in a x second span, display a CAPTCHA". – Kyle Hodgson Jan 23 '12 at 18:13

If there is a bot screen scraping your web site then your only indication of that would be in your web server logs, and it's going to be difficult (at best) to detect. Usually the way you see bots in logs is through the User-Agent string, but that really is only used properly for the well-behaving bots like search crawlers. All of the other nasties will set the user-agent string to match something common like one of the major browsers in order to hide themselves.

Most likely you're looking at tracing IP addresses that access specific URL's that you can directly tie back to a specific customer. This is further complicated if the requests are POST requests since the customer specific information is likely in the POST data and not in the URL such as you'd see with a GET request.

Honestly, best of luck with that unfortunately... Not sure you're going to be able to get very far.


If they are screen scraping, they likely access only the target page without any related content like css, jsp, and images. You would need to search your access logs to see if this kind of activity is occurring. You may catch some legitimate browsers that don't reload data that is already cached.

You might be able to detect excessive visits to the page in question from a single IP address. This may catch a few ISPs that are NATing their customers access.

Running a GeoIP check should quickly tell you if you have accesses from other countries. Some of this may be legitimate customers who are living abroad or traveling.

If your site has a News feature, it might be worthwhile putting up a posting about this situation. This may get you some more reports.

  • Considering we only collect up to 50 payments a day, seeing excessive visits wouldn't be hard to see. I hope... – Mike Wills Jan 19 '12 at 14:24
  • At that volume you could eyeball it. – BillThor Jan 20 '12 at 1:51
  • +1 A well-written bot will probably skip a few steps for efficiency. Focus on what is missing, as opposed to what it's doing. – Nic Jan 20 '12 at 2:12

Log the IP of every access to an account. After a while, go back and query your logs assigning a point to each IP for each account it accesses, then sort to find the IP addresses that have accessed the most accounts.

After you rule out some libraries and such, I bet you'll find your culprit even if they do access the accounts very slowly or rarely. That starts to stick out after a month if they're doing it from the same place -- which they more than likely are.

Some other permutations: first limit to accounts that have been accessed from more than one IP -- the real user and the scraper. Find the account of any one user that know has given out the information.


Couldn't you flag and slow down (Rate limit) and/or block a bot if you had one IP accessing multiple accounts? Sure, it is possible for multiple real users to share an IP address (say at an office) but what is the likelihood of a hundred people all using the service and the same IP? Also the bot will probably have the same user agent every time (depends on their programming).

Depending on how you get to the information in question, you can hide data on screen, or in webforms. Like most bots don't handle javascript, so you can use that to modify data in submits.


you can implement some javascript into login page. when page loads, set form action to something like this http://mydomain.ltd/login/bot , when user hits submit (login) button then change action to valid url or document.getElementById('login_form_id').action = "http://mydomain.ltd/login/human".

If bot is crawling your data, then typically bot doesn't support JS. Of course they will figure out, why bot isn't working. But you can log failed IP addresses and then compare later with success logged in addresses.

If human is collecting data, then it is difficult to figure it out.

Also you can analyze user agent string and IP address. If user is logging out and then logging into another account (in 5 minutes ?), then you can log that. If that is human, they will look one account, then another etc ...


This stinks but the way I've beaten simple screen scrapers in the past is to use silverlight or flash to put the text on the screen as images rather than text. At least at that point they have to OCR the images rather than simply capture the HTML output and parse it.

  • 2
    Do the tools you use for that allow for accessibility? – mfinni Jan 19 '12 at 14:04
  • Or mobile? Mobile is a high-priority for us. And accessibility, it is the law for us. – Mike Wills Jan 19 '12 at 14:22
  • For accessibility yes as the image was also an image map that had alt text that linked to just the records selected by name, so that screen readers wold read out the list by name which could then be drilled into. For mobile it was a whole different website that presented info in a different format. For mobile a webform gave you the information on a per record basis that you paged thru – Jim B Jan 19 '12 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.