The tag has no wiki summary.

learn more… | top users | synonyms

1
vote
0answers
42 views

Why would someone repeatedly enter “-1” into a site search? [closed]

One of the websites I work on has been getting bombarded with site searches of "-1". The IP address appears to be coming from Eastern Europe and my first suspicion is that this is malicious. However ...
0
votes
2answers
182 views

Protection against scrapping with nginx

This morning we had a crawler going nuts on our server hitting our site almost 100 times per second. We'd like to add a protection for this. I guess I'' have to use HttpLimitReqModule but I don't ...
-2
votes
3answers
79 views

Stop server crash [duplicate]

I run my python scripts and Scrapy framework for web scraping project on my Ubuntu 12.04 precise server. These scripts run whole day. This project is under developing/testing stage. So i dont know ...
1
vote
1answer
347 views

Protect nginx from hammering

I would like to protect my nginx+pessenger+rails3 HTTP server from hammering/scraping. If you try to scrape the Google it shows you a captcha in case you make too many requests from the same IP. What ...
3
votes
2answers
388 views

IP address of spiders and “official” web bots

Is there an official API to iplists.com from where I can get the list of spiders? My intention is to whitelist these IPs for site scraping.
5
votes
2answers
2k views

Most efficient (time, cost) way to scrape 5 million web pages?

I have a list of web pages that I need to scrape, parse and then store the resulting data in a database. The total is around 5,000,000. My current assumption of the best way to approach this is to ...
1
vote
1answer
133 views

iis 6 anti data harvesting/scraping

We have a page on our extranet website that exposes information we would like to prevent from being data harvested. We have done the due diligence of encrypting the URL parameters to make it hard for ...
1
vote
1answer
1k views

Amazon EC2 + S3 + Python + Scraping - The cheapest way of doing this?

I tapped in to Amazons AWS offerings and please explain this in high level - if I am thinking right. So I have few Python scraping scripts on my local machine. I want to use AWS for super fast ...
1
vote
1answer
92 views

Can a scraping bot have JavaScript enabled?

I've got a few thousand request that seem to be coming from a client with JavaScript enabled and I'm wondering if that client could be a bot.
0
votes
1answer
125 views

Cannot find source of traffic spike

I noticed in my Munin graphs for Apache that there was a large spike in traffic yesterday. However, I have been unable to correlate this with anything on the site. Google Analytics does not show ...
1
vote
1answer
690 views

How can I use fail2ban to block scrapers?

I have a media site and problems of users coming along and scraping all of the content.I placed a invisible URL on the page to catch spiders that immediately blocks the ip, but some people have ...
0
votes
1answer
88 views

Grab js+flv video without embed option

I'm running a website for a political organization and was asked to post this article to the blog along with the embedded video: http://weareaustin.com/fulltext/?nxd_id=135746 I couldn't figure out ...
3
votes
1answer
365 views

What to do about spoofed user agents? Scrapers pretending to be spiders

I've been following a few spiders in our logs and I did a traceroute on their ip to find out they are in fact EC2 instances. The user agents are listed as Google bot and msnbot but they are not ...
1
vote
0answers
158 views

methods for preventing large scale data scraping from REST api

I know the immediate answer to this is going to be there is no 100% reliable method of doing this. But I'd like to create a question that details the different possibilities, the difficulty of ...
4
votes
4answers
577 views

How easy/expensive is it to adopt Google Mini/Google Appliance for intranet search?

Out of curiosity, is anyone here using Google Mini or Google Search Appliance to provide intranet search? Was it easy to set up? What kind of prices do they charge (ball park figure, I'm sure it ...
4
votes
7answers
566 views

How to Avoid Being Scraped

We have a searchable DB, we limit the results to 15 per page and only 100 results and yet still get people trying to scrape the site. We ban sites that hit it fast enough but i was wondering if there ...