The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [scraping]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
618 views

Block website scraper in Haproxy

I am using Haproxy. I want to block scrapers from my website. In the haproxy.cfg , I have created a rule. acl blockedagent hdr_sub(user-agent) -i -f /etc/haproxy/badbots.lst http-request deny if ...
1
vote
0answers
4k views

How to Enable JavaScript for wget in linux for grabbing a website?

I use wget like this for save site : wget --page-requisites --no-parent --mirror http://example.com/index.html -P /home/ in Some cases It does NOT work, Error is : This site requires Javascript to ...
1
vote
0answers
42 views

HTTrack stores extensionless pages with a .html appended

I'd like to mirror an old site of mine to local files. I've used httrack for this in the past, but I'm having a problem this time that I really thought I figured out before, but can't seem to now. ...
0
votes
0answers
21 views

How to capture tables with different structure from web

I have thousands of web pages(need login with username and passwords) like https://XXX.incometax.XXX/Preview/ViewDetail?TIN_INFO_NO=11935# where only last four digits(11935 for this example) changes ...
-1
votes
0answers
42 views

Is offering the contents of a third party web site offline violating the law? [closed]

I have developed a nice little app that crawls a bunch of newspaper web sites and makes their latest content available on my phone offline. It's basically a Pocket app that saves contents ...
1
vote
1answer
49 views

Suspected malicious activity by one of my site's users; any way to know for sure?

In the course of about 2 hours, a logged in user on my website accessed roughly 1,600 pages in a way that looks suspiciously similar to a bot. I am concerned because users must purchase access to the ...
0
votes
1answer
325 views

Can OpsView or Nagios be set up to report on a device based on status emails it sends out?

I'm looking at setting up a Nagios (or perhaps OpsView) server for monitoring our network. I have a few periphery devices whose oid schema doesn't include nodes for some metric I want to monitor. ...
1
vote
2answers
927 views

Protection against scrapping with nginx

This morning we had a crawler going nuts on our server hitting our site almost 100 times per second. We'd like to add a protection for this. I guess I'' have to use HttpLimitReqModule but I don't ...
-2
votes
3answers
379 views

Stop server crash [duplicate]

I run my python scripts and Scrapy framework for web scraping project on my Ubuntu 12.04 precise server. These scripts run whole day. This project is under developing/testing stage. So i dont know ...
2
votes
1answer
862 views

Protect nginx from hammering

I would like to protect my nginx+pessenger+rails3 HTTP server from hammering/scraping. If you try to scrape the Google it shows you a captcha in case you make too many requests from the same IP. What ...
3
votes
2answers
829 views

IP address of spiders and “official” web bots

Is there an official API to iplists.com from where I can get the list of spiders? My intention is to whitelist these IPs for site scraping.
8
votes
2answers
7k views

Most efficient (time, cost) way to scrape 5 million web pages?

I have a list of web pages that I need to scrape, parse and then store the resulting data in a database. The total is around 5,000,000. My current assumption of the best way to approach this is to ...
1
vote
1answer
203 views

iis 6 anti data harvesting/scraping

We have a page on our extranet website that exposes information we would like to prevent from being data harvested. We have done the due diligence of encrypting the URL parameters to make it hard for ...
1
vote
2answers
5k views

Amazon EC2 + S3 + Python + Scraping - The cheapest way of doing this?

I tapped in to Amazons AWS offerings and please explain this in high level - if I am thinking right. So I have few Python scraping scripts on my local machine. I want to use AWS for super fast ...
1
vote
1answer
142 views

Can a scraping bot have JavaScript enabled?

I've got a few thousand request that seem to be coming from a client with JavaScript enabled and I'm wondering if that client could be a bot.
0
votes
1answer
188 views

Cannot find source of traffic spike

I noticed in my Munin graphs for Apache that there was a large spike in traffic yesterday. However, I have been unable to correlate this with anything on the site. Google Analytics does not show any ...
2
votes
1answer
1k views

How can I use fail2ban to block scrapers?

I have a media site and problems of users coming along and scraping all of the content.I placed a invisible URL on the page to catch spiders that immediately blocks the ip, but some people have ...
0
votes
1answer
111 views

Grab js+flv video without embed option

I'm running a website for a political organization and was asked to post this article to the blog along with the embedded video: http://weareaustin.com/fulltext/?nxd_id=135746 I couldn't figure out ...
3
votes
1answer
769 views

What to do about spoofed user agents? Scrapers pretending to be spiders

I've been following a few spiders in our logs and I did a traceroute on their ip to find out they are in fact EC2 instances. The user agents are listed as Google bot and msnbot but they are not ...
1
vote
0answers
235 views

methods for preventing large scale data scraping from REST api

I know the immediate answer to this is going to be there is no 100% reliable method of doing this. But I'd like to create a question that details the different possibilities, the difficulty of ...
4
votes
4answers
627 views

How easy/expensive is it to adopt Google Mini/Google Appliance for intranet search?

Out of curiosity, is anyone here using Google Mini or Google Search Appliance to provide intranet search? Was it easy to set up? What kind of prices do they charge (ball park figure, I'm sure it ...
8
votes
7answers
887 views

How to avoid being scraped?

We have a searchable Database(DB) , we limit the results to 15 per page and only 100 results yet still get people trying to scrape the site. We are banning sites that hit it fast enough. I was ...