Questions tagged [scraping]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
2answers
58 views

website being mirrored by another domain

So my website is being mirrored by another domain name, I tried many ways to block the access from that specific domain but no hope, I am using cloudflare CDN and the website mirroring my site using ...
0
votes
1answer
82 views

How to reduce “RAM Cache + Buffer” size due to scrapy

I am running a couple of spiders in parallel by scrapyd 1.2. Each process will raise the Buffer during the crawl significantly as seen in the chart. What is this value and how can I reduce the ...
0
votes
1answer
1k views

Block website scraper in Haproxy

I am using Haproxy. I want to block scrapers from my website. In the haproxy.cfg , I have created a rule. acl blockedagent hdr_sub(user-agent) -i -f /etc/haproxy/badbots.lst http-request deny if ...
2
votes
0answers
7k views

How to Enable JavaScript for wget in linux for grabbing a website?

I use wget like this for save site : wget --page-requisites --no-parent --mirror http://example.com/index.html -P /home/ in Some cases It does NOT work, Error is : This site requires Javascript to ...
1
vote
0answers
57 views

HTTrack stores extensionless pages with a .html appended

I'd like to mirror an old site of mine to local files. I've used httrack for this in the past, but I'm having a problem this time that I really thought I figured out before, but can't seem to now. ...
0
votes
0answers
22 views

How to capture tables with different structure from web

I have thousands of web pages(need login with username and passwords) like https://XXX.incometax.XXX/Preview/ViewDetail?TIN_INFO_NO=11935# where only last four digits(11935 for this example) changes ...
-1
votes
0answers
42 views

Is offering the contents of a third party web site offline violating the law? [closed]

I have developed a nice little app that crawls a bunch of newspaper web sites and makes their latest content available on my phone offline. It's basically a Pocket app that saves contents ...
1
vote
1answer
53 views

Suspected malicious activity by one of my site's users; any way to know for sure?

In the course of about 2 hours, a logged in user on my website accessed roughly 1,600 pages in a way that looks suspiciously similar to a bot. I am concerned because users must purchase access to the ...
0
votes
1answer
341 views

Can OpsView or Nagios be set up to report on a device based on status emails it sends out?

I'm looking at setting up a Nagios (or perhaps OpsView) server for monitoring our network. I have a few periphery devices whose oid schema doesn't include nodes for some metric I want to monitor. ...
1
vote
2answers
1k views

Protection against scrapping with nginx

This morning we had a crawler going nuts on our server hitting our site almost 100 times per second. We'd like to add a protection for this. I guess I'' have to use HttpLimitReqModule but I don't ...
-2
votes
3answers
386 views

Stop server crash [duplicate]

I run my python scripts and Scrapy framework for web scraping project on my Ubuntu 12.04 precise server. These scripts run whole day. This project is under developing/testing stage. So i dont know ...
2
votes
1answer
878 views

Protect nginx from hammering

I would like to protect my nginx+pessenger+rails3 HTTP server from hammering/scraping. If you try to scrape the Google it shows you a captcha in case you make too many requests from the same IP. What ...
3
votes
2answers
875 views

IP address of spiders and “official” web bots

Is there an official API to iplists.com from where I can get the list of spiders? My intention is to whitelist these IPs for site scraping.
8
votes
2answers
8k views

Most efficient (time, cost) way to scrape 5 million web pages?

I have a list of web pages that I need to scrape, parse and then store the resulting data in a database. The total is around 5,000,000. My current assumption of the best way to approach this is to ...
1
vote
1answer
205 views

iis 6 anti data harvesting/scraping

We have a page on our extranet website that exposes information we would like to prevent from being data harvested. We have done the due diligence of encrypting the URL parameters to make it hard for ...
1
vote
2answers
5k views

Amazon EC2 + S3 + Python + Scraping - The cheapest way of doing this?

I tapped in to Amazons AWS offerings and please explain this in high level - if I am thinking right. So I have few Python scraping scripts on my local machine. I want to use AWS for super fast ...
1
vote
1answer
150 views

Can a scraping bot have JavaScript enabled?

I've got a few thousand request that seem to be coming from a client with JavaScript enabled and I'm wondering if that client could be a bot.
0
votes
1answer
199 views

Cannot find source of traffic spike

I noticed in my Munin graphs for Apache that there was a large spike in traffic yesterday. However, I have been unable to correlate this with anything on the site. Google Analytics does not show any ...
2
votes
1answer
2k views

How can I use fail2ban to block scrapers?

I have a media site and problems of users coming along and scraping all of the content.I placed a invisible URL on the page to catch spiders that immediately blocks the ip, but some people have ...
0
votes
1answer
113 views

Grab js+flv video without embed option

I'm running a website for a political organization and was asked to post this article to the blog along with the embedded video: http://weareaustin.com/fulltext/?nxd_id=135746 I couldn't figure out ...
3
votes
1answer
773 views

What to do about spoofed user agents? Scrapers pretending to be spiders

I've been following a few spiders in our logs and I did a traceroute on their ip to find out they are in fact EC2 instances. The user agents are listed as Google bot and msnbot but they are not ...
4
votes
4answers
629 views

How easy/expensive is it to adopt Google Mini/Google Appliance for intranet search?

Out of curiosity, is anyone here using Google Mini or Google Search Appliance to provide intranet search? Was it easy to set up? What kind of prices do they charge (ball park figure, I'm sure it ...
8
votes
7answers
939 views

How to avoid being scraped?

We have a searchable Database(DB) , we limit the results to 15 per page and only 100 results yet still get people trying to scrape the site. We are banning sites that hit it fast enough. I was ...