2 votes

How to reduce "RAM Cache + Buffer" size due to scrapy

Linux will use available memory for various caches, mostly file related. slabtop command to see details. how can I reduce the footprint? You don't. These will be evicted quickly and automatically if ...
user avatar
1 vote

How to identity who is scraping my website?

The honorable bot/ webscraper will identify itself with a User-Agent header ( AND honor a robots.txt if you want to direct its behavior) making it easy to identify. A malicious bot (that is not ...
user avatar
  • 5,264
1 vote
Accepted

Block website scraper in Haproxy

use hdr_reg acl blockedagent hdr_reg(user-agent) -i -f /etc/haproxy/badbots.lst Or remove ^ from badbots.lst ==== $ cat conf global debug defaults mode http frontend web bind *:80 ...
user avatar
1 vote

Suspected malicious activity by one of my site's users; any way to know for sure?

Is it safe to say this is a bot, beyond a shadow of a doubt? No. One could have multiple tabs of your site open, crash the browser, reopen the browser window with all tabs and cause this DOS-attack-...
user avatar
  • 3,713
1 vote

Amazon EC2 + S3 + Python + Scraping - The cheapest way of doing this?

You can interact with different instance via SQS. Its a queueing service. You can queue the input url to SQS. Each instance will take url from SQS sequentially. But SQS wont give same input to ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible