Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
8

Not that I know of, and it could change at any time at the discretion of the bot operators. Google offers some specific guidance and explanation on this: The problem with that is that if/when the IP ranges of our crawlers change, not everyone will know to check. In fact, the crawl team migrated Googlebot IPs a couple years ago and it was a real hassle ...


7

The basic premise of your setup seems fine, however, there are a few items that you may want to factor in. Firstly, EC2 network (and I/O) bandwidth is dependant on instance type. If you are hoping to use t1.micro instances do not expect 'super fast internet connectivity' - even with an m1.small, you may not see the performance you are looking for. Also, ...


7

Working on the assumption that download time (and therefore bandwidth usage) is your limiting factor, I would make the following suggestions: Firstly, choose m1.large instances. Of the three 'levels' of I/O performance (which includes bandwidth), the m1.large and m1.xlarge instances both offer 'high' I/O performance. Since your task is not CPU bound, the ...


4

We tried to do something similar, and here is my 5 cents: Get 2-3 cheap unmetered servers, e.g. don't pay for the bandwidth. Use python with asyncore. Asyncore is the old way to do things, but we found it works faster than any other method. Downside is that DNS lookup is blocking, i.e. not "parallel". Using asyncore we managed to scrape 1M URL's for 40 min, ...


2

Yes - a bot can do whatever it wants with the resources that it's gathering, which could certainly include interpreting javascript. Take, for instance, Google's new little thumbnail display of pages in search results; many sites rely on javascript for basic display functionality, so interpretation of js would be a must for that to come up with previews that ...


2

Monitoring AWS has a service called CloudWatch that can monitor the health of your system. http://aws.amazon.com/cloudwatch/ You could script things like "killing programs and sending emails" yourself based off cloudwatch metrics. Of course there are plenty of other tools to monitor the health of your system such as: Server Density http://www....


2

You can throttle connections with the following http://wiki.nginx.org/NginxHttpLimitReqModule


1

There's no list of IP addresses for "good" search engine bots that I know of, and if there were it would be horribly out of date pretty quickly, as you've already discovered. One thing you can do is to create a bot trap. This is simple in theory: You create a page that is linked to in your web site but hidden from normal users (e.g. via CSS tricks) and then ...


1

use hdr_reg acl blockedagent hdr_reg(user-agent) -i -f /etc/haproxy/badbots.lst Or remove ^ from badbots.lst ==== $ cat conf global debug defaults mode http frontend web bind *:80 acl blockedagent hdr_reg(user-agent) -i -f badbots.lst1 http-request deny if blockedagent default_backend asdf backend asdf server a 127.0.0.1:8000 ...


1

Is it safe to say this is a bot, beyond a shadow of a doubt? No. One could have multiple tabs of your site open, crash the browser, reopen the browser window with all tabs and cause this DOS-attack-like fingerprint. If so, is there any possible way to find out the specific script, or is that a long shot? I don't see any data that would precisely ...


1

Also you can use ngx_http_limit_conn_module


1

Use Arno's Iptables scripts with enabled IDS plugin. It will do the job for you. Link


1

Actually, it seems like you're looking for an "easy and brute" solution and that could be something similar to: #!/bin/bash partition="/dev/sda1"; minRAM=512 minDisk=1024 maxSwapUsed=1024 freeMem=$(free -m | grep Mem | awk {'print $4'}); usedSwap=$(free -m | grep Swap | awk {'print $3'}); freeDisk=$(df -m | grep $partition | awk {'print $4'}); echo "RAM ...


1

The Monit tool can do what you need -- monitor RAM and disk usage, kill offending process and alert you.


1

It's not a complete out of the box solution, but Evan Anderson released a script here on Server Fault that is similar to fail2ban for *nix - basically after a set threshold, it makes a firewall block entry. You can find it here. Like I said, it's not a straight plug in solution, but if you don't find anything else you should be able to manipulate it to ...


1

Not sure if this is true, but possibly Apache only logs requests if they are valid http requests? So if I were to create a whole bunch socket connections to your apache server it would cause new apache processes to spawn, which is what you are looking at in your graph, but if I never send a valid http request and disconnect - or apache times out the ...


1

I'm not really sure fail2ban is the right tool here; you might want to look at something like mod_security (http://www.modsecurity.org/). You'll be able to track requests from a session or ip context, define rules to describe suspect traffic, and then deny/slow it accordingly. EDIT: You didn't specify, so I'm just assuming that you're using Apache.


1

For what it's worth, annoying but simple ways of getting the video include looking through your browser's cache (this may be easier if you clear the cache before visiting the page and loading the video) or running a packet sniffer while you load the video to find out the URL of the video. Once you have the video, you can use your own flash player, or peek at ...


Only top voted, non community-wiki answers of a minimum length are eligible