Tag Info

Hot answers tagged

5

When you really start to delve into logs, you'll find that a huge number of robots do header spoofing; most of them spoofing as IE (some of them unsuccessfully; typos get your agent string spotted fast!). There's an interesting EFF experiment that looks at uniquely identifying users with the data presented by the browser: Panopticlick. Gathering more info ...


4

[reverse DNS result]: you've been identified as a robot operating in violation of the guidelines posted at arxiv.org. If this determination is in error, please report to www-admin@arxiv.org so your problem can be investigated. Scanning, Initialized: 10 minutes to Trinity... 9 minutes to Trinity... 8 minutes to Trinity... 7 minutes to Trinity... 6 ...


4

Perhaps something on your site is being shared around on Facebook. Facebook makes requests to get little image and text previews before users post a link to their wall. Alternatively, if Facebook tries to crawl your site and fails for any reason they might retry. Go to this page: ...


3

sudo apt-get install lynx-cur lynx --dump http://serverfault.com -listonly |head 1. http://serverfault.com/opensearch.xml 2. http://serverfault.com/feeds 3. http://stackexchange.com/ 4. http://serverfault.com/users/login 5. http://careers.serverfault.com/ 6. http://blog.serverfault.com/ 7. http://meta.serverfault.com/ 8. ...


3

You can go to google and create an account with the webmaster tool and then you can control the crawl rate for each site. Go to Site Configuration::Settings::Crawl Rate. This won't let you schedule your sites in a certain order I don't believe, but you can at least slow it down for all of them.


3

Obviously this is difficult to answer empirically given we don't know what your service/application is or how it behaves - this number of connections could be by design for instance. But the best way to understand this issue is to benchmark typical site access from a single known IP, watch what happens during the 'user story' as you make your way through ...


2

I found these notes interesting to pursue Get yourself a smart robots.txt and other robots.txt posts there A post on Google's Dirty Little Secret by someone troubled with google bots Google web crawlers


2

Those types of services just scan entire network ranges for common proxy ports. Once they find one, they test to make sure it's actually "open", and post it to the site. I'm not sure how intelligent the bots are, but they may scan known "service provider" networks more often then your random home IP.


2

I notice that you do not have an answer yet (at least you did not when I posted this on Security SE), so here's my two bits. A single browser (run by a human) should never send in that many connections. However, it is possible that many users in the same network on the same site could cause the number of concurrent connections to multiply. If they are on ...


2

You will need to dig deeper in the what is going on to see if the traffic is legitimate. Legit Traffic Legitimate bots can be identified by their user agents. Checking the IP's owner or reverse DNS can often help pinpoint spoofed user agent data. I have seen Googlebot, Yahoo's Slurp and other spiders create 40+ connections to a system. Note that ...


1

You should create a robot.txt file if you don't want bots/spiders/etc to be crawling certain parts of your site.


1

This is the robot that extract objects from site when somebody post a link. For more information read http://www.facebook.com/externalhit_uatext.php. Social signals are making more and more of an impact in search results.visits from this bot, as it means that users are being directed to your page from a face book link. Social signals are great, and I ...


1

If you run BGP you could simply rate-limit AS15169 (AS-GOOGLE), but doing it by hand is likely to be far too error-prone.


1

I think sphider is what you are looking for - we had ok results with it before. Plus it can index pdfs and docs, which is very useful http://www.sphider.eu/


1

Google Analytics anyone? Free, Javascript based, so no need for server side languages. In depth coverage of hits, bots too.


1

Here is a pretty good list of known bot user-agents. I am not familiar with any free web log analytic tools written in .NET, but you could easily parse the logs yourself if all you are looking for are bots. http://www.user-agents.org



Only top voted, non community-wiki answers of a minimum length are eligible