Hot answers tagged

88

Yes, they have the right to do so - you've created a public website, what makes you think they don't? You too, of course, have the right to stop them. You can ask them not to crawl your website with robots.txt or actively prevent them from accessing it with something like fail2ban. Alternatively, don't worry about it and continue on with your life. It's ...


46

There is legal precedent for this. Field v. Google Inc., 412 F. Supp. 2d 1106, (U.S. Dist. Ct. Nevada 2006). Google won a summary judgement based on several factors, most notably that the author did not utilize a robots.txt file in the metatags on his website, which would have prevented Google from crawling and caching pages the website owner did not want ...


21

No, they'll ban the public IP and everyone who is NAT'd to that IP will also be banned. Although at least At stack if we think we are going to ban a college or something like that we'll reach out to their abuse contact to get them to track the offender down and stop the issue.


14

Welcome to the internet :) How they found you: Chances are, brute force IP scanning. Just like their constant stream of vulnerability scanning on your host once they found it. To prevent in the future: While not totally avoidable, you can inhibit security tools like Fail2Ban on Apache or rate limits - or manually banning - or setting up ACL's It's very ...


12

Google's spiders are constantly crawling the web. They have multiple machines which crawl their massive index and add new pages to it all the time. Reasons it's fast: They have tons of machines doing the crawling at ridiculous speeds They have tons of bandwidth available They already have a giant index of pages to search so it saves time looking for new ...


11

Whether this behaviour is ethical or not isn't perfectly clear cut. The act of crawling a public site is, itself, not unethical (unless you've forbidden it explicitly using a robots.txt or other technological measures, and they're circumventing them). What they are doing is the rough equivalent of cold calling you, while announcing to the world that you ...


8

For the quick and dirty answer, scroll the bottom. Otherwise, read through my narrative to understand how I came up with those numbers. In 2008, Google released some numbers that might be of interest of you. At that time, Google's spiders were aware of over 1 trillion (that's 1,000,000,000,000) unique URLs. One thing to take note of is that not all of these ...


8

The robots.txt file needs to go in the top level directory of you webserver. If your main domain and each subdomain are on different vhosts then you can put it in the top level directory of each subdomain and include something like User-agent: * Disallow: / Where the robots.txt is located depends upon how you access a particular site. Given a URL like ...


6

The answer to your first question seems to be "maybe": What file types can Google index? Google can index the content of most types of pages and files. See the most common file types. But the link to common files types are all text. Even if you search for binary files like Windows Installers (.msi), you may get a link to a page containing the ...


5

These are harmless crap requests that every web server on the internet sees - most likely script kiddies looking for a web server that is grossly misconfigured and allows you to make proxy requests and use the CONNECT method. Your server seems appropriately configured to reject attempts to use the CONNECT method (Returns HTTP/400 - Bad Request), and I ...


5

Amazon EC2 is a hosting platform. They don't directly control what people host. If you block the whole *.amazonaws.com domain then you will stop access to any hosted service using EC2. Which is quite a lot these days.


4

User-Agents.org has a pretty large database of user agents/spiders etc. It seems to be updated reguarly ( last update was 2/28/2009 ). Data is availible through RSS/XML.


4

While in development you might not want that search engines will index your site just yet.


4

To answer your question the network you're abusing (Myspace in this example) is protecting itself by redirecting your attacks to a 3rd party website that can easily handle the traffic. An automated tool, likely something similar to snort, has detected your activity. All large networks engage in this sort of monitoring. The typical response is to just ...


4

From Apache's point of view, robots.txt is just an asset to be served. You can alter the content returned when robots.txt is requested by passing it through an output filter. If you want to append some text, you could define an external filter. Assuming that Apache is running on Unix-like operating system, the filter configuration could be ...


3

I found these notes interesting to pursue Get yourself a smart robots.txt and other robots.txt posts there A post on Google's Dirty Little Secret by someone troubled with google bots Google web crawlers


3

You can go to google and create an account with the webmaster tool and then you can control the crawl rate for each site. Go to Site Configuration::Settings::Crawl Rate. This won't let you schedule your sites in a certain order I don't believe, but you can at least slow it down for all of them.


3

The internet achive does index the web like you mentioned, but only preserves Websites, not documents as far as i know. They do keep older versions of sites indexed, so their need for space might be alot larger. In their FAQ they speak about 2 petabytes of required space for that task (http://www.archive.org/about/faqs.php#9) and about hundreds of linux ...


3

We use Cisco hardware-based firewalls rather than server software-based ones and they watch out for patterns of activity and block them for quite a while (30-90 days iirc). I'm sure other firewalls can do this but don't have experience. Basically what I'm saying is that if your firewall can use rules to look for abuse then you'll see the benefit over simply ...


3

wget -r will recursively get an entire website and save it all locally in the same structure.


3

Just check the webserver logs if the visitor's User-Agent request header matches/contains Googlebot. There are lot of webserver log analyzer tools, either free or payware. Most of them are also able to categorize bots. From them all I've had the best experience with Google Analytics.


3

The quick and dirty way is to go to google and run a search like: site:mydomain.com This example shows 232 known pages for fronde.com: http://i47.tinypic.com/j0h003.jpg That will return the number of pages that google is aware of on that site. You may need to adjust your google preferences to include all content types (Turn SafeSearch off) and click the '...


3

sudo apt-get install lynx-cur lynx --dump http://serverfault.com -listonly |head 1. http://serverfault.com/opensearch.xml 2. http://serverfault.com/feeds 3. http://stackexchange.com/ 4. http://serverfault.com/users/login 5. http://careers.serverfault.com/ 6. http://blog.serverfault.com/ 7. http://meta.serverfault.com/ 8. http://...


3

The language you know the best.


3

I'm not sure why enforcing compliance with robots.txt would be the job of a proxy: The crawler (robot) is supposed to pull robots.txt and follow the instructions contained in that file, so as long as the proxy returns the correct robots.txt data and the crawler Does The Right Thing with that data, and as long as the crawler supports using a proxy, you'll get ...


3

A webcrawler has bought our site down twice If a webcrawler can bring your site down then they've demonstrated that your site is very vulnerable to DOS. While yes, a quick fix is to block that webcrawler's access, it doesn't really provide you much protection against other web crawlers / DOS / high volumes of legitimate traffic. I agree with Bobby - where ...


3

Have you tried looking for url paths where that begin with /http ? if (req.url ~ "^/https?:") { error 404 "Not found" }


3

robots.txt doesn't block anything, it is up to the crawler whether it pays attention to robots.txt or ignores it. There's also no central list of web crawlers, since anyone can run one for any reason and they can appear as ordinary browsing traffic, claiming to come from an ordinary web browser. You can do basic referrer checks to block image hotlinking, ...


3

Have you searched google for <Your IP>:22? I'm sure you haven't published it anywhere as you say, but any old idiot can put up a link to anywhere that googlebot can notice. Have you had this IP block for a while? It seems much less likely, though not impossible, that Google is starting to do something about the 'dark web' that they've talked about ...



Only top voted, non community-wiki answers of a minimum length are eligible