Tag Info

Hot answers tagged

85

Yes, they have the right to do so - you've created a public website, what makes you think they don't? You too, of course, have the right to stop them. You can ask them not to crawl your website with robots.txt or actively prevent them from accessing it with something like fail2ban. Alternatively, don't worry about it and continue on with your life. It's ...


41

There is legal precedent for this. Field v. Google Inc., 412 F. Supp. 2d 1106, (U.S. Dist. Ct. Nevada 2006). Google won a summary judgement based on several factors, most notably that the author did not utilize a robots.txt file in the metatags on his website, which would have prevented Google from crawling and caching pages the website owner did not want ...


12

Whether this behaviour is ethical or not isn't perfectly clear cut. The act of crawling a public site is, itself, not unethical (unless you've forbidden it explicitly using a robots.txt or other technological measures, and they're circumventing them). What they are doing is the rough equivalent of cold calling you, while announcing to the world that you ...


12

Welcome to the internet :) How they found you: Chances are, brute force IP scanning. Just like their constant stream of vulnerability scanning on your host once they found it. To prevent in the future: While not totally avoidable, you can inhibit security tools like Fail2Ban on Apache or rate limits - or manually banning - or setting up ACL's It's very ...


12

Google's spiders are constantly crawling the web. They have multiple machines which crawl their massive index and add new pages to it all the time. Reasons it's fast: They have tons of machines doing the crawling at ridiculous speeds They have tons of bandwidth available They already have a giant index of pages to search so it saves time looking for new ...


8

For the quick and dirty answer, scroll the bottom. Otherwise, read through my narrative to understand how I came up with those numbers. In 2008, Google released some numbers that might be of interest of you. At that time, Google's spiders were aware of over 1 trillion (that's 1,000,000,000,000) unique URLs. One thing to take note of is that not all of these ...


8

The robots.txt file needs to go in the top level directory of you webserver. If your main domain and each subdomain are on different vhosts then you can put it in the top level directory of each subdomain and include something like User-agent: * Disallow: / Where the robots.txt is located depends upon how you access a particular site. Given a URL like ...


6

The answer to your first question seems to be "maybe": What file types can Google index? Google can index the content of most types of pages and files. See the most common file types. But the link to common files types are all text. Even if you search for binary files like Windows Installers (.msi), you may get a link to a page containing the ...


5

Amazon EC2 is a hosting platform. They don't directly control what people host. If you block the whole *.amazonaws.com domain then you will stop access to any hosted service using EC2. Which is quite a lot these days.


4

While in development you might not want that search engines will index your site just yet.


4

User-Agents.org has a pretty large database of user agents/spiders etc. It seems to be updated reguarly ( last update was 2/28/2009 ). Data is availible through RSS/XML.


4

From Apache's point of view, robots.txt is just an asset to be served. You can alter the content returned when robots.txt is requested by passing it through an output filter. If you want to append some text, you could define an external filter. Assuming that Apache is running on Unix-like operating system, the filter configuration could be ...


4

To answer your question the network you're abusing (Myspace in this example) is protecting itself by redirecting your attacks to a 3rd party website that can easily handle the traffic. An automated tool, likely something similar to snort, has detected your activity. All large networks engage in this sort of monitoring. The typical response is to just ...


4

These are harmless crap requests that every web server on the internet sees - most likely script kiddies looking for a web server that is grossly misconfigured and allows you to make proxy requests and use the CONNECT method. Your server seems appropriately configured to reject attempts to use the CONNECT method (Returns HTTP/400 - Bad Request), and I ...


3

sudo apt-get install lynx-cur lynx --dump http://serverfault.com -listonly |head 1. http://serverfault.com/opensearch.xml 2. http://serverfault.com/feeds 3. http://stackexchange.com/ 4. http://serverfault.com/users/login 5. http://careers.serverfault.com/ 6. http://blog.serverfault.com/ 7. http://meta.serverfault.com/ 8. ...


3

The language you know the best.


3

A webcrawler has bought our site down twice If a webcrawler can bring your site down then they've demonstrated that your site is very vulnerable to DOS. While yes, a quick fix is to block that webcrawler's access, it doesn't really provide you much protection against other web crawlers / DOS / high volumes of legitimate traffic. I agree with Bobby - ...


3

Have you tried looking for url paths where that begin with /http ? if (req.url ~ "^/https?:") { error 404 "Not found" }


3

What are you really trying to accomplish? You're simply not going to be able to do this via HTTP. Given the absence of vulnerabilities in the HTTP server, you're going to get what the content provider publishes unless you already know direct paths. The only option here is a content crawler. With that fact in hand your other option is to index the site at ...


3

robots.txt doesn't block anything, it is up to the crawler whether it pays attention to robots.txt or ignores it. There's also no central list of web crawlers, since anyone can run one for any reason and they can appear as ordinary browsing traffic, claiming to come from an ordinary web browser. You can do basic referrer checks to block image hotlinking, ...


3

The quick and dirty way is to go to google and run a search like: site:mydomain.com This example shows 232 known pages for fronde.com: http://i47.tinypic.com/j0h003.jpg That will return the number of pages that google is aware of on that site. You may need to adjust your google preferences to include all content types (Turn SafeSearch off) and click the ...


3

Just check the webserver logs if the visitor's User-Agent request header matches/contains Googlebot. There are lot of webserver log analyzer tools, either free or payware. Most of them are also able to categorize bots. From them all I've had the best experience with Google Analytics.


3

We use Cisco hardware-based firewalls rather than server software-based ones and they watch out for patterns of activity and block them for quite a while (30-90 days iirc). I'm sure other firewalls can do this but don't have experience. Basically what I'm saying is that if your firewall can use rules to look for abuse then you'll see the benefit over simply ...


3

The internet achive does index the web like you mentioned, but only preserves Websites, not documents as far as i know. They do keep older versions of sites indexed, so their need for space might be alot larger. In their FAQ they speak about 2 petabytes of required space for that task (http://www.archive.org/about/faqs.php#9) and about hundreds of linux ...


3

wget -r will recursively get an entire website and save it all locally in the same structure.


2

Generally you would want to ban spiders from certain sections of your site or pages that you do not want to appear in search results, or offer nothing for a search engine - such as a feedback form, script directories, image directories etc... Sometimes spiders can hit your site at a high rate so blocking certain crawlers can help server load if they are ...


2

Since you have access to iptables, I will assume you have a root access on the system anyway. In this case, I would suggest instlling Fail2Ban which will just block an IP (for a certain time you decide) if they try to abuse a service (HTTP, DNS, Mail, SSH ..etc) by hitting the service port as N times within X period. (all users decided.) I am using that on ...


2

How about linkchecker? Another good one is oldie but goldie ht://Check.


2

My work actively blocks Googlebot and other crawlers on servers when the load jumps; I certainly don't agree with it, and in my opinion, it's a sign of something far worse with the server in general when we have to block it, though we are hosting thousands of many different websites; you, on the other hand, seem to have your own server. What this leads me ...



Only top voted, non community-wiki answers of a minimum length are eligible