Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
91

Yes, they have the right to do so - you've created a public website, what makes you think they don't? You too, of course, have the right to stop them. You can ask them not to crawl your website with robots.txt or actively prevent them from accessing it with something like fail2ban. Alternatively, don't worry about it and continue on with your life. It's ...


49

There is legal precedent for this. Field v. Google Inc., 412 F. Supp. 2d 1106, (U.S. Dist. Ct. Nevada 2006). Google won a summary judgement based on several factors, most notably that the author did not utilize a robots.txt file in the metatags on his website, which would have prevented Google from crawling and caching pages the website owner did not want ...


21

No, they'll ban the public IP and everyone who is NAT'd to that IP will also be banned. Although at least At stack if we think we are going to ban a college or something like that we'll reach out to their abuse contact to get them to track the offender down and stop the issue.


14

Welcome to the internet :) How they found you: Chances are, brute force IP scanning. Just like their constant stream of vulnerability scanning on your host once they found it. To prevent in the future: While not totally avoidable, you can inhibit security tools like Fail2Ban on Apache or rate limits - or manually banning - or setting up ACL's It's very ...


11

Whether this behaviour is ethical or not isn't perfectly clear cut. The act of crawling a public site is, itself, not unethical (unless you've forbidden it explicitly using a robots.txt or other technological measures, and they're circumventing them). What they are doing is the rough equivalent of cold calling you, while announcing to the world that you ...


9

The robots.txt file needs to go in the top level directory of you webserver. If your main domain and each subdomain are on different vhosts then you can put it in the top level directory of each subdomain and include something like User-agent: * Disallow: / Where the robots.txt is located depends upon how you access a particular site. Given a URL like ...


7

The answer to your first question seems to be "maybe": What file types can Google index? Google can index the content of most types of pages and files. See the most common file types. But the link to common files types are all text. Even if you search for binary files like Windows Installers (.msi), you may get a link to a page containing the file ...


5

From Apache's point of view, robots.txt is just an asset to be served. You can alter the content returned when robots.txt is requested by passing it through an output filter. If you want to append some text, you could define an external filter. Assuming that Apache is running on Unix-like operating system, the filter configuration could be ...


5

Amazon EC2 is a hosting platform. They don't directly control what people host. If you block the whole *.amazonaws.com domain then you will stop access to any hosted service using EC2. Which is quite a lot these days.


5

These are harmless crap requests that every web server on the internet sees - most likely script kiddies looking for a web server that is grossly misconfigured and allows you to make proxy requests and use the CONNECT method. Your server seems appropriately configured to reject attempts to use the CONNECT method (Returns HTTP/400 - Bad Request), and I ...


4

To answer your question the network you're abusing (Myspace in this example) is protecting itself by redirecting your attacks to a 3rd party website that can easily handle the traffic. An automated tool, likely something similar to snort, has detected your activity. All large networks engage in this sort of monitoring. The typical response is to just ...


3

robots.txt doesn't block anything, it is up to the crawler whether it pays attention to robots.txt or ignores it. There's also no central list of web crawlers, since anyone can run one for any reason and they can appear as ordinary browsing traffic, claiming to come from an ordinary web browser. You can do basic referrer checks to block image hotlinking, ...


3

Have you tried looking for url paths where that begin with /http ? if (req.url ~ "^/https?:") { error 404 "Not found" }


3

I'm not sure why enforcing compliance with robots.txt would be the job of a proxy: The crawler (robot) is supposed to pull robots.txt and follow the instructions contained in that file, so as long as the proxy returns the correct robots.txt data and the crawler Does The Right Thing with that data, and as long as the crawler supports using a proxy, you'll get ...


3

The language you know the best.


3

sudo apt-get install lynx-cur lynx --dump http://serverfault.com -listonly |head 1. http://serverfault.com/opensearch.xml 2. http://serverfault.com/feeds 3. http://stackexchange.com/ 4. http://serverfault.com/users/login 5. http://careers.serverfault.com/ 6. http://blog.serverfault.com/ 7. http://meta.serverfault.com/ 8. http://...


3

A webcrawler has bought our site down twice If a webcrawler can bring your site down then they've demonstrated that your site is very vulnerable to DOS. While yes, a quick fix is to block that webcrawler's access, it doesn't really provide you much protection against other web crawlers / DOS / high volumes of legitimate traffic. I agree with Bobby - where ...


3

Have you searched google for <Your IP>:22? I'm sure you haven't published it anywhere as you say, but any old idiot can put up a link to anywhere that googlebot can notice. Have you had this IP block for a while? It seems much less likely, though not impossible, that Google is starting to do something about the 'dark web' that they've talked about ...


3

What are you really trying to accomplish? You're simply not going to be able to do this via HTTP. Given the absence of vulnerabilities in the HTTP server, you're going to get what the content provider publishes unless you already know direct paths. The only option here is a content crawler. With that fact in hand your other option is to index the site at ...


3

This is, perhaps, a bit too obvious of a statement, but the blocking behavior has to be based on the information that the remote server is privy to. That would include: The source IP address The source TCP port, which should be ephemeral and changing on each request There's probably some passive fingerprinting of the client's IP stack that's possible The ...


3

I would search for some of the excellent answers on https://webmasters.stackexchange.com/ but simply put most people use log analyzers to generate visitor stats. Most log analyzers convert those referers to active links and too many people don't shield their stats pages, so when indexed your spammer gets pageRank At least that was the reason more than a ...


3

As long as it follows robots.txt you can throttle the requests User-agent: * Crawl-delay: 10


3

Would you consider an entropy-gathering daemon like haveged? See: Is it appropriate to use haveged as a source of entropy on virtual machines? Or a low cost hardware device?


2

My work actively blocks Googlebot and other crawlers on servers when the load jumps; I certainly don't agree with it, and in my opinion, it's a sign of something far worse with the server in general when we have to block it, though we are hosting thousands of many different websites; you, on the other hand, seem to have your own server. What this leads me ...


2

Since you have access to iptables, I will assume you have a root access on the system anyway. In this case, I would suggest instlling Fail2Ban which will just block an IP (for a certain time you decide) if they try to abuse a service (HTTP, DNS, Mail, SSH ..etc) by hitting the service port as N times within X period. (all users decided.) I am using that on ...


2

Instead of taking a guess, why not check the access_logs to see what the User Agent or the requesting host is? That way you can even tell how much bandwidth Google (or other crawlers) are taking, by adding the data traffic per request.


2

Have you considered implementing caching of your 404s? In order to block these with regexes, you'd need to keep your VCL and your backend in sync to avoid blocking valid URLs. If you just cache 404s for a significant amount of time, you will still need to load from the backend once per URL but subsequent requests will get a cached 404 page. You can do a ...


2

You need to put robots.txt in you root directory The Disallow rules are not domian/sub-domain specific and will apply to all urls For example: Lets assume you are using a sub.mydomain.com and mydomain.com (both are linked to the same ftp folder). For this setup, if you set a Disallow: /admin/ rule then all URL sub.mydomain.com/admin/ and in mydomain....


2

You have to put it in your root directory, otherwise it won't be found.


2

Try ls passing the -R switch. It lists subdirectories. Here is an example: ls -R /path/to/whatever > folders.txt


Only top voted, non community-wiki answers of a minimum length are eligible