Tag Info

Hot answers tagged

86

Yes, they have the right to do so - you've created a public website, what makes you think they don't? You too, of course, have the right to stop them. You can ask them not to crawl your website with robots.txt or actively prevent them from accessing it with something like fail2ban. Alternatively, don't worry about it and continue on with your life. It's ...


44

There is legal precedent for this. Field v. Google Inc., 412 F. Supp. 2d 1106, (U.S. Dist. Ct. Nevada 2006). Google won a summary judgement based on several factors, most notably that the author did not utilize a robots.txt file in the metatags on his website, which would have prevented Google from crawling and caching pages the website owner did not want ...


12

Google's spiders are constantly crawling the web. They have multiple machines which crawl their massive index and add new pages to it all the time. Reasons it's fast: They have tons of machines doing the crawling at ridiculous speeds They have tons of bandwidth available They already have a giant index of pages to search so it saves time looking for new ...


12

Welcome to the internet :) How they found you: Chances are, brute force IP scanning. Just like their constant stream of vulnerability scanning on your host once they found it. To prevent in the future: While not totally avoidable, you can inhibit security tools like Fail2Ban on Apache or rate limits - or manually banning - or setting up ACL's It's very ...


11

Whether this behaviour is ethical or not isn't perfectly clear cut. The act of crawling a public site is, itself, not unethical (unless you've forbidden it explicitly using a robots.txt or other technological measures, and they're circumventing them). What they are doing is the rough equivalent of cold calling you, while announcing to the world that you ...


8

For the quick and dirty answer, scroll the bottom. Otherwise, read through my narrative to understand how I came up with those numbers. In 2008, Google released some numbers that might be of interest of you. At that time, Google's spiders were aware of over 1 trillion (that's 1,000,000,000,000) unique URLs. One thing to take note of is that not all of these ...


8

The robots.txt file needs to go in the top level directory of you webserver. If your main domain and each subdomain are on different vhosts then you can put it in the top level directory of each subdomain and include something like User-agent: * Disallow: / Where the robots.txt is located depends upon how you access a particular site. Given a URL like ...


6

The answer to your first question seems to be "maybe": What file types can Google index? Google can index the content of most types of pages and files. See the most common file types. But the link to common files types are all text. Even if you search for binary files like Windows Installers (.msi), you may get a link to a page containing the ...


5

Amazon EC2 is a hosting platform. They don't directly control what people host. If you block the whole *.amazonaws.com domain then you will stop access to any hosted service using EC2. Which is quite a lot these days.


4

User-Agents.org has a pretty large database of user agents/spiders etc. It seems to be updated reguarly ( last update was 2/28/2009 ). Data is availible through RSS/XML.


4

While in development you might not want that search engines will index your site just yet.


4

These are harmless crap requests that every web server on the internet sees - most likely script kiddies looking for a web server that is grossly misconfigured and allows you to make proxy requests and use the CONNECT method. Your server seems appropriately configured to reject attempts to use the CONNECT method (Returns HTTP/400 - Bad Request), and I ...


4

To answer your question the network you're abusing (Myspace in this example) is protecting itself by redirecting your attacks to a 3rd party website that can easily handle the traffic. An automated tool, likely something similar to snort, has detected your activity. All large networks engage in this sort of monitoring. The typical response is to just ...


4

From Apache's point of view, robots.txt is just an asset to be served. You can alter the content returned when robots.txt is requested by passing it through an output filter. If you want to append some text, you could define an external filter. Assuming that Apache is running on Unix-like operating system, the filter configuration could be ...


3

The internet achive does index the web like you mentioned, but only preserves Websites, not documents as far as i know. They do keep older versions of sites indexed, so their need for space might be alot larger. In their FAQ they speak about 2 petabytes of required space for that task (http://www.archive.org/about/faqs.php#9) and about hundreds of linux ...


3

We use Cisco hardware-based firewalls rather than server software-based ones and they watch out for patterns of activity and block them for quite a while (30-90 days iirc). I'm sure other firewalls can do this but don't have experience. Basically what I'm saying is that if your firewall can use rules to look for abuse then you'll see the benefit over simply ...


3

wget -r will recursively get an entire website and save it all locally in the same structure.


3

Just check the webserver logs if the visitor's User-Agent request header matches/contains Googlebot. There are lot of webserver log analyzer tools, either free or payware. Most of them are also able to categorize bots. From them all I've had the best experience with Google Analytics.


3

The quick and dirty way is to go to google and run a search like: site:mydomain.com This example shows 232 known pages for fronde.com: http://i47.tinypic.com/j0h003.jpg That will return the number of pages that google is aware of on that site. You may need to adjust your google preferences to include all content types (Turn SafeSearch off) and click the ...


3

sudo apt-get install lynx-cur lynx --dump http://serverfault.com -listonly |head 1. http://serverfault.com/opensearch.xml 2. http://serverfault.com/feeds 3. http://stackexchange.com/ 4. http://serverfault.com/users/login 5. http://careers.serverfault.com/ 6. http://blog.serverfault.com/ 7. http://meta.serverfault.com/ 8. ...


3

The language you know the best.


3

A webcrawler has bought our site down twice If a webcrawler can bring your site down then they've demonstrated that your site is very vulnerable to DOS. While yes, a quick fix is to block that webcrawler's access, it doesn't really provide you much protection against other web crawlers / DOS / high volumes of legitimate traffic. I agree with Bobby - ...


3

Have you tried looking for url paths where that begin with /http ? if (req.url ~ "^/https?:") { error 404 "Not found" }


3

robots.txt doesn't block anything, it is up to the crawler whether it pays attention to robots.txt or ignores it. There's also no central list of web crawlers, since anyone can run one for any reason and they can appear as ordinary browsing traffic, claiming to come from an ordinary web browser. You can do basic referrer checks to block image hotlinking, ...


3

What are you really trying to accomplish? You're simply not going to be able to do this via HTTP. Given the absence of vulnerabilities in the HTTP server, you're going to get what the content provider publishes unless you already know direct paths. The only option here is a content crawler. With that fact in hand your other option is to index the site at ...


3

This is, perhaps, a bit too obvious of a statement, but the blocking behavior has to be based on the information that the remote server is privy to. That would include: The source IP address The source TCP port, which should be ephemeral and changing on each request There's probably some passive fingerprinting of the client's IP stack that's possible The ...


2

Calls to executables (like whoami) in Java require making an entire copy of the Java process first. You will want to drop your maximum heap size (-Xmx256m) to where you may have two copies in RAM at the same time.


2

Read the output: Cannot run program "whoami": java.io.IOException: error=12, Cannot allocate memory Looks like you don't have enough RAM or no swap file/partition.


2

Signup for Google Webmaster Central, aka Webmasters Tools, verify your site, and wait for the stats to appear.


2

If the scrapers are BOTS and not humans, you could try creating a honeypot directory that they would crawl to and be blocked (by IP address) automatically via a "default page" script in that directory. Humans could easily unblock themselves, but it would thwart bots as they would get a 403 "not authorized" error on any further access. I use a technique like ...



Only top voted, non community-wiki answers of a minimum length are eligible