Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [web-crawler]

A web-crawler (also known as a web-spider) traverses the webpages of the internet by following the links of urls contained within each webpage. There is usually an initial seed of URLs from which the crawler is given to initialize its crawl.

-2
votes
0answers
23 views

Allowed for crawling? [closed]

Good morning everyone, I recently get into crawling for research purposes and I discovered that a crawler is limited by the robots.txt file of a site. Unfortunately I'm not sure how to interpret what'...
-4
votes
2answers
45 views

what ip will logged in a website if I access a website via another website through my PC? [closed]

If http://example2.com sends cURL connection to a website called http://example1.com. If I access http://example2.com from my pc to see the content of http://example1.com, than would http://example1....
0
votes
0answers
13 views

More efficient sitemap to handle google crawling

My server running apache is getting slammed by googlebot. I have sitemap index which points to several sitemaps with 50000 entries each. Google is requesting the sitemaps so aggressively (several ...
1
vote
0answers
31 views

HTTrack stores extensionless pages with a .html appended

I'd like to mirror an old site of mine to local files. I've used httrack for this in the past, but I'm having a problem this time that I really thought I figured out before, but can't seem to now. ...
-1
votes
2answers
556 views

How to disallow crawling for all subdomains using my main domain's physical robots.txt file [closed]

I have multiple physical sub-domains and I don't want to change any robots.txt file of any of that sub-domains. Is there any way to disallow all the sub-domains from my main domain's physical robots....
0
votes
0answers
59 views

Strange Google Behavior with indexing SSL Mismatch content

Here is a strange one for you. We have a server with multiple VHOSTS that include both SSL and Non-SSL domains. Domain1 is SSL enabled, while Domain2 doesn't have SSL. Since all these domains are ...
0
votes
1answer
221 views

Yandex/Google Bot Spam

I recently logged into a vps I have (with vultr, if that is of any concern). To find a large amount of nginx logs and higher than expected load average. This server is doing effectively nothing, and ...
-1
votes
1answer
68 views

40.96.18.165 keeps visiting my web server

Something/someone from 40.96.18.165 has been hitting my web server exactly eight times a day everyday since Feb 5, 2017. The user agent used is Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0). ...
0
votes
1answer
159 views

How many requests can a router handle?

I made a script to scan a file which contains a portion of ipv4 addresses (about 50 million), it attempts to connect to the website using OpenSSL and extract a small piece of it and writes it into a ...
0
votes
1answer
117 views

Bots/crawlers adding numbers to GET parameters

I've got some errors showing up in my site logs where some bots are trying to access URLs with strange GET params. # normal url example.com?foo=123456 # odd url triggering integer error by bots ...
2
votes
0answers
28 views

Barracuda.com and crawling / pinger services causing unusual load on web servers

I recently received a large number of hits on my home page from 64.235.153.8. It revolves to barracuda.com I know Barracuda as an enterprise class spam detection/prevention solution. Do they also ...
0
votes
1answer
936 views

Bash script - wait for all xargs processes to be finished

I have written a small bash script for crawling an XML sitemap of URLs. It retrieves 5 URLs in parallel using xargs. Now I want an E-Mail to be sent when all URLs have been crawled, so it has to wait ...
1
vote
0answers
1k views

Block Bad Bots in Nginx for Multiple Sites

I need to block a bunch of robots from crawling a few hundred sites hosted on a Nginx web server running on a Ubuntu 16.04 machine. I've found a fairly simple example here (important part of the ...
-1
votes
0answers
42 views

Is offering the contents of a third party web site offline violating the law? [closed]

I have developed a nice little app that crawls a bunch of newspaper web sites and makes their latest content available on my phone offline. It's basically a Pocket app that saves contents ...
12
votes
3answers
3k views

How do sites detect bots behind proxies or company networks

How do large sites (e.g. Wikipedia) deal with bots that are behind other IP masker? For instance, in my university, everybody searches Wikipedia, giving it a significant load. But, as far as I know, ...
1
vote
1answer
49 views

Suspected malicious activity by one of my site's users; any way to know for sure?

In the course of about 2 hours, a logged in user on my website accessed roughly 1,600 pages in a way that looks suspiciously similar to a bot. I am concerned because users must purchase access to the ...
2
votes
1answer
356 views

Q: Strange web traffic - Is this an attack?

I recently noticed some strange traffic in my nginx access logs. I'm not sure if these indicate an attack, a mistake, or something else. I've started sending these to HTTP 444, so these logs will ...
3
votes
1answer
781 views

Baidu Spider causing 3Gb of traffic a day - but I do business in China

I'm in a difficult situation, the Baidu spider is hitting my site causing about 3Gb a day worth of bandwidth. At the same time I do business in China so don't want to just block it. Has anyone else ...
0
votes
1answer
185 views

Moved website to new server - updated DNS - web crawlers still hitting old site by IP

About ten days ago I moved a site - mostly a Joomla discussion board - to a new server at a different IP address. During a brief scheduled downtime I replicated the content over and completed DNS ...
-3
votes
2answers
77 views

Restricting Access from BOTS

I would like to protect my server from too many hits from Bots. Considering a scenario, where in a server (physical) located in a private network and hitting my server continuously. Do i have a ...
0
votes
1answer
582 views

Nginx log shows suspicious directory access!!! How to block them?

On my Nginx log recently i have noticed 100's entries like this where a directory search was executed with error, because those directory does not exist on my webserver. now, how can I block them once ...
2
votes
2answers
49 views

What's the purpose of spammy HTTP referers?

In the logs of my website, there's a lot of visits with a HTTP referer set to spam-like websites (usually Russian sites, I've noticed). I assume what they're doing is just using a web crawler to visit ...
1
vote
0answers
463 views

What are the symptoms of an overloaded webserver

I'm maintaining some web crawlers. I want to improve our load/throttling system to be more intelligent. Of cause I look at response codes, and throttle up or down based on that. I would though like ...
0
votes
0answers
369 views

How to make Google crawl my site using IPv6 address when my domain name has both IPv4 and IPv6 addresses?

My domain name has both IPv4 and IPv6 addresses assigned. IPv4 connection to Google can't be available all the time due to restrictions of my campus network, but IPv6 is available all the time. ...
0
votes
1answer
617 views

Is it possible to block HTTP traffic from specific machines?

I have some web crawlers, and a specific website seems to be blocking traffic temporarily after some time. The thing is, even though all clients have the same external IP address (they access the ...
0
votes
2answers
800 views

Switching between multiple authentication types on same URL

I have a secure SSO site that uses Shibboleth authentication and SAML identity provider. I need to allow a Google Search Appliance crawler to come index the URL's. I have a requirement to change on ...
30
votes
4answers
9k views

Does a company have implied right to crawl my website?

I have found out that McAfee SiteAdvisor has reported my website as "may be having security issues". I care little about whatever McAfee thinks of my website (I can secure it myself and if not, ...
-9
votes
2answers
936 views

Get all urls of a website [closed]

I wanna build a tool which scans a website for all urls, but not the urls within the page but of the site self, but I don't know how. Could anyone give me an example how I can start? Example: www....
0
votes
1answer
523 views

Tool or website to check links

We supply Magento and Typo3 installations to customers. To improve QA we wanted to use an automatic link checker to check for broken and/or outdated links - automatically. We want to check all links ...
3
votes
1answer
239 views

Why is googlebot requesting robots.txt from my SSH server?

I run ossec on my server and periodically I receive a warning like this: Received From: myserver->/var/log/auth.log Rule: 5701 fired (level 8) -> "Possible attack on the ssh server (or version ...
-3
votes
1answer
431 views

is there any web crawler can get access to users region for download? [closed]

Actually I am using Httrack as a web crawler, can it use my credentials to access members area and download the zip files because they are restricted from public access. Thank you in advance. ...
1
vote
2answers
889 views

Protection against scrapping with nginx

This morning we had a crawler going nuts on our server hitting our site almost 100 times per second. We'd like to add a protection for this. I guess I'' have to use HttpLimitReqModule but I don't ...
0
votes
1answer
53 views

Referrer in access.log is a directory

It seems that the referrer on the following log is a folder. 112.200.208.5 - - [29/Jul/2013:20:43:14 +0800] "GET /sites/default/files/download/argie/pos-code.zip HTTP/1.1" 206 294677 "http://www....
1
vote
1answer
600 views

How to Block Web Crawler from Downloading File

Is it possible to block web crawler from downloading files (like zip file) in my server? I supposed to create a PHP script using cookies to track visitors specially web crawlers to login/register ...
1
vote
0answers
897 views

How to Exclude Log Using Fail2Ban logpath with Wildcard Settings

I'm using wildcard in the logpath value as shown below: [http-get-dos] enabled = true filter = http-get-dos logpath = /var/log/ispconfig/httpd/*/access.log maxretry = 250 findtime = 300 #ban for 10 ...
1
vote
1answer
1k views

How to ban web crawler using fail2ban

I am using nginx and I am always hit by web crawler if I am correct. I tried to configure fail2ban but the IP address cannot be detected by fail2ban. The reason that it is not detected because it ...
0
votes
1answer
206 views

Is there a chance to block images spiders / bots on dedicated servers without using robots.txt or .htaccess?

We know that we can block certain spiders from crawling websites pages using robots.txt or .htaccess or maybe via the Apache configuration File httpd.conf. But that requires to edit may be a large ...
2
votes
2answers
89 views

Strange “GET /api/levels/ ” and “GET /play/” requests in logs

I've setup new Amazon EC2 instance. In a day or two started to get strange "GET" requests from the "google bot-like" IP's (eg 66.249.76.84, 66.249.74.152) about one in 10 seconds (some examples): 66....
-2
votes
1answer
30 views

How to fix the correct html file as homepage? [closed]

A search for my domain shows defaulthomepage along with html file that my hosting service, using plesk 9.5.4, provides as a placeholder. I have since deleted the old index file and added a new index....
0
votes
1answer
347 views

How do I scan my folders for a website? Like a crawler?

I'd like to scan all the url's on my website as well as get the files in them, but the thing is, there are too many for me to do this manually so how would I do this? I'd like it formatted anyway as ...
1
vote
1answer
270 views

Googlebot repeatedly looks for files that aren't on my server

I'm hosting a site for a volunteer organization. I've moved the site to WordPress, but it wasn't always that way. I suspect at one point it was hacked badly. My Apache error log file has grown to ...
1
vote
2answers
1k views

How to create a global robots.txt that gets appended to each domain's own robots.txt on Apache?

I know can create ONE robots.txt file for all domains on an Apache server*, but I want to append to each domain's (if pre-existing) robots.txt. I want some general rules in place for all domains, but ...
1
vote
3answers
5k views

How to block this URL pattern in Varnish VCL?

My website is getting badly hit by spambots and scrappers. I've used Cloudflare but the problem still remains there. The problem is spambots accessing non-existing urls causing a lot of load to my ...
7
votes
3answers
7k views

How do I use robots.txt to disallow crawling for only my subdomains?

If I want my main website to on search engines, but none of the subdomains to be, should I just put the "disallow all" robots.txt in the directories of the subdomains? If I do, will my main domain ...
0
votes
1answer
79 views

Webmaster randomly reporting a massive increase in 404s (apparently from old sitemaps)

Well, I'm stumped. Several months back, we launched a totally new website, replacing a legacy system that was pretty messy. Part of the mess was many, many pages created that really didn't need to be ...
2
votes
1answer
264 views

Ethical/legal considerations when redirecting

A webcrawler has bought our site down twice. It ignores our robots.txt and we have had no reply from their customer services or support using both e-mail and twitter. I have had to create url ...
4
votes
3answers
927 views

Does google's web crawler download binary files?

My Google-fu is failing me right now. I'm trying to figure out whether Google's web crawler downloads non-image binary files when it spiders sites. I know it downloads (and indexes) images and PDFs, ...
1
vote
3answers
7k views

Blocking 'good' bots in nginx with multiple conditions for certain off-limits URL's where humans can go

After 2 days of searching/trying/failing I decided to post this here, I haven't found any example of someone doing the same nor what I tried seems to be working OK. I'm trying to send a 403 to bots ...
-4
votes
1answer
558 views

Bot being redirected to Google.com when requests Myspace.com… what? [closed]

first time on Serverfault. I'm having a problem connecting to Myspace.com through my server. I've been using mechanize via Python to run a bot, (not spam, crawling for information on musicians) on a ...
1
vote
1answer
480 views

Does a forward web proxy exist that checks and obeys robots.txt on remote domains?

Does there exist a forward proxy server that will lookup and obey robots.txt files on remote internet domains and enforce them on behalf of requesters going via the proxy? e.g. Imagine a website at ...