Questions tagged [web-crawler]
A web-crawler (also known as a web-spider) traverses the webpages of the internet by following the links of urls contained within each webpage. There is usually an initial seed of URLs from which the crawler is given to initialize its crawl.
99
questions
2
votes
2
answers
3k
views
Allow only Cloudflare access to my website and block all other requests?
Original question title: "Allow only cloudflare access to my website and block all visits, bots or crawlers to my IP address"
I have a question, I use cloudflare DNS on my domain.
My VPS 30....
1
vote
1
answer
164
views
Probed for .env files in commonly used resource paths [closed]
I am using an NGINX server to host a static website exposed to the open internet. While glancing through the access logs I came across a cluster of requests for resources ending with .env, e.g:
"...
- 80.3k
0
votes
1
answer
65
views
Weird traffic behavior on Ubuntu server
top - 19:51:36 up 1 day, 12:27, 1 user, load average: 19.14, 11.33, 4.74
Tasks: 172 total, 18 running, 154 sleeping, 0 stopped, 0 zombie
%Cpu(s): 90.0 us, 10.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0....
- 2,558
1
vote
1
answer
132
views
What things should I consider when identifying and rate limiting bots?
// Not sure if this question is best fit for serverfault or webmasters stack exchange...
I am thinking to rate limit access to my sites because identifying and blocking bad bots take most of my time.
...
- 1,849
0
votes
1
answer
78
views
DNS redirecting to mantainance page during planned mantainance - what happens to google indexing?
We are planning a mantainance that could take down the services for a whole day. I would like therefore to show a mantainance page, explaining the issue and providing additional info/links. During ...
- 1,849
0
votes
2
answers
2k
views
Switching between multiple authentication types on same URL
I have a secure SSO site that uses Shibboleth authentication and SAML identity provider. I need to allow a Google Search Appliance crawler to come index the URL's. I have a requirement to change on ...
CommunityBot
- 1
0
votes
0
answers
89
views
Can many connections cause dns lookup or request timeout?
I'm running crawler on my company's internet. 10 raspberry pi * 45 crawlers each, 2 desktops * 70 crawlers each
These processes are sending requests 24/7.
3~5% of packets are getting lost. This is ...
1
vote
1
answer
281
views
Getting requests for suspicious php files
I am getting weird GET requests on my (non php supporting) web server for some curious looking php files. Was just wondering whether these are harmless requests of certain browser tools or attempts ...
- 1,964
1
vote
1
answer
441
views
Website blocks my requests from linux ubuntu server
I'm a Java engineer with zero dev ops experience. Lately I was playing around with linux ubuntu server first time and used docker with my selenium project and faced this problem:
I try to scrape HTML ...
0
votes
1
answer
440
views
How to avoid emails sent to Google's deep web crawler
My website has an area restricted to users who sign up with a valid email. I have got requests with bogus emails, and I want to avoid sending emails to non-existent addresses lest they increase the ...
- 269
11
votes
5
answers
40k
views
Finding all IP ranges belonging to a specific ISP
I'm having an issue with a certain individual who keeps scraping my site in an aggressive manner; wasting bandwidth and CPU resources. I've already implemented a system which tails my web server ...
- 436
1
vote
0
answers
79
views
Spotify Bot Using Massive Bandwidth on NGINX Cached Server?
I have a couple of podcasts I host on my site and I've noticed a disturbing trend the last couple of months: my site's bandwidth usage has gone up by 10x, but it appears most of it was a series of ...
0
votes
1
answer
297
views
Can missing HTTP referrer's identify web crawlers?
I am currently trying to analyze the traffic of a website.
Besides specifics regarding the requested resource and timestamps,
the tracking system only provides the request's HTTP referrer.
In most ...
- 241
0
votes
2
answers
242
views
Strange behavior in Apache log
I have a Nextcloud server running on Apache, and disabled my firewall for about 5 minutes while I ran an apt-update. I decided to check the logs after, and found this from an unknown IP. It looks like ...
1
vote
1
answer
729
views
Why would Apache log different response sizes for the same url?
I noticed a couple (ostensibly-)harmless log entries, and--I'm admittedly overthinking this by a mile--got curious about Apache2 response sizes.
This Ukranian crawler † hit my web daemon, two seconds ...
- 3,019
4
votes
3
answers
480
views
How often does Google's web spiders crawl the web?
Just a few hours after having made some changes in the HTML of my site, I found that Google had updated its search result against my website. The Internet is so huge, how did the Google crawler do ...
CommunityBot
- 1
0
votes
1
answer
87
views
How to block attempts for phpMyAdmin? [closed]
I converted my website from asp.net to .net core and host on same server. Now, website gets hundred of hits daily from different IP's trying to access like below
/php-myadmin/
/wp-content/
/mysql/
...
8
votes
3
answers
11k
views
How do I use robots.txt to disallow crawling for only my subdomains?
If I want my main website to on search engines, but none of the subdomains to be, should I just put the "disallow all" robots.txt in the directories of the subdomains? If I do, will my main domain ...
0
votes
0
answers
76
views
Strange Google Behavior with indexing SSL Mismatch content
Here is a strange one for you. We have a server with multiple VHOSTS that include both SSL and Non-SSL domains.
Domain1 is SSL enabled, while Domain2 doesn't have SSL.
Since all these domains are ...
- 25k
-3
votes
1
answer
205
views
How to gather in a save, cheap and easy way high quality entropy on a Linux machine? [closed]
When no radioactive decay is available and good entropy is strongly advised for security reasons you experience a real problem. HTTPS connections consume a lot of entropy. If you have thousands of ...
- 198k
-4
votes
2
answers
67
views
what ip will logged in a website if I access a website via another website through my PC? [closed]
If http://example2.com sends cURL connection to a website called http://example1.com. If I access http://example2.com from my pc to see the content of http://example1.com, than would http://example1....
- 103
1
vote
0
answers
145
views
HTTrack stores extensionless pages with a .html appended
I'd like to mirror an old site of mine to local files. I've used httrack for this in the past, but I'm having a problem this time that I really thought I figured out before, but can't seem to now.
...
- 151
-1
votes
2
answers
2k
views
How to disallow crawling for all subdomains using my main domain's physical robots.txt file [closed]
I have multiple physical sub-domains and I don't want to change any robots.txt file of any of that sub-domains.
Is there any way to disallow all the sub-domains from my main domain's physical robots....
- 49.8k
0
votes
1
answer
611
views
Yandex/Google Bot Spam
I recently logged into a vps I have (with vultr, if that is of any concern). To find a large amount of nginx logs and higher than expected load average. This server is doing effectively nothing, and ...
- 4,638
-1
votes
1
answer
73
views
40.96.18.165 keeps visiting my web server
Something/someone from 40.96.18.165 has been hitting my web server exactly eight times a day everyday since Feb 5, 2017. The user agent used is Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0).
...
- 10.2k
1
vote
1
answer
2k
views
How many requests can a router handle?
I made a script to scan a file which contains a portion of ipv4 addresses (about 50 million), it attempts to connect to the website using OpenSSL and extract a small piece of it and writes it into a ...
- 1,876
0
votes
1
answer
198
views
Bots/crawlers adding numbers to GET parameters
I've got some errors showing up in my site logs where some bots are trying to access URLs with strange GET params.
# normal url
example.com?foo=123456
# odd url triggering integer error by bots
...
- 303
2
votes
0
answers
35
views
Barracuda.com and crawling / pinger services causing unusual load on web servers
I recently received a large number of hits on my home page from 64.235.153.8. It revolves to barracuda.com I know Barracuda as an enterprise class spam detection/prevention solution.
Do they also ...
- 151
1
vote
1
answer
3k
views
Bash script - wait for all xargs processes to be finished
I have written a small bash script for crawling an XML sitemap of URLs. It retrieves 5 URLs in parallel using xargs.
Now I want an E-Mail to be sent when all URLs have been crawled, so it has to wait ...
CommunityBot
- 1
3
votes
0
answers
2k
views
Block Bad Bots in Nginx for Multiple Sites
I need to block a bunch of robots from crawling a few hundred sites hosted on a Nginx web server running on a Ubuntu 16.04 machine.
I've found a fairly simple example here
(important part of the ...
- 193
-1
votes
0
answers
44
views
Is offering the contents of a third party web site offline violating the law? [closed]
I have developed a nice little app that crawls a bunch of newspaper web sites and makes their latest content available on my phone offline. It's basically a Pocket app that saves contents ...
- 101
12
votes
3
answers
3k
views
How do sites detect bots behind proxies or company networks
How do large sites (e.g. Wikipedia) deal with bots that are behind other IP masker? For instance, in my university, everybody searches Wikipedia, giving it a significant load. But, as far as I know, ...
- 303
1
vote
1
answer
62
views
Suspected malicious activity by one of my site's users; any way to know for sure?
In the course of about 2 hours, a logged in user on my website accessed roughly 1,600 pages in a way that looks suspiciously similar to a bot. I am concerned because users must purchase access to the ...
- 4,021
2
votes
1
answer
500
views
Q: Strange web traffic - Is this an attack?
I recently noticed some strange traffic in my nginx access logs. I'm not sure if these indicate an attack, a mistake, or something else. I've started sending these to HTTP 444, so these logs will ...
- 206
5
votes
2
answers
4k
views
Strange request in access.log, how to block?
I am using nginx on my own server, and I noticed a few days ago some strange request in my access.log :
77.50.217.37 - - [19/Aug/2011:17:50:50 +0200] "GET http://images.google.com/ HTTP/1.1" 200 151 "...
- 2,276
3
votes
1
answer
843
views
Baidu Spider causing 3Gb of traffic a day - but I do business in China
I'm in a difficult situation, the Baidu spider is hitting my site causing about 3Gb a day worth of bandwidth. At the same time I do business in China so don't want to just block it.
Has anyone else ...
- 22.5k
0
votes
1
answer
486
views
Moved website to new server - updated DNS - web crawlers still hitting old site by IP
About ten days ago I moved a site - mostly a Joomla discussion board - to a new server at a different IP address. During a brief scheduled downtime I replicated the content over and completed DNS ...
- 908
-3
votes
2
answers
112
views
Restricting Access from BOTS
I would like to protect my server from too many hits from Bots.
Considering a scenario, where in a server (physical) located in a private network and hitting my server continuously. Do i have a ...
- 4,757
7
votes
4
answers
5k
views
How do I rate limit google's crawl of my IP block?
I have several sites in a /24 network that all get crawled by google on a pretty regular basis. Normally this is fine. However, when google starts crawling all the sites at the same time, the small ...
- 18.4k
0
votes
1
answer
1k
views
Nginx log shows suspicious directory access!!! How to block them?
On my Nginx log recently i have noticed 100's entries like this where a directory search was executed with error, because those directory does not exist on my webserver. now, how can I block them once ...
CommunityBot
- 1
0
votes
1
answer
330
views
Any good webcrawler besides DRKSpider [closed]
I was having a look at DRKSpider to find problems with a website in our production server, but it seems its export feature generates different outputs (with different content).
My goal is to find a ...
- 18.4k
2
votes
2
answers
57
views
What's the purpose of spammy HTTP referers?
In the logs of my website, there's a lot of visits with a HTTP referer set to spam-like websites (usually Russian sites, I've noticed). I assume what they're doing is just using a web crawler to visit ...
CommunityBot
- 1
1
vote
0
answers
680
views
What are the symptoms of an overloaded webserver
I'm maintaining some web crawlers. I want to improve our load/throttling system to be more intelligent.
Of cause I look at response codes, and throttle up or down based on that. I would though like ...
- 358
0
votes
0
answers
701
views
How to make Google crawl my site using IPv6 address when my domain name has both IPv4 and IPv6 addresses?
My domain name has both IPv4 and IPv6 addresses assigned.
IPv4 connection to Google can't be available all the time due to restrictions of my campus network,
but IPv6 is available all the time.
...
1
vote
3
answers
6k
views
How to block this URL pattern in Varnish VCL?
My website is getting badly hit by spambots and scrappers. I've used Cloudflare but the problem still remains there. The problem is spambots accessing non-existing urls causing a lot of load to my ...
- 18.4k
2
votes
1
answer
298
views
Ethical/legal considerations when redirecting
A webcrawler has bought our site down twice. It ignores our robots.txt and we have had no reply from their customer services or support using both e-mail and twitter.
I have had to create url ...
CommunityBot
- 1
0
votes
1
answer
4k
views
Is it possible to block HTTP traffic from specific machines?
I have some web crawlers, and a specific website seems to be blocking traffic temporarily after some time. The thing is, even though all clients have the same external IP address (they access the ...
- 142k
1
vote
3
answers
10k
views
Blocking 'good' bots in nginx with multiple conditions for certain off-limits URL's where humans can go
After 2 days of searching/trying/failing I decided to post this here, I haven't found any example of someone doing the same nor what I tried seems to be working OK. I'm trying to send a 403 to bots ...
CommunityBot
- 1
4
votes
3
answers
1k
views
Does google's web crawler download binary files?
My Google-fu is failing me right now.
I'm trying to figure out whether Google's web crawler downloads non-image binary files when it spiders sites. I know it downloads (and indexes) images and PDFs, ...
CommunityBot
- 1
30
votes
4
answers
9k
views
Does a company have implied right to crawl my website?
I have found out that McAfee SiteAdvisor has reported my website as "may be having security issues".
I care little about whatever McAfee thinks of my website (I can secure it myself and if not, ...
- 113