The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
1answer
53 views

Files repeatedly downloaded form same IP - suggestions?

I have a few files that can be downloaded from my website. Every few hours all files are downloaded from the same IP address (98.137.207.97 - might be Yahoo?). How should I proceed? Should I just ...
2
votes
3answers
2k views

Why are Facebook's servers hitting my site nearly once per minute (and the time gap is decreasing)?

For about 4 hours now Facebook's servers have been hitting my site every 5 mins at the start, but now at least once per minute. Getting this in logs 66.220.153.246 - - [25/Apr/2012:20:08:49 +0000] ...
-1
votes
1answer
683 views

Using wget to get count of pages below a link?

I've been using a sitemapping tool to get a simple count of links below a specific url. The free trial has ended, so I figure that rather than paying $70 for what is very simple functionality, I ...
1
vote
4answers
2k views

Netstat: How to know if it's Human or Bot/Spider/DDOS

I'm using the following command to check how many connections I have from a single ip netstat -anp |grep ':80' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n If it's DDOS attack(500+ ...
1
vote
2answers
164 views

VPS Proxy Leeching

<embarrassing> I recently put up a proxy (squid) on a personal server, made the noobish mistake of not removing the access-all group I put in whilst testing, and came back a few days later to ...
3
votes
1answer
365 views

What to do about spoofed user agents? Scrapers pretending to be spiders

I've been following a few spiders in our logs and I did a traceroute on their ip to find out they are in fact EC2 instances. The user agents are listed as Google bot and msnbot but they are not ...
4
votes
4answers
2k views

How do I rate limit google's crawl of my IP block?

I have several sites in a /24 network that all get crawled by google on a pretty regular basis. Normally this is fine. However, when google starts crawling all the sites at the same time, the small ...
4
votes
1answer
662 views

Site crawler/spider that tosses results into mysql

It's been suggested that we use mysql for our site's search as it'd be running on the same server that hosts our web server (nginx) and our db (mysql). Since not all of our pages are created from the ...
1
vote
1answer
424 views

Techniques for Filtering Spiders/Bots during Log Files Analysis

I'll begin by telling you what we do. The measures we have implemented catch a lot of spiders, but we have no idea how many we are missing. Currently, we apply a set of measures that are obviously ...
1
vote
5answers
625 views

Is there any way to find out how many hits you receive from bots, on an IIS6 site?

Is there any way to figure out how many bots (even if it's just one type of bot) that hits our site, on a daily, weekly and monthly average? we have the standard IIS6 web site logs in lots of files. ...
2
votes
2answers
578 views

What does the arxiv.org anti-bot “search and destroy” actually do?

The lanl.arxiv.org math and scientific preprint service (formerly known as xxx.lanl.gov) has a strict policy against bots that ignore its robots.txt, Robots Beware. On that page, the have a link ...