Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've been using a sitemapping tool to get a simple count of links below a specific url. The free trial has ended, so I figure that rather than paying $70 for what is very simple functionality, I should just use wget.

Here's what I have so far: wget --spider --recursive

I'm not sure, however, how to somehow calculate the number of links found from this. I'm also slightly nervous that this is doing what I want it to - will this only get links below the domain of

Any ideas on how to accomplish this?


share|improve this question
Did this get downvoted because this question is not appropriate for server fault? If so, which stack exchange site would it be appropriate for? I feel that it's a fair question, and I'm still looking for a satisfactory answer. – rybosome Dec 28 '11 at 15:05
You could try Pro WebMasters but DO NOT repost it there yet, it is likely that this will be migrated. – tombull89 Dec 28 '11 at 23:00
sudo apt-get install lynx-cur

lynx --dump -listonly |head

And so on.

Edit: For the lazy OP.

tom@altoid ~ $ lynx -dump -nonumbers -listonly|egrep -v "^$"|egrep -v "(Visible|Hidden) links"| while read link; do echo -n "$link   :" ;curl -I -s $link |grep HTTP; done
Visible links   :HTTP/1.1 200 OK
HTTP/1.1 200 OK   :HTTP/1.1 200 OK   :HTTP/1.1 200 OK   :HTTP/1.1 200 OK   :HTTP/1.1 200 OK   :HTTP/1.1 302 Found   :HTTP/1.1 200 OK


share|improve this answer
This won't recursively descend a site. lynx -help on the option -dump says the following: 'dump the first file to stdout and exit'. Even using the option -traversal, this will exit after dumping links from the homepage. – rybosome Dec 28 '11 at 15:14
If you can't figure out how to pipeline that list of links, then i'm afraid there's little anyone here can help you with. – Tom O'Connor Dec 28 '11 at 16:59
So, you're talking about piping the output of this into another run of lynx? Great, two levels of depth, not accounting for repeated links. So to fully spider the site, I'd end up needing to write a shell script to continually call this. At that point, I may as well just modify the Ruby crawler I've already written to only count links instead of scrape content. – rybosome Dec 28 '11 at 20:53
@Ryan See edited answer. It's really easy. – Tom O'Connor Dec 28 '11 at 23:08
I appreciate your help, but I'm not sure what benefit you get by being rude. – rybosome Dec 29 '11 at 14:58

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.