up vote 1 down vote favorite
share [g+] share [fb]

I'm looking to monitor changes on websites and my current approach is being defeated by a rotating top banner. Is there a UNIX tool that takes a selection parameter (id attribute or XPath), reads HTML from stdin and prints to stdout the subtree based on the selection?

For example, given an html document I want to filter out everything but the subtree of the element with id="content". Basically, I'm looking for the simplest HTML/XML equivalent to grep.

link|improve this question
76% accept rate
feedback

2 Answers

up vote 1 down vote accepted

Possibly not what you're looking for, but how about writing a quick script in Python, using BeautifulSoup to process the HTML, and give you a sensibly structured object which you can access the content.

import urllib2
from BeautifulSoup import BeautifulSoup as BS
soup = BS(urllib2.urlopen('http://www.google.com').read())
soup.findAll('a')[1]
#returns
<a onclick="gbar.qs(this)" href="http://video.google.co.uk/?hl=en&amp;tab=wv" class="gb1">Videos</a>
link|improve this answer
I was thinking about using Beautiful Soup but figured someone had recognized the problem and make a reusable component out of it already. Thanks for the example though. – jldugger May 5 '10 at 23:16
I bet nobody has. If you can be arsed to make it generic enough, i'd suggest you bung the project on googlecode or similar. – Tom O'Connor May 5 '10 at 23:54
This is the strategy I went with. There's a lot of XML tools, but they don't cope with poorly written HTML. I've generalized the code a bit, but need to spend a bit more time UNIX styling it before publishing. – jldugger May 6 '10 at 21:04
:) Cool. Glad it was useful! – Tom O'Connor May 6 '10 at 22:59
feedback
up vote 1 down vote

write a Perl script with LWP and HTML::TreeBuilder::XPath perhaps.

link|improve this answer
Effectively the same as my suggestion. I find BeautifulSoup to be somewhat more semantic than Perl's offering. Depends what the OP prefers I guess! – Tom O'Connor May 5 '10 at 23:12
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.