Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I use MRTG to monitor devices' CPU/MEM/traffic usage, and each device has different ports/vlans/interfaces. When I want to create a table/excel showing statistics of all the interfaces, I found it very inefficient because I need to click into about 400 web pages and copy tables (In/Out traffic, Max/Avg/Current CPU usage, Max/Avg Connections, etc.) from the html.

My question: Is there any scripts/tools to automatize this work? [Extract certain table values from each web page and fill in a table/excel].

share|improve this question
add comment

2 Answers

You might find that there are better ways to get the data you want than parsing the HTML generated by MRTG (e.g., such as using SNMP directly). However, this might help if really need to parse the HTML:

If you are willing to use Python then a the BeautifulSoup library will help you. You will still have to write some code but BeautifulSoup is pretty flexible.

Here is a crude example that dumps all HTML tables in a page to CSV:

import sys
import csv
import urllib2

import BeautifulSoup

page    = urllib2.urlopen(sys.argv[1]).read()
soup    = BeautifulSoup.BeautifulSoup(page)
csvout  = csv.writer(sys.stdout)

for table in soup.findAll('table'):
    print '#'
    print '# Table'
    print '# Fields: ' + ','.join([tr.text for tr in table.findAll('th')])
    for row in table.findAll('tr'):
        csvout.writerow([tr.text for tr in row.findAll('td')])
    print

Assuming that you have Python installed as well as BeautifulSoup (pip install BeautifulSoup or easy_install BeautifulSoup) and assuming that you have saved the above code as htmltable2csv.py you can display all the tables on a page like this:

python htmltable2csv.py http://www.w3schools.com/tags/tag_table.asp
share|improve this answer
add comment
up vote 0 down vote accepted

I finally find Firefox extension iMacros for Firefox very useful in recording and replaying repetitious work. It works well and efficiently in extracting data from massive web pages with same structures.

What's more, iMacros has detailed tutorials and sample codes. iMacros for Firefox is free for personal and commercial use, if you have concern on this (like me).

I would strongly recommend iMacros if you ever want to automate some cumbersome work!

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.