Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I use MRTG to monitor devices' CPU/MEM/traffic usage, and each device has different ports/vlans/interfaces. When I want to create a table/excel showing statistics of all the interfaces, I found it very inefficient because I need to click into about 400 web pages and copy tables (In/Out traffic, Max/Avg/Current CPU usage, Max/Avg Connections, etc.) from the html.

My question: Is there any scripts/tools to automatize this work? [Extract certain table values from each web page and fill in a table/excel].

share|improve this question

You might find that there are better ways to get the data you want than parsing the HTML generated by MRTG (e.g., such as using SNMP directly). However, this might help if really need to parse the HTML:

If you are willing to use Python then a the BeautifulSoup library will help you. You will still have to write some code but BeautifulSoup is pretty flexible.

Here is a crude example that dumps all HTML tables in a page to CSV:

import sys
import csv
import urllib2

import BeautifulSoup

page    = urllib2.urlopen(sys.argv[1]).read()
soup    = BeautifulSoup.BeautifulSoup(page)
csvout  = csv.writer(sys.stdout)

for table in soup.findAll('table'):
    print '#'
    print '# Table'
    print '# Fields: ' + ','.join([tr.text for tr in table.findAll('th')])
    for row in table.findAll('tr'):
        csvout.writerow([tr.text for tr in row.findAll('td')])

Assuming that you have Python installed as well as BeautifulSoup (pip install BeautifulSoup or easy_install BeautifulSoup) and assuming that you have saved the above code as you can display all the tables on a page like this:

share|improve this answer
up vote 0 down vote accepted

I finally find Firefox extension iMacros for Firefox very useful in recording and replaying repetitious work. It works well and efficiently in extracting data from massive web pages with same structures.

What's more, iMacros has detailed tutorials and sample codes. iMacros for Firefox is free for personal and commercial use, if you have concern on this (like me).

I would strongly recommend iMacros if you ever want to automate some cumbersome work!

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.