Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Let's say I have a large website which may have a number of email addresses on it that are getting picked up by spammers. I plan to obfuscate or remove them all.

What's the easiest way to crawl my website to find any email addresses I may be exposing?

Either through on-page text (which Google can pick up, but not very well) or mailto: links (which Google can't).

share|improve this question
    
What kind of website content? Static HTML, dynamic? –  LukeR Jun 3 '10 at 4:49
    
The same way that spammers do. –  Dennis Williamson Jun 3 '10 at 9:23
    
Sorry LukeR didn't see your question. I was looking for a solution that wasn't dependant on server-side technology - so would work with static, dynamic etc equally. The site in question uses a combination of static site plus blog software plus other software. –  thomasrutter Jun 15 '10 at 2:09
add comment

1 Answer

up vote 2 down vote accepted

Something like:

wget --mirror --html-extension http://www.mydomain.org/ -o /home/user/temp_site/

then

grep -Hinr "@mydomain.org" /home/user/temp_site/ > /home/user/list_of_pages_with_email

Should return a list of all pages (with line numbers) that contain your domains email addresses.

Might need tweaking.

Manuals for wget and grep

share|improve this answer
    
Ah, I'd forgotten about wget's --mirror option. Thanks! –  thomasrutter Jun 3 '10 at 8:03
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.