1

Let's say I have a large website which may have a number of email addresses on it that are getting picked up by spammers. I plan to obfuscate or remove them all.

What's the easiest way to crawl my website to find any email addresses I may be exposing?

Either through on-page text (which Google can pick up, but not very well) or mailto: links (which Google can't).

  • What kind of website content? Static HTML, dynamic? – LukeR Jun 3 '10 at 4:49
  • The same way that spammers do. – Paused until further notice. Jun 3 '10 at 9:23
  • Sorry LukeR didn't see your question. I was looking for a solution that wasn't dependant on server-side technology - so would work with static, dynamic etc equally. The site in question uses a combination of static site plus blog software plus other software. – thomasrutter Jun 15 '10 at 2:09
2

Something like:

wget --mirror --html-extension http://www.mydomain.org/ -o /home/user/temp_site/

then

grep -Hinr "@mydomain.org" /home/user/temp_site/ > /home/user/list_of_pages_with_email

Should return a list of all pages (with line numbers) that contain your domains email addresses.

Might need tweaking.

Manuals for wget and grep

| improve this answer | |
  • Ah, I'd forgotten about wget's --mirror option. Thanks! – thomasrutter Jun 3 '10 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.