11

I want to download an entire website using wget but I don't want wget to download images, videos etc.

I tried

wget -bqre robots=off -A.html example.com –user-agent=”Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6″

but when I do that it doesn't download .php files, just downloads static .html files.

Is there a solution to this problem with wget?

Improve this question

3 Answers 3

Reset to default
6

You've explicitly told wget to only accept files which have .html as a suffix.

Assuming that the php pages have .php, you can do this:

wget -bqre robots=off -A.html,.php example.com –user-agent=”Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6″

Note that this will downloaded the rendered html, not the source of the php. If the page is sufficiently dynamic, you might not get the rendered result you expect.

However, I'd suggest that another tool such as httrack may do a better job - it depends on exactly what you need to do.

Improve this answer
1
  • I have to disagree with httrack doing a better job. I find its GUI buggy, unclear and in need of a good cleanup. Plus, wget has (almost) all the same options and is much more straigthforward. I have had only bad experiences with HTTRACK and I have played around with it a lot.
    – avia
    Commented Sep 17, 2021 at 9:52
3

-A takes a list, so -A.html,.php should fit the bill. You should also look in to -R (it also takes a reject list).

Improve this answer
3

Yes, there is, and it's pretty simple. Take a look at this SO answer: https://superuser.com/questions/709702/how-to-crawl-using-wget-to-download-only-html-files-ignore-images-css-js

tl/dr; use --follow-tags=a which will follow only a tags.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .