I'm using the
wget program, but I want it not to save the html file I'm downloading. I want it to be discarded after it is received. How do I do that?
I'm using the
If you don't want to save the file, and you have accepted the solution of downloading the page in
/dev/null, I suppose you are using wget not to get and parse the page contents.
If your real need is to trigger some remote action, check that the page exists and so on I think it would be better to avoid downloading the html body page at all.
wget options in order to retrieve only what you really need, i.e. http headers, request status, etc.
assuming you need to check the page is ok (ie, the status returned is 200) you can do the following:
wget --no-cache --spider http://your.server.tld/your/page.html
if you want to parse server returned headers do the following:
wget --no-cache -S http://your.server.tld/your/page.html
If you need to crawl a website using wget and want to minimize disk churn...
For a *NIX box and using
wget, I suggest skipping writing to a file . I noticed on my Ubuntu 10.04 box that
wget -O /dev/null caused wget to abort downloads after the first download.
I also noticed that
wget -O real-file causes wget to forget the actual links on the page. It insists on an
index.html to be present on each page. Such pages may not always be present and wget will not remember links it has seen previously.
For crawling without writing to disk, the best I came up with is the following
mkdir /dev/shm/1 cd /dev/shm/1 wget --recursive --relative --no-parent ...
Notice there is no
-O file option. wget will write to the $PWD directory. In this case that is a RAM-only tmpfs file system. Writing here should bypass disk churn (depending upon swap space) AND keep track of all links. This should crawl the entire website successfully.
Afterward, of course,
rm --recursive --force /dev/shm/1/*