9
votes

Using wget command, how do I allow/instruct to overwrite my local file everytime, irrespective of how many times I invoke.

Let's say, I want to download a file from the location: http://server/folder/file1.html

Here, whenever I say wget http://server/folder/file1.html, I want this file1.html to be overwritten in my local system irrespective of the time it is changed, already downloaded, etc. My intention/use case here is that when I call wget, I'm very sure that I want to replace/overwrite the existing file.

I've tried out the following options, but each option is intended/meant for some other purpose.

  1. -nc => --no-clobber
  2. -N => Turn on time-stamping
  3. -r => Turn on recursive retrieving
7
votes
wget -q http://www.whatever.com/filename.txt -O /path/filename.txt 

-q is quiet mode so you can throw it in a cron without any output from the command

  • There is no direct option in wget command that does this without me specifying explicitly using -O filename? – Gnanam Aug 17 '10 at 13:14
  • 3
    It seems that there is no way to force overwriting every files when downloading files using wget. However, use -N option can surely force downloading and overwriting newer files. wget -N Will overwrite original file if the size or timestamp change – aleroot Aug 17 '10 at 13:21
  • Not true. direct the output of the command into stdout and pipe it to the file: wget -q $urlYouNeedToGrap > $fileYouWantToOverwriteEverytime – rasjani Aug 21 '10 at 22:35
8
votes

This option works

wget -N http://server/folder/file1.html

info

-N,  --timestamping            don't re-retrieve files unless newer than
                               local.
4
votes

Use curl instead?

curl http://server/folder/file1.html > file1.html
  • I'm not a Linux expert. What is the basic difference between wget and curl? I'm sure that each command is meant for some specific purpose. – Gnanam Aug 17 '10 at 13:27
  • 1
    @Gnanam: They overlap a lot in basic CLI utility, actually. Both can make an HTTP connection and save the result to disk. For a run down on the differences check out daniel.haxx.se/docs/curl-vs-wget.html Regardless, the above usage is complete valid. There are other tools in this general area, too: curl.haxx.se/docs/comparison-table.html – Stu Thompson Aug 17 '10 at 14:13
  • Those 2 links are really helpful to understand the difference. – Gnanam Aug 18 '10 at 4:48
0
votes

I don't think you can do it unless you also download the directories (so pass the -x flag). If you know what the file is, you can do use -O filename, so for example:
wget http://yourdomain.com/index.html -O index.html

0
votes

Untried: maybe you can work with wget -r --level=0.

Another possibility: curl -O overwrites (but it uses a different way of choosing the file name, which may or may not matter to you).

  • That wget line doesn't really do the correct thing because it creates a hierarchy of of subdirectories. curl -O on the other hand correctly downloads the file, overwriting existing copies. – sth Sep 16 '13 at 18:04
0
votes

Why not put a small wrapper around the wget in your script?

The script could move all the files to a temporary location, then wget the remote files / web pages.

On success delete the files in the temporary location. On failure move the files back and raise an error.

There isn't a simple way to do what you want using just wget unless you know specifically the name of all files, in which case the -O option will allow you to force the filename of the file downloaded.

Not the answer you're looking for? Browse other questions tagged or ask your own question.