Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Using wget command, how do I allow/instruct to overwrite my local file everytime, irrespective of how many times I invoke.

Let's say, I want to download a file from the location: http://server/folder/file1.html

Here, whenever I say wget http://server/folder/file1.html, I want this file1.html to be overwritten in my local system irrespective of the time it is changed, already downloaded, etc. My intention/use case here is that when I call wget, I'm very sure that I want to replace/overwrite the existing file.

I've tried out the following options, but each option is intended/meant for some other purpose.

  1. -nc => --no-clobber
  2. -N => Turn on time-stamping
  3. -r => Turn on recursive retrieving
share

locked by Chris S Feb 10 at 20:32

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

comments disabled on deleted / locked posts

6 Answers 6

up vote 7 down vote accepted
wget -q http://www.whatever.com/filename.txt -O /path/filename.txt 

-q is quiet mode so you can throw it in a cron without any output from the command

share
    
There is no direct option in wget command that does this without me specifying explicitly using -O filename? –  Gnanam Aug 17 '10 at 13:14
3  
It seems that there is no way to force overwriting every files when downloading files using wget. However, use -N option can surely force downloading and overwriting newer files. wget -N Will overwrite original file if the size or timestamp change –  aleroot Aug 17 '10 at 13:21
    
Not true. direct the output of the command into stdout and pipe it to the file: wget -q $urlYouNeedToGrap > $fileYouWantToOverwriteEverytime –  rasjani Aug 21 '10 at 22:35
add comment

This option works

wget -N http://server/folder/file1.html

info

-N,  --timestamping            don't re-retrieve files unless newer than
                               local.
share
add comment

Use curl instead?

curl http://server/folder/file1.html > file1.html
share
    
I'm not a Linux expert. What is the basic difference between wget and curl? I'm sure that each command is meant for some specific purpose. –  Gnanam Aug 17 '10 at 13:27
1  
@Gnanam: They overlap a lot in basic CLI utility, actually. Both can make an HTTP connection and save the result to disk. For a run down on the differences check out daniel.haxx.se/docs/curl-vs-wget.html Regardless, the above usage is complete valid. There are other tools in this general area, too: curl.haxx.se/docs/comparison-table.html –  Stu Thompson Aug 17 '10 at 14:13
    
Those 2 links are really helpful to understand the difference. –  Gnanam Aug 18 '10 at 4:48
add comment

I don't think you can do it unless you also download the directories (so pass the -x flag). If you know what the file is, you can do use -O filename, so for example:
wget http://yourdomain.com/index.html -O index.html

share
add comment

Untried: maybe you can work with wget -r --level=0.

Another possibility: curl -O overwrites (but it uses a different way of choosing the file name, which may or may not matter to you).

share
    
That wget line doesn't really do the correct thing because it creates a hierarchy of of subdirectories. curl -O on the other hand correctly downloads the file, overwriting existing copies. –  sth Sep 16 '13 at 18:04
add comment

Why not put a small wrapper around the wget in your script?

The script could move all the files to a temporary location, then wget the remote files / web pages.

On success delete the files in the temporary location. On failure move the files back and raise an error.

There isn't a simple way to do what you want using just wget unless you know specifically the name of all files, in which case the -O option will allow you to force the filename of the file downloaded.

share
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.