GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.
40
votes
3answers
35k views
how to download ssl certificate from a website?
I want to download the ssl certificate from, say https://www.google.com, using wget or any other commands. Any unix command line? wget or openssl?
25
votes
8answers
18k views
Making `wget` not save the page
I'm using the wget program, but I want it not to save the html file I'm downloading. I want it to be discarded after it is received. How do I do that?
14
votes
2answers
695 views
Wget: Is there a way to pass username and password from a file?
Is there a way to pass the username and password from a file instead of the command line via --user and --password?
Background: I want to run wget via cron and don't want the username/password show ...
8
votes
5answers
9k views
How do I pipe a downloaded file to standard output in bash?
How is it possible to pipe out wget's downloaded file? If not what alternatives should I use?
UPD: Thanks all for your answers, I managed to tune my downloader (curl) to send files to stout. However ...
7
votes
2answers
12k views
Where can I (safely) get libintl3.dll?
I am having problems running the Windows version of wget on a Windows 7 machine. When I run the command I get this error:
The program can't start because libintl3.dll is missing from your ...
6
votes
6answers
3k views
wget recursive download, but I don't want to follow all links
I'm trying to mirror a website using wget, but I don't want to download lots of files, so I'm using wget's --reject option to not save all the files. However wget will still download all the files and ...
6
votes
4answers
2k views
How to refresh an online website mirror created with `wget --mirror`?
One month ago, I used "wget --mirror" to create a mirror of our public website for temporary use during an upcoming scheduled maintenance window. Our primary website runs HTML, PHP & MySQL, but ...
5
votes
3answers
10k views
Why wget doesn't verify SSL certificates?
I have a problem with my Fedora 8 installation. It looks that wget doesn't know how to verify SSL certificates any more. It's strange because I have another Fedora 8 box which I believe has the same ...
5
votes
1answer
235 views
What to do if wget's progress:mega doen't cut it anymore?
I have script for downloading large files in a queue. Now I would really appreciate it, if I had a decent log. Specifying the -o or -a option is apparently specially made for that, however as of the ...
4
votes
3answers
2k views
Why is only one wget command working in my crontab?
I wish to fetch content from a PHP script on my server two times a day, altering a query variable lang to set what language we want, and save this content in two language specific files. This is my ...
4
votes
8answers
11k views
How to allow wget to overwrite files
Using wget command, how do I allow/instruct to overwrite my local file everytime, irrespective of how many times I invoke.
Let's say, I want to download a file from the location: ...
4
votes
1answer
8k views
Accessing a windows shared folder from Linux?
I am trying to copy a (.html) file from a Windows XP Professional shared folder onto a server running Ubuntu Linux 10.04 LTS.
As it's a shared folder the usual 'wget' doesn't seem to work. I suspect ...
4
votes
5answers
1k views
wget to print errors, but nothing otherwise
How can I have wget print errors, but nothing otherwise?
In the default behavior, it shows a progress bar and lots of stuff.
In the --no-verbose version still prints one line per downloaded file, ...
4
votes
4answers
186 views
How do I reconcile Network Speed from Network Throughput?
I have been arguing with my ISP about the service they are providing; primarily with regard to throughput speeds.
I have set up an edge router (see hardware details listed below) and on it I have ...
4
votes
2answers
3k views
Using wget to fetch password protected website (trac wiki page)
I want to write a bash script that will notify me about changes on a trac wiki timeline. Unfortunately I can't "login" (it needs cookie for next logins).
wget http://someserver.com/trac/xxx/login \
...
4
votes
4answers
827 views
wget and pretty urls
In order to automatize stuff, I need to recursively download a web page. I'm using wget as it's probably the most programmer-friendly tool available, using -r flag to trigger link following.
wget, ...
4
votes
2answers
339 views
wget starts downloading then stops “cannot write to”
I'm using wget to mirror some files across from one server to another. I'm using the following command:
wget -x -N -i http://domain.com/filelist.txt
-x = Because I want to keep the directory ...
3
votes
4answers
2k views
Wget a script and run it
I want to download a script from: http://dl.dropbox.com/u/11210438/flockonus-stack.sh and execute it.
My guess is, to use wget renaming it, chmod it, and execute.
What are the commands for doing ...
3
votes
1answer
738 views
“wget --mirror” Alternative
I'm looking for an alternative to using this, it doesn't have to use FTP, but it should provide the same functionality as the "--mirror" option of wget where it only downloads new and changed files.
...
3
votes
4answers
106 views
Download a file from the shell when button click is required
I have a link to a file that I would like to download from the shell. Unfortunately, the URL redirects to a software license agreement page which requires clicking an accept button. This is fine when ...
3
votes
4answers
3k views
How to download with wget without following links with parameters
I'm trying to download two sites for inclusion on a CD:
http://boinc.berkeley.edu/trac/wiki
http://www.boinc-wiki.info
The problem I'm having is that these are both wikis. So when downloading with ...
3
votes
2answers
286 views
pgrep wget: what is the details of process id?
when typing pgrep wget, it show process id 10144
but how to know what is the details of this process id
3
votes
1answer
2k views
How do I fix certificate errors when running wget on an HTTPS URL in Cygwin Windows?
I have Cygwin installed in my windows system
I am trying to execute wget commad for a website but I am gettign the following error message
ERROR: The certificate of xxx.com' is not trusted.
ERROR: ...
3
votes
2answers
329 views
Curl always returns same 404 page
No matter what URL I specify for curl I always get the same HTML 404 Error page back.
If I use the --verbose option, it looks like curl always connects to the same IP address.
$ curl --verbose ...
3
votes
1answer
463 views
How to exclude certain directories while using wget?
I'd like to download a directory from a FTP, which contains some source codes.
Initially, I did this:
wget -r ftp://path/to/src
Unfortunately, the directory itself is a result of a SVN checkout, so ...
3
votes
1answer
664 views
Can wget convert links within a local html document?
I know that wget can fetch a remote page and its dependencies and rewrite the html so that image src attributes reference the newly downloaded images.
I am trying to convert local html files that ...
2
votes
3answers
1k views
How to use “wget” to save a file as a different name
wget "http://flakejfl2ke3jrlwkefj38j2ofiwejfoifjalsdkfje3fklwje.exe" is long.
How do I save it as my own name?
2
votes
3answers
194 views
wget application replacement
Is there any other application similar to wget?
If yes can you post a link?
2
votes
2answers
4k views
Ubuntu 11.10, using wget/curl fails with ssl
On a completely new install of Ubuntu I'm getting the following errors when using wget:
wget https://test.sagepay.com
--2012-03-27 12:55:12-- https://test.sagepay.com/
Resolving test.sagepay.com... ...
2
votes
2answers
3k views
Alternatives to CURL, WGET
Is anyone aware of any alternatives to curl and wget? The key functional requirement I'm looking for is to be able to execute an HTTPS GET request against a known URL.
The reason I'm looking for an ...
2
votes
1answer
30 views
Wget Minimum Download Rate
Can anyone help with the parameter that ensures that wget will fetch the file at a minimum specified download rate?
2
votes
1answer
172 views
Downloading from wget - folder issue
I have a website - http://myradio1.localhost and the directory is currently empty.
The virtual host works fine, no issues there.
This is my command I'm using to download the site via wget:
wget ...
2
votes
2answers
266 views
How can i make wget download only pages not css images etc?
I want to download an entire website using wget but I don't want wget to download images, videos etc.
I tried
wget -bqre robots=off -A.html example.com –user-agent=”Mozilla/5.0 (Windows; U; Windows ...
2
votes
2answers
107 views
how can i use apache log files to recreate usage scenario
Recently i installed a website that had too many requests and it was too slow. Many improvements have been made to the web site code and we've also bought a new server.
I want to test the new server ...
2
votes
2answers
622 views
Why is this bash command not echoing into a variable and what can I do to improve?
I've got a bash script I'm working to improve, and have put on a great fix thanks to Dennis Williamson. Unfortunately, one of the lines no longer echoes into a variable I can manipulate, rather dumps ...
2
votes
1answer
191 views
Why does the time it takes to run this script differ so much between user and system and how I might correct the discrepancy?
My application sits behind a load balancer, and every once in a while I like to do a status check on each machine to get an idea of the time it takes to return an index.html document on each machine.
...
2
votes
2answers
706 views
Retrieve FTP directory tree from the command line
What would be the fastest way to recursively retrieve entire directory listing from an ftp server using wget/curl/whatever? I don't need to download any files, just directory and file names. Basically ...
2
votes
1answer
230 views
How to prevent external website from being downloaded in wget?
I'm using wget to download useful website recursively:
wget -k -m -r -q -t 1 http://www.web.com/
But external website also downloaded along with the website I want.
How to prevent this external ...
2
votes
4answers
1k views
Escape a ! in the password parameter of wget
I'm trying to execute something like this:
wget --user=foo --password=bar! url
The ! in the password is causing problems. I've tried escaping it with \, as in
--password=bar\!
I've tried ...
2
votes
2answers
371 views
How to serve a wget --mirror'ed directory of files with questionmarks in them
I'm trying to create a static mirror of a php application (an old php Gallery installation, specifically). The app produces URLs such as:
view_album.php?set_albumName=MyAlbum
wget downloads these ...
2
votes
2answers
421 views
wget 5xx error code handling
i'm running a shell script that uses wget and pulls something over the web, my question is is there a way to tell wget to abort as soon as it hits 500 error code?
2
votes
6answers
8k views
Using client certificates with wget
I cannot get wget to use the client certificates. The documentation speaks about using the --certificate flag.
The use of the certificate flag is clear, I set it to use the PEM version of the client ...
2
votes
1answer
87 views
Find out what's running wget on my server
In my access.log I see requests coming from 127.0.0.1 and retrieving server's "/" and then all of graphics and CSS referenced in it. This happens regularly, every 5 minutes.
I want to find out what, ...
2
votes
1answer
3k views
HTTP downloads stop after some time, resuming is not possible
When I try to download a file via HTTP, the downloads sometimes stop after around 30 MB. The download rates goes down to 0 B/s and no data keeps coming. When I stop the download and resume again, the ...
2
votes
1answer
200 views
Reboot if tomcat7 service is not responding
I am running a web app on a Tomcat server. There is a hard-to-detect problem within the server code that causes it to crash once or twice everyday. I will dig in to correct it when I have time. But ...
2
votes
1answer
998 views
using wget through .pac config based proxy server
I want to use wget through a proxy which uses .pac config.
When I googled a bit, I found that .pac is a javascript file and wget cannot parse it based on the following url
...
2
votes
3answers
959 views
The --reject feature in wget doesn't seem to work
I am trying to mirror most of a ftp sever using wget.
This particular sever keeps old copies of the data files in a folder called 'backup' inside every subfolder. e.g. '/MaffiaOffShore/backup' ...
1
vote
4answers
302 views
download files from text file that contain files list under linux
i have a text file that contains a list of urls for files of my word
for example :
http://domain.com/file1.zip
http://domain.com/file2.zip
http://domain.com/file3.zip
http://domain.com/file4.zip
...
1
vote
6answers
16k views
WGET can't resolve host
I've got 2 machines on the same sub-net running Debian 5.0. They use the same DNS (/etc/resolv.conf), they route to the same gateway (#route) and they have the same iptables settings (#iptables -L). ...
1
vote
3answers
1k views
Downloading php files from python simple http server
I started python -m SimpleHTTPServer on one computer on lan and used wget to download php files from it to another. As far as i see, they seem to be downloaded correctly - i got php sources instead of ...
