GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.

learn more… | top users | synonyms

6
votes
1answer
13k views

How do I fix certificate errors when running wget on an HTTPS URL in Cygwin Windows?

I have Cygwin installed in my windows system I am trying to execute wget commad for a website but I am gettign the following error message ERROR: The certificate of xxx.com' is not trusted. ERROR: ...
0
votes
2answers
2k views

wget works for all sites on the web but not the one hosted on that server

I currently have 2 Ubuntu 12.04 servers which are load balanced. If I go to anyone on them from the shell and type: wget stackoverflow.com The page is fetched into index.html. However, assuming ...
1
vote
1answer
114 views

Serve mirrored (static) web-page with original headers

I have a dynamic webpage which I want to create a "frozen" copy of. Typically I would do something like wget -m http://example.com, and then put the files in the document root of the web-server. ...
2
votes
1answer
331 views

Stream tar.gz file from FTP server

Here is the situation: I have a tar.gz file on a FTP server which can contain an arbitrary number of files. Now what I'm trying to accomplish is have this file streamed and uploaded to HDFS through a ...
4
votes
4answers
779 views

How do I reconcile Network Speed from Network Throughput?

I have been arguing with my ISP about the service they are providing; primarily with regard to throughput speeds. I have set up an edge router (see hardware details listed below) and on it I have ...
-1
votes
2answers
521 views

wget not saving file as the triggered file to disk [closed]

What I am trying to do is download a file from a url like this: http://www.example.com/down/files/425/document-preview? When I visit this url in my browser, it triggers a file called for example: ...
1
vote
1answer
535 views

iptables redirect tcp to checkip.dyndns.org from localhost to 127.0.0.1:8118

I've tried several different combinations of rules and nothing seems to be working. I know that you can't use prerouting table for a request coming from the localhost so I used the output table and ...
1
vote
3answers
3k views

Download Git Zipball in Unix

I'm trying to download a zipball of a git repo: e.g. wget https://github.com/zeromq/jzmq/zipball/master This works fine in a web browser but on unix the file gets a weird name...how do I do this?
-1
votes
2answers
573 views

How download all files, directories and subdirectoires with wget [closed]

I have some links such as: http:/foo.com/1/1/1/1.jpg http:/foo.com/1/2/2/2.jpg http:/foo.com/1/3/3/3.jpg ... How I can I download all files and directories with wget?
1
vote
2answers
582 views

Using wget and Awk to count similar expressions

I am trying to create a script that uses wget to download a data set and then awk to sort though the file and tell you the most common filter used which is $14 column. So far I have the wget function ...
0
votes
2answers
1k views

406 Not Acceptable error when runing wget

I have this command in crontab: wget --quiet --delete-after http://boms.ro/admincp/cron/s/9abf0f42c1e4f55fdb87d8237cdde And when I run it with the --debug argument I get the following response: ...
11
votes
1answer
46k views

Accessing a windows shared folder from Linux?

I am trying to copy a (.html) file from a Windows XP Professional shared folder onto a server running Ubuntu Linux 10.04 LTS. As it's a shared folder the usual 'wget' doesn't seem to work. I suspect ...
0
votes
2answers
940 views

Curl and Wget returns different response code

I am trying to wget a link, this wget works fine on my local machine but it doesn't do so on the server. i tried to check the response header and i got the following on my local laptop curl -I http:...
0
votes
2answers
1k views

Use number of downloaded file in wget as variable in bash script

Can I get the number of downloaded files with wget -r and use that as a variable? I want to write a script that runs multiple wget commands in it (using -q so i can control output) and then at the end,...
3
votes
2answers
11k views

Ubuntu 11.10, using wget/curl fails with ssl

On a completely new install of Ubuntu I'm getting the following errors when using wget: wget https://test.sagepay.com --2012-03-27 12:55:12-- https://test.sagepay.com/ Resolving test.sagepay.com... ...
1
vote
2answers
836 views

How to untar a single file from an archive with randomly generated directory names?

I'm trying write a script that will download a tarball from github and extract a single file from it. However the top level directory inside the tarball has some random characters in it, which I ...
1
vote
2answers
1k views

Cron not running commands as root

I have a file called /scripts/checkInternet and it contains: #!/bin/bash WGET="/usr/bin/wget" rm /tmp/index.google $WGET -q --tries=10 --timeout=5 http://www.google.com -O /tmp/index....
1
vote
1answer
678 views

How to block some robots and referer pages in Apache root level

When I was using Lighttpd I can easily achieve this by such entries. So all websites were protected. Wget robots: $HTTP["useragent"] =~ "Wget" { $HTTP["url"] =~ "^/tagi(.*)" { # $HTTP[...
-1
votes
2answers
1k views

wget downloading index.html instead of tarball?

When I type the following: http://www.archlinux.org/packages/extra/x86_64/enca/download/ wget is downloading index.html instead of the correct file which is: enca-1.13-2-x86_64.pkg.tar.xz When I ...
0
votes
1answer
3k views

Wget - download all links from a http location (not recursivly)

I have a link to an http page that has a structure like this: Parent Directory - [DIR] _OLD/ 01-Feb-2012 06:05 - [DIR] _Jan/ ...
1
vote
3answers
215 views

Limiting specific process to just one per user

My scripts are using wget to retrieve data from Internet. When many users are using this script I got very high load (about 20.00), because of disk I/O. Wget is automatically started each hour by cron....
0
votes
2answers
2k views

FTP file download using Wget

I am using the wget command for download FTP files, when i download the FTP file its showing error "Event not fount". Here i use the password like some below charater ! so its showing this error bash:...
4
votes
2answers
2k views

How can i make wget download only pages not css images etc?

I want to download an entire website using wget but I don't want wget to download images, videos etc. I tried wget -bqre robots=off -A.html example.com –user-agent=”Mozilla/5.0 (Windows; U; Windows ...
3
votes
1answer
364 views

Downloading from wget - folder issue

I have a website - http://myradio1.localhost and the directory is currently empty. The virtual host works fine, no issues there. This is my command I'm using to download the site via wget: wget -...
1
vote
1answer
3k views

Wget from HTTPS url using user/password -still getting 400 error

I'm trying to run Wget for a certain HTTP url, also using http-user and http-password command line options (tried the --user and --password, same results). Using the --no-check-certificate options as ...
4
votes
3answers
2k views

How to serve a wget --mirror'ed directory of files with questionmarks in them

I'm trying to create a static mirror of a php application (an old php Gallery installation, specifically). The app produces URLs such as: view_album.php?set_albumName=MyAlbum wget downloads these ...
-1
votes
1answer
1k views

Using wget to get count of pages below a link?

I've been using a sitemapping tool to get a simple count of links below a specific url. The free trial has ended, so I figure that rather than paying $70 for what is very simple functionality, I ...
2
votes
2answers
1k views

Can someone explain to me what “wget -O - -q icanhazip.com” mean?

I've used this command to find the IP of the server. Can someone explain what the command means? I want to learn. So far I know: "wget" is a free utility for non-interactive download of files from ...
0
votes
1answer
76 views

How to download files that are dynamically served

I've run into this a few times when a site serves a variable file i try to download wget http://trac-hacks.org/changeset/latest/tracajaxcommentsplugin?old_path=/&filename=tracajaxcommentsplugin&...
2
votes
2answers
192 views

how can i use apache log files to recreate usage scenario

Recently i installed a website that had too many requests and it was too slow. Many improvements have been made to the web site code and we've also bought a new server. I want to test the new server ...
3
votes
2answers
4k views

How to exclude certain directories while using wget?

I'd like to download a directory from a FTP, which contains some source codes. Initially, I did this: wget -r ftp://path/to/src Unfortunately, the directory itself is a result of a SVN checkout, so ...
0
votes
2answers
1k views

host resolution extremely slow using wget on local dev machine. fast using browser

On my local Ubuntu 10.04 dev machine, if I do a wget for a web address such as "wget http://www.google.com", it gets stuck on "Resolving www.google.com" for up to 30 seconds before the response is ...
0
votes
1answer
452 views

Wget not working

wget is not working properly on my debian server. if I wget http://www.google.com, it says it cannot resolve the host, when I ping google.com, I receive a successful pong and when I wget http://www....
11
votes
3answers
39k views

Why wget doesn't verify SSL certificates?

I have a problem with my Fedora 8 installation. It looks that wget doesn't know how to verify SSL certificates any more. It's strange because I have another Fedora 8 box which I believe has the same ...
7
votes
1answer
10k views

How do I pipe the output of uptime/df to curl?

I want to send the output of uptime and df commands to a web app using cURL. I tried: uptime | curl http://someurl.com -T - But that didn't seem to send the data. What is the proper way to send ...
3
votes
2answers
1k views

Curl always returns same 404 page

No matter what URL I specify for curl I always get the same HTML 404 Error page back. If I use the --verbose option, it looks like curl always connects to the same IP address. $ curl --verbose http:/...
0
votes
1answer
1k views

wget and connection errors / timeouts

I was using wget the in the last week to recursively download a whole website of html pages. I used it this way: wget --recursive --no-clobber --page-requisites --html-extension --convert-links --...
-3
votes
3answers
2k views

Automatic download a site with *wget*: how to prevent? [closed]

How to prevent automatic download with wget for a website? Could you give links to articles please? Thanks.
4
votes
2answers
3k views

what is the default rate unit in wget?

When I use wget to download a large file, it gives me "56M/s". I want to know if it means 56 megabits per second or 56 megabytes per second. Thanks! Like this: 231,997,440 61.4M/s in 4.0s
-1
votes
1answer
2k views

wget --timeout doesn't time out an active connection

When using --timeout with wget, I was expecting that if the transaction hasn't completed within the timeout period, then it would abort. However, it seems to mean (with respect to read-timeout) that ...
2
votes
4answers
10k views

Why is wget hanging?

wget to a single specific url from one of my servers keeps getting timeouts. All other urls from this box work fine. This url works OK from any other boxes I have. Here's the output: wget -T 10 http:/...
3
votes
1answer
2k views

Can wget convert links within a local html document?

I know that wget can fetch a remote page and its dependencies and rewrite the html so that image src attributes reference the newly downloaded images. I am trying to convert local html files that ...
0
votes
3answers
3k views

download only new files from ftp using WGET/cUrl

i have Windows 2008 Server on which i store DB backups on daily basis. I want to be able to download new files only using wget, curl or windows built-in FTP doesn't matter. Can you help me with ...
0
votes
1answer
318 views

Am I doing ubuntu 10.0.4 software installation correctly (packages and none-packages)?

Background: I first installed everything using a terminal and "sudo su", i.e. as root. Then I installed jenkins, and then the jenkins user could not run any android sdk tools in /root/androidsdk/tools ...
1
vote
2answers
610 views

Download file in Linux when file location is unknown

Sorry if this is unclear, but I'm trying to set up a script that downloads a file. Currently, my method of downloading the file is by clicking on a link like so: https://www.URL.com/view?downloadFile=...
15
votes
3answers
7k views

Wget: Is there a way to pass username and password from a file?

Is there a way to pass the username and password from a file instead of the command line via --user and --password? Background: I want to run wget via cron and don't want the username/password show ...
1
vote
1answer
604 views

Translate web page from shell

I'm trying to do translate a web page automatically with a shell script. Here is the interesting part: wget http://www.google.com/translate?langpair=en%7Ces&u=http%3A%2F%2Fasdf.com I'm getting ...
2
votes
3answers
479 views

wget application replacement

Is there any other application similar to wget? If yes can you post a link?
1
vote
4answers
1k views

wget an remote website index directory

Is it possible to wget an entire index/parent directory with wget? Because I am trying to transfer images to a VPS of mine from my old shared host. When I say an index/parent directory, I meant like ...
1
vote
2answers
133 views

wget has different response times for same fetch

Any ideas why one fetch would fetch a file (big picture in this example) fast while another fetch would fetch same file slow? Not seeing this issue at home. Not seeing this issue through ip ...