GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.

learn more… | top users | synonyms

1
vote
2answers
4k views

why i cannot download jdk from oracle web site directly without AuthParam?

that is download with the following command, why it fails to download that file? wget http://download.oracle.com/otn-pub/java/jdk/6u35-b10/jdk-6u35-linux-i586.bin the following command works, but ...
5
votes
3answers
2k views

wget starts downloading then stops “cannot write to”

I'm using wget to mirror some files across from one server to another. I'm using the following command: wget -x -N -i http://domain.com/filelist.txt -x = Because I want to keep the directory ...
1
vote
3answers
677 views

Setting wget permission to 755 so users other than root can execute it a big security risk?

I read recently in blogs that by default wget on linux is 750, so only root can execute it. I would like to allow users wget and change it to 755, but I read around the web that it is a big security ...
0
votes
1answer
49 views

wget does not mirror if ip is provided instead of domain name

I am trying to get a site mirroring with wget, and came across a strange behavior. Say I am mirroring an internal site named www.example.com in the following way, all seems to be working fine: wget ...
1
vote
2answers
97 views

Changed DNS, no effect locally

I know it can take up to a few days for DNS changes to take effect, but this has me baffled, maybe somone can offer a plausible explanation.. $ wget http://***OLDIP***/ -O oldserver.html --2012-07-25 ...
5
votes
1answer
5k views

How do I fix certificate errors when running wget on an HTTPS URL in Cygwin Windows?

I have Cygwin installed in my windows system I am trying to execute wget commad for a website but I am gettign the following error message ERROR: The certificate of xxx.com' is not trusted. ERROR: ...
0
votes
2answers
263 views

wget works for all sites on the web but not the one hosted on that server

I currently have 2 Ubuntu 12.04 servers which are load balanced. If I go to anyone on them from the shell and type: wget stackoverflow.com The page is fetched into index.html. However, assuming ...
1
vote
1answer
98 views

Serve mirrored (static) web-page with original headers

I have a dynamic webpage which I want to create a "frozen" copy of. Typically I would do something like wget -m http://example.com, and then put the files in the document root of the web-server. ...
2
votes
1answer
155 views

Stream tar.gz file from FTP server

Here is the situation: I have a tar.gz file on a FTP server which can contain an arbitrary number of files. Now what I'm trying to accomplish is have this file streamed and uploaded to HDFS through a ...
4
votes
4answers
394 views

How do I reconcile Network Speed from Network Throughput?

I have been arguing with my ISP about the service they are providing; primarily with regard to throughput speeds. I have set up an edge router (see hardware details listed below) and on it I have ...
-1
votes
2answers
261 views

wget not saving file as the triggered file to disk [closed]

What I am trying to do is download a file from a url like this: http://www.example.com/down/files/425/document-preview? When I visit this url in my browser, it triggers a file called for example: ...
1
vote
1answer
332 views

iptables redirect tcp to checkip.dyndns.org from localhost to 127.0.0.1:8118

I've tried several different combinations of rules and nothing seems to be working. I know that you can't use prerouting table for a request coming from the localhost so I used the output table and ...
1
vote
3answers
1k views

Download Git Zipball in Unix

I'm trying to download a zipball of a git repo: e.g. wget https://github.com/zeromq/jzmq/zipball/master This works fine in a web browser but on unix the file gets a weird name...how do I do this?
-1
votes
2answers
268 views

How download all files, directories and subdirectoires with wget [closed]

I have some links such as: http:/foo.com/1/1/1/1.jpg http:/foo.com/1/2/2/2.jpg http:/foo.com/1/3/3/3.jpg ... How I can I download all files and directories with wget?
1
vote
2answers
315 views

Using wget and Awk to count similar expressions

I am trying to create a script that uses wget to download a data set and then awk to sort though the file and tell you the most common filter used which is $14 column. So far I have the wget function ...
0
votes
2answers
589 views

406 Not Acceptable error when runing wget

I have this command in crontab: wget --quiet --delete-after http://boms.ro/admincp/cron/s/9abf0f42c1e4f55fdb87d8237cdde And when I run it with the --debug argument I get the following response: ...
9
votes
1answer
21k views

Accessing a windows shared folder from Linux?

I am trying to copy a (.html) file from a Windows XP Professional shared folder onto a server running Ubuntu Linux 10.04 LTS. As it's a shared folder the usual 'wget' doesn't seem to work. I suspect ...
0
votes
2answers
427 views

Curl and Wget returns different response code

I am trying to wget a link, this wget works fine on my local machine but it doesn't do so on the server. i tried to check the response header and i got the following on my local laptop curl -I ...
0
votes
2answers
646 views

Use number of downloaded file in wget as variable in bash script

Can I get the number of downloaded files with wget -r and use that as a variable? I want to write a script that runs multiple wget commands in it (using -q so i can control output) and then at the ...
2
votes
2answers
6k views

Ubuntu 11.10, using wget/curl fails with ssl

On a completely new install of Ubuntu I'm getting the following errors when using wget: wget https://test.sagepay.com --2012-03-27 12:55:12-- https://test.sagepay.com/ Resolving test.sagepay.com... ...
1
vote
2answers
455 views

How to untar a single file from an archive with randomly generated directory names?

I'm trying write a script that will download a tarball from github and extract a single file from it. However the top level directory inside the tarball has some random characters in it, which I ...
1
vote
2answers
890 views

Cron not running commands as root

I have a file called /scripts/checkInternet and it contains: #!/bin/bash WGET="/usr/bin/wget" rm /tmp/index.google $WGET -q --tries=10 --timeout=5 http://www.google.com -O ...
1
vote
1answer
422 views

How to block some robots and referer pages in Apache root level

When I was using Lighttpd I can easily achieve this by such entries. So all websites were protected. Wget robots: $HTTP["useragent"] =~ "Wget" { $HTTP["url"] =~ "^/tagi(.*)" { # ...
-1
votes
2answers
603 views

wget downloading index.html instead of tarball?

When I type the following: http://www.archlinux.org/packages/extra/x86_64/enca/download/ wget is downloading index.html instead of the correct file which is: enca-1.13-2-x86_64.pkg.tar.xz When I ...
0
votes
1answer
1k views

Wget - download all links from a http location (not recursivly)

I have a link to an http page that has a structure like this: Parent Directory - [DIR] _OLD/ 01-Feb-2012 06:05 - [DIR] _Jan/ ...
1
vote
3answers
158 views

Limiting specific process to just one per user

My scripts are using wget to retrieve data from Internet. When many users are using this script I got very high load (about 20.00), because of disk I/O. Wget is automatically started each hour by ...
0
votes
2answers
709 views

FTP file download using Wget

I am using the wget command for download FTP files, when i download the FTP file its showing error "Event not fount". Here i use the password like some below charater ! so its showing this error ...
2
votes
2answers
614 views

How can i make wget download only pages not css images etc?

I want to download an entire website using wget but I don't want wget to download images, videos etc. I tried wget -bqre robots=off -A.html example.com –user-agent=”Mozilla/5.0 (Windows; U; Windows ...
2
votes
1answer
228 views

Downloading from wget - folder issue

I have a website - http://myradio1.localhost and the directory is currently empty. The virtual host works fine, no issues there. This is my command I'm using to download the site via wget: wget ...
1
vote
1answer
2k views

Wget from HTTPS url using user/password -still getting 400 error

I'm trying to run Wget for a certain HTTP url, also using http-user and http-password command line options (tried the --user and --password, same results). Using the --no-check-certificate options as ...
2
votes
2answers
833 views

How to serve a wget --mirror'ed directory of files with questionmarks in them

I'm trying to create a static mirror of a php application (an old php Gallery installation, specifically). The app produces URLs such as: view_album.php?set_albumName=MyAlbum wget downloads these ...
-1
votes
1answer
687 views

Using wget to get count of pages below a link?

I've been using a sitemapping tool to get a simple count of links below a specific url. The free trial has ended, so I figure that rather than paying $70 for what is very simple functionality, I ...
1
vote
2answers
362 views

Can someone explain to me what “wget -O - -q icanhazip.com” mean?

I've used this command to find the IP of the server. Can someone explain what the command means? I want to learn. So far I know: "wget" is a free utility for non-interactive download of files from ...
0
votes
1answer
72 views

How to download files that are dynamically served

I've run into this a few times when a site serves a variable file i try to download wget ...
2
votes
2answers
149 views

how can i use apache log files to recreate usage scenario

Recently i installed a website that had too many requests and it was too slow. Many improvements have been made to the web site code and we've also bought a new server. I want to test the new server ...
3
votes
1answer
1k views

How to exclude certain directories while using wget?

I'd like to download a directory from a FTP, which contains some source codes. Initially, I did this: wget -r ftp://path/to/src Unfortunately, the directory itself is a result of a SVN checkout, so ...
0
votes
2answers
562 views

host resolution extremely slow using wget on local dev machine. fast using browser

On my local Ubuntu 10.04 dev machine, if I do a wget for a web address such as "wget http://www.google.com", it gets stuck on "Resolving www.google.com" for up to 30 seconds before the response is ...
0
votes
1answer
228 views

Wget not working

wget is not working properly on my debian server. if I wget http://www.google.com, it says it cannot resolve the host, when I ping google.com, I receive a successful pong and when I wget ...
5
votes
3answers
19k views

Why wget doesn't verify SSL certificates?

I have a problem with my Fedora 8 installation. It looks that wget doesn't know how to verify SSL certificates any more. It's strange because I have another Fedora 8 box which I believe has the same ...
3
votes
2answers
3k views

How do I pipe the output of uptime/df to curl?

I want to send the output of uptime and df commands to a web app using cURL. I tried: uptime | curl http://someurl.com -T - But that didn't seem to send the data. What is the proper way to send ...
3
votes
2answers
629 views

Curl always returns same 404 page

No matter what URL I specify for curl I always get the same HTML 404 Error page back. If I use the --verbose option, it looks like curl always connects to the same IP address. $ curl --verbose ...
0
votes
1answer
470 views

wget and connection errors / timeouts

I was using wget the in the last week to recursively download a whole website of html pages. I used it this way: wget --recursive --no-clobber --page-requisites --html-extension --convert-links ...
-3
votes
3answers
1k views

Automatic download a site with *wget*: how to prevent? [closed]

How to prevent automatic download with wget for a website? Could you give links to articles please? Thanks.
2
votes
2answers
1k views

what is the default rate unit in wget?

When I use wget to download a large file, it gives me "56M/s". I want to know if it means 56 megabits per second or 56 megabytes per second. Thanks! Like this: 231,997,440 61.4M/s in 4.0s
-1
votes
1answer
677 views

wget --timeout doesn't time out an active connection

When using --timeout with wget, I was expecting that if the transaction hasn't completed within the timeout period, then it would abort. However, it seems to mean (with respect to read-timeout) that ...
1
vote
4answers
3k views

Why is wget hanging?

wget to a single specific url from one of my servers keeps getting timeouts. All other urls from this box work fine. This url works OK from any other boxes I have. Here's the output: wget -T 10 ...
3
votes
1answer
1k views

Can wget convert links within a local html document?

I know that wget can fetch a remote page and its dependencies and rewrite the html so that image src attributes reference the newly downloaded images. I am trying to convert local html files that ...
0
votes
3answers
2k views

download only new files from ftp using WGET/cUrl

i have Windows 2008 Server on which i store DB backups on daily basis. I want to be able to download new files only using wget, curl or windows built-in FTP doesn't matter. Can you help me with ...
0
votes
1answer
249 views

Am I doing ubuntu 10.0.4 software installation correctly (packages and none-packages)?

Background: I first installed everything using a terminal and "sudo su", i.e. as root. Then I installed jenkins, and then the jenkins user could not run any android sdk tools in /root/androidsdk/tools ...
1
vote
2answers
363 views

Download file in Linux when file location is unknown

Sorry if this is unclear, but I'm trying to set up a script that downloads a file. Currently, my method of downloading the file is by clicking on a link like so: ...