GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.

learn more… | top users | synonyms

2
votes
2answers
1k views

Retrieve FTP directory tree from the command line

What would be the fastest way to recursively retrieve entire directory listing from an ftp server using wget/curl/whatever? I don't need to download any files, just directory and file names. Basically ...
2
votes
1answer
259 views

How to prevent external website from being downloaded in wget?

I'm using wget to download useful website recursively: wget -k -m -r -q -t 1 http://www.web.com/ But external website also downloaded along with the website I want. How to prevent this external ...
2
votes
2answers
542 views

wget 5xx error code handling

i'm running a shell script that uses wget and pulls something over the web, my question is is there a way to tell wget to abort as soon as it hits 500 error code?
2
votes
3answers
2k views

Reboot if tomcat7 service is not responding

I am running a web app on a Tomcat server. There is a hard-to-detect problem within the server code that causes it to crash once or twice everyday. I will dig in to correct it when I have time. But ...
2
votes
1answer
167 views

Find out what's running wget on my server

In my access.log I see requests coming from 127.0.0.1 and retrieving server's "/" and then all of graphics and CSS referenced in it. This happens regularly, every 5 minutes. I want to find out what, ...
2
votes
2answers
2k views

WGet or cURL: Mirror Site from http://site.com And No Internal Access

I have tried wget -m wget -r and a whole bunch of variations. I am getting some of the images on http://site.com, one of the scripts, and none of the CSS, even with the fscking -p parameter. The ...
2
votes
1answer
4k views

HTTP downloads stop after some time, resuming is not possible

When I try to download a file via HTTP, the downloads sometimes stop after around 30 MB. The download rates goes down to 0 B/s and no data keeps coming. When I stop the download and resume again, the ...
2
votes
1answer
353 views

ShellShock test shows wget and curl access

I've fixed shellshock bug on my Debian 6 server and while testing on http://shellshock.brandonpotter.com/ I get "No Vulnerabilities Found" and that's OK but they also check other things and in the ...
2
votes
3answers
313 views

wget and port forwarding

I have a weird problem. I can run this command just fine from various servers: wget --debug '--http-user=USER123' '--http-passwd=PASSWORD' http://GW-BOX:9091/weijhkdsvn/v9_odbc//CRONTAB.2014020 ...
2
votes
7answers
5k views

Wget, Curl, Yum Fail but Ping works - CentOS 5

We currently have three webservers. Servers one and two behave, but I'm having real problems with the third. wget, curl and yum all fail to make their connections - that is, they all hang after ...
2
votes
3answers
1k views

The --reject feature in wget doesn't seem to work

I am trying to mirror most of a ftp sever using wget. This particular sever keeps old copies of the data files in a folder called 'backup' inside every subfolder. e.g. '/MaffiaOffShore/backup' ...
2
votes
1answer
1k views

curl and wget attempting an ipv6 connection

On my dedicated server (hosted by OVH), that is running a fresh install of Ubuntu 14.04, curl and wget take approximately 10 seconds to complete a simple request. $ curl -v google.com * Rebuilt URL ...
1
vote
3answers
5k views

Execute curl requests in parallel in bash

What is the best way to execute 5 curl requests in parallel from a bash script? I can't run them in serial for performance reasons.
1
vote
4answers
585 views

download files from text file that contain files list under linux

i have a text file that contains a list of urls for files of my word for example : http://domain.com/file1.zip http://domain.com/file2.zip http://domain.com/file3.zip http://domain.com/file4.zip ...
1
vote
3answers
2k views

Downloading php files from python simple http server

I started python -m SimpleHTTPServer on one computer on lan and used wget to download php files from it to another. As far as i see, they seem to be downloaded correctly - i got php sources instead of ...
1
vote
3answers
526 views

error while executing bash script -> command not found

i made this script here, and get the error IA-exporto.sh: 13: wget: not found i tried changing the " by ` and mixed all together and rearranged, but it just won't do.. #!/bin/bash ...
1
vote
3answers
1k views

Setting wget permission to 755 so users other than root can execute it a big security risk?

I read recently in blogs that by default wget on linux is 750, so only root can execute it. I would like to allow users wget and change it to 755, but I read around the web that it is a big security ...
1
vote
1answer
136 views

wget only works for some users

Currently have a strange situation: wget only works for one user on a system -- and root isn't it. There are three user accounts, user A, B and root. User A can use wget with no problems. User B and ...
1
vote
2answers
1k views

Cron not running commands as root

I have a file called /scripts/checkInternet and it contains: #!/bin/bash WGET="/usr/bin/wget" rm /tmp/index.google $WGET -q --tries=10 --timeout=5 http://www.google.com -O ...
1
vote
2answers
395 views

Using wget and Awk to count similar expressions

I am trying to create a script that uses wget to download a data set and then awk to sort though the file and tell you the most common filter used which is $14 column. So far I have the wget function ...
1
vote
4answers
248 views

Internet speed in Linux via command

I am running several sites and I was trying wget and I noticed slow speed. So I suspect that my sites are taking considerable amount of traffic. Question is, how I can really detect the total ...
1
vote
2answers
132 views

Has anyone experienced unintentional traffic overload due to clients' use of wget?

I am wondering if it is wise to indiscriminately block wget on servers that I maintain. Has anyone experienced unintentional traffic overload due to clients' use of wget?
1
vote
3answers
175 views

Limiting specific process to just one per user

My scripts are using wget to retrieve data from Internet. When many users are using this script I got very high load (about 20.00), because of disk I/O. Wget is automatically started each hour by ...
1
vote
2answers
598 views

Can someone explain to me what “wget -O - -q icanhazip.com” mean?

I've used this command to find the IP of the server. Can someone explain what the command means? I want to learn. So far I know: "wget" is a free utility for non-interactive download of files from ...
1
vote
4answers
841 views

wget an remote website index directory

Is it possible to wget an entire index/parent directory with wget? Because I am trying to transfer images to a VPS of mine from my old shared host. When I say an index/parent directory, I meant like ...
1
vote
3answers
17k views

How to install wget on this?

I did download RubyStack 2.0.3 for VMWare (Ubuntu 9.10) but I cannot download anything on it! It appears that all basic utilities are missing/screwed: bitnami@linux:/var/tmp$ wget -bash: wget: ...
1
vote
1answer
2k views

Wget from HTTPS url using user/password -still getting 400 error

I'm trying to run Wget for a certain HTTP url, also using http-user and http-password command line options (tried the --user and --password, same results). Using the --no-check-certificate options as ...
1
vote
3answers
2k views

wget is working only when used with sudo

I'm having quite a strange behavior with wget since yesterday. I can download files by using sudo wget, but when I try the same file with only wget, I can get this error: yusufh@ubuntu-yuh:~$ wget ...
1
vote
2answers
586 views

How to untar a single file from an archive with randomly generated directory names?

I'm trying write a script that will download a tarball from github and extract a single file from it. However the top level directory inside the tarball has some random characters in it, which I ...
1
vote
2answers
435 views

Download file in Linux when file location is unknown

Sorry if this is unclear, but I'm trying to set up a script that downloads a file. Currently, my method of downloading the file is by clicking on a link like so: ...
1
vote
3answers
4k views

wget: download a website which is password protected using a form

I have a problem with downloading a website using wget. First I need to login to the website with a username and password: wget --save-cookies cookies.txt --post-data --cookies=on ...
1
vote
1answer
2k views

Using lynx or wget in cron to call a php script?

I'm examining a particular setup, and they have their crontab as follows: 0 * * * * lynx http://www.example.com/cron/scriptA.php Of course, this relies on the 'security-by-obscurity' precept, ...
1
vote
3answers
377 views

File timestamp not changing even with touch

I have the following problem. I have a bash script that uses wget to get a few files from the build server, and then scp's them over to the production system: Here is the relevant snippet: #Several ...
1
vote
1answer
270 views

Hourly CRON task running more frequently than one hour

I have a cron task that calls a special PHP script via wget. Here is the crontab entry: 0 * * * * wget http://www.... It will work perfect for several days, running on the hour. However, after a few ...
1
vote
1answer
418 views

On Debian server, downloads stall out after one or two seconds using apt-get

I have a Debian server that is used primarily as a very low-traffic web server. Recently it began having some trouble regarding its network connection. I am able to use SFTP to upload a file to this ...
1
vote
1answer
67 views

How to make wget trust my self signed certificate (without using --no-check-certificate)? [duplicate]

Ubuntu 12.04 OpenSSL 1.0.1 14 Wget 1.13.4 My setup: create our own CA (our_own_ca.crt) generate a certificate which is signed with the above CA (graphite.local.crt) Concatenate that cert and the ...
1
vote
1answer
146 views

Under what scenarios would Apache allow a mismatched HTTP_HOST to pass through?

Someone has managed to pass an undefined HTTP_HOST server variable to my application script, triggering a series of errors. I am quite perturbed but am unable to replicate this behaviour. My httpd ...
1
vote
1answer
102 views

Wget and recursive downloads

A directory listings enabled folder has 2 zip files. I am trying to download only one of them (the latest). Problem is the zip file name is a random string. The command I have so far is wget -r -nH ...
1
vote
1answer
217 views

Ghost running on docker container does not respond to http requests from other container

I am setting up a ghost blog instance inside a docker container. I bound the ghost server to host 0.0.0.0 and exposed the port on which it's listening (port 2368). If I run wget to the ghost ...
1
vote
1answer
55 views

Use tmux for managing multiple downloaders as a supervisorctl service?

I have many data-servers I need to download data from via http as soon as it is available. For each server I start a bash "while true"-loop and within that a wget to poll the server for new data. To ...
1
vote
1answer
175 views

Command line tools to monitor POST/GET requests of a loaded website

When I load a website with Firefox and open up the Network Profiler, I can see all the POST and GET requests the site is making. I would like this functionality from the command line. Does any one ...
1
vote
1answer
3k views

SSL error curl/wget unknown protocol/wrong version number

I've been trying to use local hosted https urls in command line/cron jobs and i get those errors. curl output is similar. The same commands, when used in other servers works perfectly. Default ...
1
vote
1answer
417 views

Why would an HTTP(S) request not be logged in IIS?

I'm troubleshooting a batch file that uses Wget to send requests to a website running in IIS 7. The batch file runs Wget twice, the first time to login to the site via a POST, the second to run some ...
1
vote
2answers
58 views

Download and save a file with proof that it was downloaded from a specific server/url on a specific date?

Assuming the file was downloaded over http is this possible? I'm guessing there could be some way to do it using a hash and a timestamp however I'm not sure how you could prove nothing was tampered ...
1
vote
3answers
202 views

Problems with wget AWS ubuntu apache2

I am trying to use wget to download from my AWS ubuntu server with apache2. I have tried several different options but they all result in either a file by the directory name, or an index.html file. ...
1
vote
1answer
415 views

iptables redirect tcp to checkip.dyndns.org from localhost to 127.0.0.1:8118

I've tried several different combinations of rules and nothing seems to be working. I know that you can't use prerouting table for a request coming from the localhost so I used the output table and ...
1
vote
3answers
2k views

Download Git Zipball in Unix

I'm trying to download a zipball of a git repo: e.g. wget https://github.com/zeromq/jzmq/zipball/master This works fine in a web browser but on unix the file gets a weird name...how do I do this?
1
vote
1answer
570 views

How to block some robots and referer pages in Apache root level

When I was using Lighttpd I can easily achieve this by such entries. So all websites were protected. Wget robots: $HTTP["useragent"] =~ "Wget" { $HTTP["url"] =~ "^/tagi(.*)" { # ...
1
vote
1answer
472 views

Translate web page from shell

I'm trying to do translate a web page automatically with a shell script. Here is the interesting part: wget http://www.google.com/translate?langpair=en%7Ces&u=http%3A%2F%2Fasdf.com I'm getting ...
1
vote
2answers
127 views

wget has different response times for same fetch

Any ideas why one fetch would fetch a file (big picture in this example) fast while another fetch would fetch same file slow? Not seeing this issue at home. Not seeing this issue through ip ...