GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.

learn more… | top users | synonyms

2
votes
4answers
2k views

Escape a ! in the password parameter of wget

I'm trying to execute something like this: wget --user=foo --password=bar! url The ! in the password is causing problems. I've tried escaping it with \, as in --password=bar\! I've tried ...
2
votes
4answers
4k views

curl succeeds on an https request, wget does not

When I run curl from the command line curl "https://example.com" It succeeds immediately, returning the result of the request. When I run the same wget command wget https://example.com ...
2
votes
2answers
523 views

wget 5xx error code handling

i'm running a shell script that uses wget and pulls something over the web, my question is is there a way to tell wget to abort as soon as it hits 500 error code?
2
votes
3answers
1k views

Reboot if tomcat7 service is not responding

I am running a web app on a Tomcat server. There is a hard-to-detect problem within the server code that causes it to crash once or twice everyday. I will dig in to correct it when I have time. But ...
2
votes
1answer
162 views

Find out what's running wget on my server

In my access.log I see requests coming from 127.0.0.1 and retrieving server's "/" and then all of graphics and CSS referenced in it. This happens regularly, every 5 minutes. I want to find out what, ...
2
votes
2answers
2k views

WGet or cURL: Mirror Site from http://site.com And No Internal Access

I have tried wget -m wget -r and a whole bunch of variations. I am getting some of the images on http://site.com, one of the scripts, and none of the CSS, even with the fscking -p parameter. The ...
2
votes
1answer
3k views

HTTP downloads stop after some time, resuming is not possible

When I try to download a file via HTTP, the downloads sometimes stop after around 30 MB. The download rates goes down to 0 B/s and no data keeps coming. When I stop the download and resume again, the ...
2
votes
1answer
198 views

ShellShock test shows wget and curl access

I've fixed shellshock bug on my Debian 6 server and while testing on http://shellshock.brandonpotter.com/ I get "No Vulnerabilities Found" and that's OK but they also check other things and in the ...
2
votes
1answer
786 views

curl and wget attempting an ipv6 connection

On my dedicated server (hosted by OVH), that is running a fresh install of Ubuntu 14.04, curl and wget take approximately 10 seconds to complete a simple request. $ curl -v google.com * Rebuilt URL ...
2
votes
3answers
287 views

wget and port forwarding

I have a weird problem. I can run this command just fine from various servers: wget --debug '--http-user=USER123' '--http-passwd=PASSWORD' http://GW-BOX:9091/weijhkdsvn/v9_odbc//CRONTAB.2014020 ...
2
votes
7answers
4k views

Wget, Curl, Yum Fail but Ping works - CentOS 5

We currently have three webservers. Servers one and two behave, but I'm having real problems with the third. wget, curl and yum all fail to make their connections - that is, they all hang after ...
2
votes
3answers
1k views

The --reject feature in wget doesn't seem to work

I am trying to mirror most of a ftp sever using wget. This particular sever keeps old copies of the data files in a folder called 'backup' inside every subfolder. e.g. '/MaffiaOffShore/backup' ...
1
vote
4answers
519 views

download files from text file that contain files list under linux

i have a text file that contains a list of urls for files of my word for example : http://domain.com/file1.zip http://domain.com/file2.zip http://domain.com/file3.zip http://domain.com/file4.zip ...
1
vote
3answers
2k views

Downloading php files from python simple http server

I started python -m SimpleHTTPServer on one computer on lan and used wget to download php files from it to another. As far as i see, they seem to be downloaded correctly - i got php sources instead of ...
1
vote
3answers
4k views

Execute curl requests in parallel in bash

What is the best way to execute 5 curl requests in parallel from a bash script? I can't run them in serial for performance reasons.
1
vote
3answers
501 views

error while executing bash script -> command not found

i made this script here, and get the error IA-exporto.sh: 13: wget: not found i tried changing the " by ` and mixed all together and rearranged, but it just won't do.. #!/bin/bash ...
1
vote
3answers
883 views

Setting wget permission to 755 so users other than root can execute it a big security risk?

I read recently in blogs that by default wget on linux is 750, so only root can execute it. I would like to allow users wget and change it to 755, but I read around the web that it is a big security ...
1
vote
1answer
134 views

wget only works for some users

Currently have a strange situation: wget only works for one user on a system -- and root isn't it. There are three user accounts, user A, B and root. User A can use wget with no problems. User B and ...
1
vote
2answers
1k views

Cron not running commands as root

I have a file called /scripts/checkInternet and it contains: #!/bin/bash WGET="/usr/bin/wget" rm /tmp/index.google $WGET -q --tries=10 --timeout=5 http://www.google.com -O ...
1
vote
2answers
365 views

Using wget and Awk to count similar expressions

I am trying to create a script that uses wget to download a data set and then awk to sort though the file and tell you the most common filter used which is $14 column. So far I have the wget function ...
1
vote
4answers
243 views

Internet speed in Linux via command

I am running several sites and I was trying wget and I noticed slow speed. So I suspect that my sites are taking considerable amount of traffic. Question is, how I can really detect the total ...
1
vote
2answers
129 views

Has anyone experienced unintentional traffic overload due to clients' use of wget?

I am wondering if it is wise to indiscriminately block wget on servers that I maintain. Has anyone experienced unintentional traffic overload due to clients' use of wget?
1
vote
3answers
169 views

Limiting specific process to just one per user

My scripts are using wget to retrieve data from Internet. When many users are using this script I got very high load (about 20.00), because of disk I/O. Wget is automatically started each hour by ...
1
vote
2answers
479 views

Can someone explain to me what “wget -O - -q icanhazip.com” mean?

I've used this command to find the IP of the server. Can someone explain what the command means? I want to learn. So far I know: "wget" is a free utility for non-interactive download of files from ...
1
vote
4answers
734 views

wget an remote website index directory

Is it possible to wget an entire index/parent directory with wget? Because I am trying to transfer images to a VPS of mine from my old shared host. When I say an index/parent directory, I meant like ...
1
vote
3answers
15k views

How to install wget on this?

I did download RubyStack 2.0.3 for VMWare (Ubuntu 9.10) but I cannot download anything on it! It appears that all basic utilities are missing/screwed: bitnami@linux:/var/tmp$ wget -bash: wget: ...
1
vote
2answers
405 views

On Debian server, downloads stall out after one or two seconds using apt-get

I have a Debian server that is used primarily as a very low-traffic web server. Recently it began having some trouble regarding its network connection. I am able to use SFTP to upload a file to this ...
1
vote
1answer
2k views

Wget from HTTPS url using user/password -still getting 400 error

I'm trying to run Wget for a certain HTTP url, also using http-user and http-password command line options (tried the --user and --password, same results). Using the --no-check-certificate options as ...
1
vote
3answers
2k views

wget is working only when used with sudo

I'm having quite a strange behavior with wget since yesterday. I can download files by using sudo wget, but when I try the same file with only wget, I can get this error: yusufh@ubuntu-yuh:~$ wget ...
1
vote
2answers
520 views

How to untar a single file from an archive with randomly generated directory names?

I'm trying write a script that will download a tarball from github and extract a single file from it. However the top level directory inside the tarball has some random characters in it, which I ...
1
vote
2answers
410 views

Download file in Linux when file location is unknown

Sorry if this is unclear, but I'm trying to set up a script that downloads a file. Currently, my method of downloading the file is by clicking on a link like so: ...
1
vote
3answers
4k views

wget: download a website which is password protected using a form

I have a problem with downloading a website using wget. First I need to login to the website with a username and password: wget --save-cookies cookies.txt --post-data --cookies=on ...
1
vote
1answer
2k views

Using lynx or wget in cron to call a php script?

I'm examining a particular setup, and they have their crontab as follows: 0 * * * * lynx http://www.example.com/cron/scriptA.php Of course, this relies on the 'security-by-obscurity' precept, ...
1
vote
3answers
363 views

File timestamp not changing even with touch

I have the following problem. I have a bash script that uses wget to get a few files from the build server, and then scp's them over to the production system: Here is the relevant snippet: #Several ...
1
vote
1answer
268 views

Hourly CRON task running more frequently than one hour

I have a cron task that calls a special PHP script via wget. Here is the crontab entry: 0 * * * * wget http://www.... It will work perfect for several days, running on the hour. However, after a few ...
1
vote
1answer
90 views

Under what scenarios would Apache allow a mismatched HTTP_HOST to pass through?

Someone has managed to pass an undefined HTTP_HOST server variable to my application script, triggering a series of errors. I am quite perturbed but am unable to replicate this behaviour. My httpd ...
1
vote
1answer
88 views

Wget and recursive downloads

A directory listings enabled folder has 2 zip files. I am trying to download only one of them (the latest). Problem is the zip file name is a random string. The command I have so far is wget -r -nH ...
1
vote
1answer
186 views

Ghost running on docker container does not respond to http requests from other container

I am setting up a ghost blog instance inside a docker container. I bound the ghost server to host 0.0.0.0 and exposed the port on which it's listening (port 2368). If I run wget to the ghost ...
1
vote
1answer
155 views

Command line tools to monitor POST/GET requests of a loaded website

When I load a website with Firefox and open up the Network Profiler, I can see all the POST and GET requests the site is making. I would like this functionality from the command line. Does any one ...
1
vote
2answers
56 views

Download and save a file with proof that it was downloaded from a specific server/url on a specific date?

Assuming the file was downloaded over http is this possible? I'm guessing there could be some way to do it using a hash and a timestamp however I'm not sure how you could prove nothing was tampered ...
1
vote
3answers
192 views

Problems with wget AWS ubuntu apache2

I am trying to use wget to download from my AWS ubuntu server with apache2. I have tried several different options but they all result in either a file by the directory name, or an index.html file. ...
1
vote
1answer
370 views

iptables redirect tcp to checkip.dyndns.org from localhost to 127.0.0.1:8118

I've tried several different combinations of rules and nothing seems to be working. I know that you can't use prerouting table for a request coming from the localhost so I used the output table and ...
1
vote
3answers
2k views

Download Git Zipball in Unix

I'm trying to download a zipball of a git repo: e.g. wget https://github.com/zeromq/jzmq/zipball/master This works fine in a web browser but on unix the file gets a weird name...how do I do this?
1
vote
1answer
545 views

How to block some robots and referer pages in Apache root level

When I was using Lighttpd I can easily achieve this by such entries. So all websites were protected. Wget robots: $HTTP["useragent"] =~ "Wget" { $HTTP["url"] =~ "^/tagi(.*)" { # ...
1
vote
1answer
434 views

Translate web page from shell

I'm trying to do translate a web page automatically with a shell script. Here is the interesting part: wget http://www.google.com/translate?langpair=en%7Ces&u=http%3A%2F%2Fasdf.com I'm getting ...
1
vote
2answers
125 views

wget has different response times for same fetch

Any ideas why one fetch would fetch a file (big picture in this example) fast while another fetch would fetch same file slow? Not seeing this issue at home. Not seeing this issue through ip ...
1
vote
1answer
740 views

wget not getting all files

I have issues using wget, basically I use it like: wget -m ftp://username:password@hostname.com:/path/to/dirs After everything's complete (it says xxx files in xx MB for xx seconds), however, when ...
1
vote
1answer
950 views

wget mirroring, subdomains and directories and cookies

I have an account on a web page that is now "full" (ie I have used up all my allocated space) and I would like to make a mirror of that site. wget seems like the thing to use. The problem is that I ...
1
vote
1answer
133 views

Why does wget from ftp-sites sometimes create a few empty files when it downloads the rest of the files without problems?

I have downloaded 80.000+ image files stored on a FTP-site using wget(1) and most of the files where downloaded without problems but when I did a small check (using file(1)) I just realized that a few ...
1
vote
1answer
173 views

SElinux stopping LVS from working with https

LVS/piranha is setup and trying to get it to balance https instead of http. Setup https testing with wget - idea from this link. Works when I do it at the command prompt. With SELinux enforcing, ...