GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.

learn more… | top users | synonyms

2
votes
2answers
173 views

how can i use apache log files to recreate usage scenario

Recently i installed a website that had too many requests and it was too slow. Many improvements have been made to the web site code and we've also bought a new server. I want to test the new server ...
2
votes
4answers
6k views

Why is wget hanging?

wget to a single specific url from one of my servers keeps getting timeouts. All other urls from this box work fine. This url works OK from any other boxes I have. Here's the output: wget -T 10 ...
2
votes
2answers
1k views

Why is this bash command not echoing into a variable and what can I do to improve?

I've got a bash script I'm working to improve, and have put on a great fix thanks to Dennis Williamson. Unfortunately, one of the lines no longer echoes into a variable I can manipulate, rather dumps ...
2
votes
1answer
242 views

Why does the time it takes to run this script differ so much between user and system and how I might correct the discrepancy?

My application sits behind a load balancer, and every once in a while I like to do a status check on each machine to get an idea of the time it takes to return an index.html document on each machine. ...
2
votes
2answers
2k views

Retrieve FTP directory tree from the command line

What would be the fastest way to recursively retrieve entire directory listing from an ftp server using wget/curl/whatever? I don't need to download any files, just directory and file names. Basically ...
2
votes
1answer
261 views

How to prevent external website from being downloaded in wget?

I'm using wget to download useful website recursively: wget -k -m -r -q -t 1 http://www.web.com/ But external website also downloaded along with the website I want. How to prevent this external ...
2
votes
1answer
62 views

wget - Only save if return code is 200, delete otherwise

I have a script that checks my public ip address every few minutes. The problem is the ISP sometimes gives me cached pages (I know, I've used all the related args in wget, the isp is formed by a bunch ...
2
votes
1answer
2k views

Using lynx or wget in cron to call a php script?

I'm examining a particular setup, and they have their crontab as follows: 0 * * * * lynx http://www.example.com/cron/scriptA.php Of course, this relies on the 'security-by-obscurity' precept, ...
2
votes
2answers
559 views

wget 5xx error code handling

i'm running a shell script that uses wget and pulls something over the web, my question is is there a way to tell wget to abort as soon as it hits 500 error code?
2
votes
3answers
2k views

Reboot if tomcat7 service is not responding

I am running a web app on a Tomcat server. There is a hard-to-detect problem within the server code that causes it to crash once or twice everyday. I will dig in to correct it when I have time. But ...
2
votes
1answer
172 views

Find out what's running wget on my server

In my access.log I see requests coming from 127.0.0.1 and retrieving server's "/" and then all of graphics and CSS referenced in it. This happens regularly, every 5 minutes. I want to find out what, ...
2
votes
2answers
3k views

WGet or cURL: Mirror Site from http://site.com And No Internal Access

I have tried wget -m wget -r and a whole bunch of variations. I am getting some of the images on http://site.com, one of the scripts, and none of the CSS, even with the fscking -p parameter. The ...
2
votes
1answer
4k views

HTTP downloads stop after some time, resuming is not possible

When I try to download a file via HTTP, the downloads sometimes stop after around 30 MB. The download rates goes down to 0 B/s and no data keeps coming. When I stop the download and resume again, the ...
2
votes
1answer
424 views

ShellShock test shows wget and curl access

I've fixed shellshock bug on my Debian 6 server and while testing on http://shellshock.brandonpotter.com/ I get "No Vulnerabilities Found" and that's OK but they also check other things and in the ...
2
votes
3answers
382 views

wget and port forwarding

I have a weird problem. I can run this command just fine from various servers: wget --debug '--http-user=USER123' '--http-passwd=PASSWORD' http://GW-BOX:9091/weijhkdsvn/v9_odbc//CRONTAB.2014020 ...
2
votes
0answers
283 views

How to use sitemap.xml to create a static mirror of a CMS

Is there a tool to create a static mirror of a content management system (CMS) that provides a sitemap.xml file? Ideally, I would point a tool like wget or curl to a sitemap.xml file and have it ...
2
votes
3answers
1k views

The --reject feature in wget doesn't seem to work

I am trying to mirror most of a ftp sever using wget. This particular sever keeps old copies of the data files in a folder called 'backup' inside every subfolder. e.g. '/MaffiaOffShore/backup' ...
1
vote
4answers
742 views

download files from text file that contain files list under linux

i have a text file that contains a list of urls for files of my word for example : http://domain.com/file1.zip http://domain.com/file2.zip http://domain.com/file3.zip http://domain.com/file4.zip ...
1
vote
3answers
2k views

Downloading php files from python simple http server

I started python -m SimpleHTTPServer on one computer on lan and used wget to download php files from it to another. As far as i see, they seem to be downloaded correctly - i got php sources instead of ...
1
vote
3answers
621 views

error while executing bash script -> command not found

i made this script here, and get the error IA-exporto.sh: 13: wget: not found i tried changing the " by ` and mixed all together and rearranged, but it just won't do.. #!/bin/bash ...
1
vote
3answers
1k views

Setting wget permission to 755 so users other than root can execute it a big security risk?

I read recently in blogs that by default wget on linux is 750, so only root can execute it. I would like to allow users wget and change it to 755, but I read around the web that it is a big security ...
1
vote
1answer
147 views

wget only works for some users

Currently have a strange situation: wget only works for one user on a system -- and root isn't it. There are three user accounts, user A, B and root. User A can use wget with no problems. User B and ...
1
vote
2answers
1k views

Cron not running commands as root

I have a file called /scripts/checkInternet and it contains: #!/bin/bash WGET="/usr/bin/wget" rm /tmp/index.google $WGET -q --tries=10 --timeout=5 http://www.google.com -O ...
1
vote
2answers
140 views

Has anyone experienced unintentional traffic overload due to clients' use of wget?

I am wondering if it is wise to indiscriminately block wget on servers that I maintain. Has anyone experienced unintentional traffic overload due to clients' use of wget?
1
vote
2answers
789 views

“Issued certificate not yet valid.” with wget?

I am getting the following error when using wget to fetch a file: Issued certificate not yet valid. To connect to www.atlassian.com insecurely, use ‘--no-check-certificate’. Testing with openssl ...
1
vote
2answers
473 views

Using wget and Awk to count similar expressions

I am trying to create a script that uses wget to download a data set and then awk to sort though the file and tell you the most common filter used which is $14 column. So far I have the wget function ...
1
vote
4answers
251 views

Internet speed in Linux via command

I am running several sites and I was trying wget and I noticed slow speed. So I suspect that my sites are taking considerable amount of traffic. Question is, how I can really detect the total ...
1
vote
3answers
184 views

Limiting specific process to just one per user

My scripts are using wget to retrieve data from Internet. When many users are using this script I got very high load (about 20.00), because of disk I/O. Wget is automatically started each hour by ...
1
vote
2answers
743 views

Can someone explain to me what “wget -O - -q icanhazip.com” mean?

I've used this command to find the IP of the server. Can someone explain what the command means? I want to learn. So far I know: "wget" is a free utility for non-interactive download of files from ...
1
vote
4answers
993 views

wget an remote website index directory

Is it possible to wget an entire index/parent directory with wget? Because I am trying to transfer images to a VPS of mine from my old shared host. When I say an index/parent directory, I meant like ...
1
vote
3answers
20k views

How to install wget on this?

I did download RubyStack 2.0.3 for VMWare (Ubuntu 9.10) but I cannot download anything on it! It appears that all basic utilities are missing/screwed: bitnami@linux:/var/tmp$ wget -bash: wget: ...
1
vote
1answer
3k views

Wget from HTTPS url using user/password -still getting 400 error

I'm trying to run Wget for a certain HTTP url, also using http-user and http-password command line options (tried the --user and --password, same results). Using the --no-check-certificate options as ...
1
vote
3answers
5k views

wget: download a website which is password protected using a form

I have a problem with downloading a website using wget. First I need to login to the website with a username and password: wget --save-cookies cookies.txt --post-data --cookies=on ...
1
vote
3answers
2k views

wget is working only when used with sudo

I'm having quite a strange behavior with wget since yesterday. I can download files by using sudo wget, but when I try the same file with only wget, I can get this error: yusufh@ubuntu-yuh:~$ wget ...
1
vote
2answers
682 views

How to untar a single file from an archive with randomly generated directory names?

I'm trying write a script that will download a tarball from github and extract a single file from it. However the top level directory inside the tarball has some random characters in it, which I ...
1
vote
2answers
494 views

Download file in Linux when file location is unknown

Sorry if this is unclear, but I'm trying to set up a script that downloads a file. Currently, my method of downloading the file is by clicking on a link like so: ...
1
vote
3answers
391 views

File timestamp not changing even with touch

I have the following problem. I have a bash script that uses wget to get a few files from the build server, and then scp's them over to the production system: Here is the relevant snippet: #Several ...
1
vote
1answer
282 views

Hourly CRON task running more frequently than one hour

I have a cron task that calls a special PHP script via wget. Here is the crontab entry: 0 * * * * wget http://www.... It will work perfect for several days, running on the hour. However, after a few ...
1
vote
1answer
423 views

On Debian server, downloads stall out after one or two seconds using apt-get

I have a Debian server that is used primarily as a very low-traffic web server. Recently it began having some trouble regarding its network connection. I am able to use SFTP to upload a file to this ...
1
vote
1answer
24 views

Wget, abort retrying after failure or timeout

I'm calling a url with wget: /usr/bin/wget --read-timeout=7200 https://site_url/s Wget performs a GET request every 15 minutes in this case, despite the timeout being set, why does this happen? ...
1
vote
1answer
1k views

How to make wget trust my self signed certificate (without using --no-check-certificate)? [duplicate]

Ubuntu 12.04 OpenSSL 1.0.1 14 Wget 1.13.4 My setup: create our own CA (our_own_ca.crt) generate a certificate which is signed with the above CA (graphite.local.crt) Concatenate that cert and the ...
1
vote
1answer
234 views

Under what scenarios would Apache allow a mismatched HTTP_HOST to pass through?

Someone has managed to pass an undefined HTTP_HOST server variable to my application script, triggering a series of errors. I am quite perturbed but am unable to replicate this behaviour. My httpd ...
1
vote
1answer
112 views

Wget and recursive downloads

A directory listings enabled folder has 2 zip files. I am trying to download only one of them (the latest). Problem is the zip file name is a random string. The command I have so far is wget -r -nH ...
1
vote
1answer
309 views

Ghost running on docker container does not respond to http requests from other container

I am setting up a ghost blog instance inside a docker container. I bound the ghost server to host 0.0.0.0 and exposed the port on which it's listening (port 2368). If I run wget to the ghost ...
1
vote
1answer
69 views

Use tmux for managing multiple downloaders as a supervisorctl service?

I have many data-servers I need to download data from via http as soon as it is available. For each server I start a bash "while true"-loop and within that a wget to poll the server for new data. To ...
1
vote
1answer
258 views

Command line tools to monitor POST/GET requests of a loaded website

When I load a website with Firefox and open up the Network Profiler, I can see all the POST and GET requests the site is making. I would like this functionality from the command line. Does any one ...
1
vote
1answer
4k views

SSL error curl/wget unknown protocol/wrong version number

I've been trying to use local hosted https urls in command line/cron jobs and i get those errors. curl output is similar. The same commands, when used in other servers works perfectly. Default ...
1
vote
1answer
570 views

Why would an HTTP(S) request not be logged in IIS?

I'm troubleshooting a batch file that uses Wget to send requests to a website running in IIS 7. The batch file runs Wget twice, the first time to login to the site via a POST, the second to run some ...
1
vote
2answers
64 views

Download and save a file with proof that it was downloaded from a specific server/url on a specific date?

Assuming the file was downloaded over http is this possible? I'm guessing there could be some way to do it using a hash and a timestamp however I'm not sure how you could prove nothing was tampered ...
1
vote
3answers
214 views

Problems with wget AWS ubuntu apache2

I am trying to use wget to download from my AWS ubuntu server with apache2. I have tried several different options but they all result in either a file by the directory name, or an index.html file. ...