Questions tagged [wget]

GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.

Filter by
Sorted by
Tagged with
5 votes
4 answers
14k views

using wget through .pac config based proxy server

I want to use wget through a proxy which uses .pac config. When I googled a bit, I found that .pac is a javascript file and wget cannot parse it based on the following url http://www.linuxquestions....
gsk's user avatar
  • 153
5 votes
2 answers
23k views

Wget, self signed cert and --no-check-certificate not working

I recently installed a self signed SSL cert on one of our nginx webservers. If I attempt to wget a file with --no-check-certificate I receive the errors below. You can reach the site with a browser, ...
Minelr G's user avatar
5 votes
1 answer
15k views

curl and wget attempting an ipv6 connection

On my dedicated server (hosted by OVH), that is running a fresh install of Ubuntu 14.04, curl and wget take approximately 10 seconds to complete a simple request. $ curl -v google.com * Rebuilt URL ...
Daniel S's user avatar
  • 415
4 votes
3 answers
11k views

How to use "wget" to save a file as a different name

wget "http://flakejfl2ke3jrlwkefj38j2ofiwejfoifjalsdkfje3fklwje.exe" is long. How do I save it as my own name?
Alex's user avatar
  • 8,591
4 votes
2 answers
6k views

what is the default rate unit in wget?

When I use wget to download a large file, it gives me "56M/s". I want to know if it means 56 megabits per second or 56 megabytes per second. Thanks! Like this: 231,997,440 61.4M/s in 4.0s
larry's user avatar
  • 4,107
4 votes
1 answer
2k views

"wget --mirror" Alternative

I'm looking for an alternative to using this, it doesn't have to use FTP, but it should provide the same functionality as the "--mirror" option of wget where it only downloads new and changed files. ...
joebert's user avatar
  • 195
4 votes
1 answer
2k views

Downloading from wget - folder issue

I have a website - http://myradio1.localhost and the directory is currently empty. The virtual host works fine, no issues there. This is my command I'm using to download the site via wget: wget --...
avenas8808's user avatar
4 votes
1 answer
9k views

Cannot get cURL or wget to validate some SSL certificates

I've noticed that our link checker, which uses cURL, fails more and more often to validate SSL certificates. I'm trying to get to the bottom of this. https://www.bgetem.de/, for instance, opens just ...
user2323470's user avatar
4 votes
2 answers
8k views

Retrieve FTP directory tree from the command line

What would be the fastest way to recursively retrieve entire directory listing from an ftp server using wget/curl/whatever? I don't need to download any files, just directory and file names. Basically ...
user187291's user avatar
4 votes
5 answers
31k views

Fixing wget certificates

Basically, I was testing different versions of OpenSSL, and somewhere in the process broke wget's path to certificates. wget https://www.google.com Results now in: ERROR: cannot verify www.google....
Ilya's user avatar
  • 143
4 votes
4 answers
1k views

How do I reconcile Network Speed from Network Throughput?

I have been arguing with my ISP about the service they are providing; primarily with regard to throughput speeds. I have set up an edge router (see hardware details listed below) and on it I have ...
Lorin S.'s user avatar
4 votes
1 answer
27k views

How to make wget trust my self signed certificate (without using --no-check-certificate)? [duplicate]

Ubuntu 12.04 OpenSSL 1.0.1 14 Wget 1.13.4 My setup: create our own CA (our_own_ca.crt) generate a certificate which is signed with the above CA (graphite.local.crt) Concatenate that cert and the CA ...
quanta's user avatar
  • 51.8k
4 votes
0 answers
17k views

How to Enable JavaScript for wget in linux for grabbing a website?

I use wget like this for save site : wget --page-requisites --no-parent --mirror http://example.com/index.html -P /home/ in Some cases It does NOT work, Error is : This site requires Javascript to ...
AriaData's user avatar
3 votes
2 answers
31k views

Using curl with cookies

I hope someone can help me out on this one. I have been trying this for a while now, with all sorts of variations and cannot seem to get this to work. I am trying to script a way to log in to https://...
user1718443's user avatar
3 votes
4 answers
4k views

Download a file from the shell when button click is required

I have a link to a file that I would like to download from the shell. Unfortunately, the URL redirects to a software license agreement page which requires clicking an accept button. This is fine when ...
Brett's user avatar
  • 165
3 votes
3 answers
8k views

nginx block curl and wget

I have nginx webserver. I have a rich content site and i found that some malicious bots are trying to crawl my content. I blocked any curl or wget coming to my site like this if ($http_user_agent ~* ...
Alaa Alomari's user avatar
3 votes
2 answers
4k views

Can someone explain to me what "wget -O - -q icanhazip.com" mean?

I've used this command to find the IP of the server. Can someone explain what the command means? I want to learn. So far I know: "wget" is a free utility for non-interactive download of files from ...
user101699's user avatar
3 votes
1 answer
1k views

Cron Daemon Wget blocked

I've got nginx on Debian 7 without cpanel. I am seting-up my Crontab like this: */45 * * * * wget "http://example.com/cron-url.php" >/dev/null 2>&1 The above cron is being blocked with ...
Neel's user avatar
  • 1,471
3 votes
2 answers
643 views

pgrep wget: what is the details of process id?

when typing pgrep wget, it show process id 10144 but how to know what is the details of this process id
kopeklan's user avatar
  • 169
3 votes
2 answers
4k views

Curl always returns same 404 page

No matter what URL I specify for curl I always get the same HTML 404 Error page back. If I use the --verbose option, it looks like curl always connects to the same IP address. $ curl --verbose http:/...
Jon's user avatar
  • 161
3 votes
3 answers
9k views

Not able to wget to ftp server

I am trying to wget a ftp server from a remote machine. The command is not getting past 'Logging in as anonymous'. This is what i am getting. wget ftp://hgdownload.cse.ucsc.edu/goldenPath/hg19/...
Ashwin 's user avatar
  • 133
3 votes
2 answers
2k views

Why is this bash command not echoing into a variable and what can I do to improve?

I've got a bash script I'm working to improve, and have put on a great fix thanks to Dennis Williamson. Unfortunately, one of the lines no longer echoes into a variable I can manipulate, rather dumps ...
editor's user avatar
  • 383
3 votes
2 answers
10k views

HTTP downloads stop after some time, resuming is not possible

When I try to download a file via HTTP, the downloads sometimes stop after around 30 MB. The download rates goes down to 0 B/s and no data keeps coming. When I stop the download and resume again, the ...
cdauth's user avatar
  • 993
3 votes
1 answer
15k views

multiple post requests using wget and same base-url

wget has nice option that lets you allow downloading multiple files from same location (I mean combination of --base and --input-file) Advantage of this, is that if possible wget tries to reuse ...
GiM's user avatar
  • 131
3 votes
3 answers
14k views

why i cannot download jdk from oracle web site directly without AuthParam?

that is download with the following command, why it fails to download that file? wget http://download.oracle.com/otn-pub/java/jdk/6u35-b10/jdk-6u35-linux-i586.bin the following command works, but ...
giantforest's user avatar
3 votes
3 answers
5k views

Download Git Zipball in Unix

I'm trying to download a zipball of a git repo: e.g. wget https://github.com/zeromq/jzmq/zipball/master This works fine in a web browser but on unix the file gets a weird name...how do I do this?
DD.'s user avatar
  • 3,114
3 votes
3 answers
5k views

Reboot if tomcat7 service is not responding

I am running a web app on a Tomcat server. There is a hard-to-detect problem within the server code that causes it to crash once or twice everyday. I will dig in to correct it when I have time. But ...
Bahadir Ismail Aydin's user avatar
2 votes
3 answers
565 views

wget application replacement

Is there any other application similar to wget? If yes can you post a link?
zozo's user avatar
  • 823
2 votes
1 answer
9k views

How would I write a terminal command to download a folder with wget from a Media Temple (gs) server?

I'm trying to download a folder using wget on the Terminal (I'm usin a Mac if that matters) because my ftp client sucks and keeps timing out. It doesn't stay connected for long. So I was wondering if ...
user41157's user avatar
  • 189
2 votes
1 answer
3k views

wget download only certain folders from site

I want to download this site with wget, and to do this I'm using this command: wget.exe --no-clobber --no-parent -r --convert-links --page-requisites -U Mozilla "http://www.s-manuals.com/smd/" That'...
efirvida's user avatar
  • 141
2 votes
1 answer
134 views

Wget Minimum Download Rate

Can anyone help with the parameter that ensures that wget will fetch the file at a minimum specified download rate?
user152666's user avatar
2 votes
1 answer
770 views

Stream tar.gz file from FTP server

Here is the situation: I have a tar.gz file on a FTP server which can contain an arbitrary number of files. Now what I'm trying to accomplish is have this file streamed and uploaded to HDFS through a ...
Charles Menguy's user avatar
2 votes
2 answers
311 views

how can i use apache log files to recreate usage scenario

Recently i installed a website that had too many requests and it was too slow. Many improvements have been made to the web site code and we've also bought a new server. I want to test the new server ...
daigorocub's user avatar
2 votes
1 answer
277 views

Why does the time it takes to run this script differ so much between user and system and how I might correct the discrepancy?

My application sits behind a load balancer, and every once in a while I like to do a status check on each machine to get an idea of the time it takes to return an index.html document on each machine. ...
editor's user avatar
  • 383
2 votes
1 answer
342 views

How to prevent external website from being downloaded in wget?

I'm using wget to download useful website recursively: wget -k -m -r -q -t 1 http://www.web.com/ But external website also downloaded along with the website I want. How to prevent this external ...
user avatar
2 votes
2 answers
3k views

mirror a site with wget and download static media

I'd like to mirror a site with wget and convert all the links to the local copies I've downloaded. So far that's easy all I have to do is wget -mk http://site.com However all of the static media is ...
mountainswhim's user avatar
2 votes
2 answers
395 views

Can't serve static files with ampersand in name

I have a website mirrored with wget and I'd like to serve it via Apache. The problem is that I can't access files with special chars. For example, I have files with names like this: $ ls -N components/...
Grzegorz Adam Kowalski's user avatar
2 votes
1 answer
2k views

wget hangs/freezes when downloading file into NFS

I'm running some experiments with Amazon EFS (General purpose) and EC2 and I have the issue that EFS seems to act unstable. Commands that involve the mounted file system would hang or freeze. For ...
mitchkman's user avatar
  • 159
2 votes
1 answer
3k views

how can I download web pages from the command line in windows

Well, I'm trying to do some automation using batch file to speed up my daily routine at work, and I need to solve these little questions: I want to install wget command, but using only the pure ...
DaMonki's user avatar
  • 121
2 votes
1 answer
792 views

APT-GET behind a proxy with Digest Authentication

I'd like to use apt-get (and other ubuntu tools) to download software and keep it updated. Unfortunately, my company has set a squid proxy to accept digest authentication only. I've seen somewhere (...
Victor Ribeiro's user avatar
2 votes
2 answers
2k views

nginx can't execute many requests

I have a question about nginx tune. I have an application which I want execute 200 times every second. I created bash file and used wget with bqO switch for execute. But it has a problem. When the ...
Amir Hosseinzadeh's user avatar
2 votes
1 answer
495 views

Wget and recursive downloads

A directory listings enabled folder has 2 zip files. I am trying to download only one of them (the latest). Problem is the zip file name is a random string. The command I have so far is wget -r -nH ...
user3227965's user avatar
2 votes
2 answers
18k views

Wget on busybox v1.9.1

I have busybox 1.9.1 runing in huawei router. There is few commands, one of them wget and I want to use it, but there is some strange syntax. When I type wget www.google.com like I do normaly on my ...
busybox's user avatar
  • 21
2 votes
3 answers
2k views

wget not working with domain on local machine

Basically - I have some PHP scripts that need to be run as cron jobs. Lets say the script needing to be run is: http://admin.somedomain.com/cron_jobs/get_stats If I run the script from the local ...
user568829's user avatar
2 votes
1 answer
4k views

Using lynx or wget in cron to call a php script?

I'm examining a particular setup, and they have their crontab as follows: 0 * * * * lynx http://www.example.com/cron/scriptA.php Of course, this relies on the 'security-by-obscurity' precept, ...
siliconpi's user avatar
  • 1,817
2 votes
2 answers
1k views

wget 5xx error code handling

I'm executing a shell script that utilizes wget to retrieve data from the web. I'm wondering if there's a way to instruct wget to terminate the process immediately upon encountering a 500 error code.
alexus's user avatar
  • 13.4k
2 votes
2 answers
855 views

`wget`-ting a website for "local" browsing on a different domain

I need to mirror a website and deploy the copy under a different domain name. The mirroring procedure should be all automatic, so that I can update the copy on a regular basis with cron. The mirror ...
Lucio Crusca's user avatar
2 votes
1 answer
1k views

How to set index.html to redirect wget query to an other page/file?

If my server is at example.com, running $ wget example.com will only download the file index.html. How can I make wget either download another file instead of index.html download this file along ...
user123456's user avatar
2 votes
1 answer
9k views

Download an SSL certificate from a remote website through a proxy

I want a script checking my server's certificate. There are very good answers around here how to get this with openssl s_client or gnutls-cli, which works fine.... but NOT if you are behind a proxy! ...
avh's user avatar
  • 41
2 votes
1 answer
1k views

Under what scenarios would Apache allow a mismatched HTTP_HOST to pass through?

Someone has managed to pass an undefined HTTP_HOST server variable to my application script, triggering a series of errors. I am quite perturbed but am unable to replicate this behaviour. My httpd ...
Question Overflow's user avatar