Questions tagged [wget]
GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.
294
questions
5
votes
4
answers
14k
views
using wget through .pac config based proxy server
I want to use wget through a proxy which uses .pac config.
When I googled a bit, I found that .pac is a javascript file and wget cannot parse it based on the following url
http://www.linuxquestions....
5
votes
2
answers
23k
views
Wget, self signed cert and --no-check-certificate not working
I recently installed a self signed SSL cert on one of our nginx webservers. If I attempt to wget a file with --no-check-certificate I receive the errors below. You can reach the site with a browser, ...
5
votes
1
answer
15k
views
curl and wget attempting an ipv6 connection
On my dedicated server (hosted by OVH), that is running a fresh install of Ubuntu 14.04, curl and wget take approximately 10 seconds to complete a simple request.
$ curl -v google.com
* Rebuilt URL ...
4
votes
3
answers
11k
views
How to use "wget" to save a file as a different name
wget "http://flakejfl2ke3jrlwkefj38j2ofiwejfoifjalsdkfje3fklwje.exe" is long.
How do I save it as my own name?
4
votes
2
answers
6k
views
what is the default rate unit in wget?
When I use wget to download a large file, it gives me "56M/s". I want to know if it means 56 megabits per second or 56 megabytes per second. Thanks!
Like this:
231,997,440 61.4M/s in 4.0s
4
votes
1
answer
2k
views
"wget --mirror" Alternative
I'm looking for an alternative to using this, it doesn't have to use FTP, but it should provide the same functionality as the "--mirror" option of wget where it only downloads new and changed files.
...
4
votes
1
answer
2k
views
Downloading from wget - folder issue
I have a website - http://myradio1.localhost and the directory is currently empty.
The virtual host works fine, no issues there.
This is my command I'm using to download the site via wget:
wget --...
4
votes
1
answer
9k
views
Cannot get cURL or wget to validate some SSL certificates
I've noticed that our link checker, which uses cURL, fails more and more often to validate SSL certificates. I'm trying to get to the bottom of this.
https://www.bgetem.de/, for instance, opens just ...
4
votes
2
answers
8k
views
Retrieve FTP directory tree from the command line
What would be the fastest way to recursively retrieve entire directory listing from an ftp server using wget/curl/whatever? I don't need to download any files, just directory and file names. Basically ...
4
votes
5
answers
31k
views
Fixing wget certificates
Basically, I was testing different versions of OpenSSL, and somewhere in the process broke wget's path to certificates.
wget https://www.google.com
Results now in:
ERROR: cannot verify www.google....
4
votes
4
answers
1k
views
How do I reconcile Network Speed from Network Throughput?
I have been arguing with my ISP about the service they are providing; primarily with regard to throughput speeds.
I have set up an edge router (see hardware details listed below) and on it I have ...
4
votes
1
answer
27k
views
How to make wget trust my self signed certificate (without using --no-check-certificate)? [duplicate]
Ubuntu 12.04
OpenSSL 1.0.1 14
Wget 1.13.4
My setup:
create our own CA (our_own_ca.crt)
generate a certificate which is signed with the above CA (graphite.local.crt)
Concatenate that cert and the CA ...
4
votes
0
answers
17k
views
How to Enable JavaScript for wget in linux for grabbing a website?
I use wget like this for save site :
wget --page-requisites --no-parent --mirror http://example.com/index.html -P /home/
in Some cases It does NOT work, Error is :
This site requires Javascript to ...
3
votes
2
answers
31k
views
Using curl with cookies
I hope someone can help me out on this one.
I have been trying this for a while now, with all sorts of variations and cannot seem to get this to work.
I am trying to script a way to log in to https://...
3
votes
4
answers
4k
views
Download a file from the shell when button click is required
I have a link to a file that I would like to download from the shell. Unfortunately, the URL redirects to a software license agreement page which requires clicking an accept button. This is fine when ...
3
votes
3
answers
8k
views
nginx block curl and wget
I have nginx webserver.
I have a rich content site and i found that some malicious bots are trying to crawl my content. I blocked any curl or wget coming to my site like this
if ($http_user_agent ~* ...
3
votes
2
answers
4k
views
Can someone explain to me what "wget -O - -q icanhazip.com" mean?
I've used this command to find the IP of the server. Can someone explain what the command means? I want to learn.
So far I know:
"wget" is a free utility for non-interactive download of files from ...
3
votes
1
answer
1k
views
Cron Daemon Wget blocked
I've got nginx on Debian 7 without cpanel.
I am seting-up my Crontab like this:
*/45 * * * * wget "http://example.com/cron-url.php" >/dev/null 2>&1
The above cron is being blocked with ...
3
votes
2
answers
643
views
pgrep wget: what is the details of process id?
when typing pgrep wget, it show process id 10144
but how to know what is the details of this process id
3
votes
2
answers
4k
views
Curl always returns same 404 page
No matter what URL I specify for curl I always get the same HTML 404 Error page back.
If I use the --verbose option, it looks like curl always connects to the same IP address.
$ curl --verbose http:/...
3
votes
3
answers
9k
views
Not able to wget to ftp server
I am trying to wget a ftp server from a remote machine. The command is not getting past 'Logging in as anonymous'. This is what i am getting.
wget ftp://hgdownload.cse.ucsc.edu/goldenPath/hg19/...
3
votes
2
answers
2k
views
Why is this bash command not echoing into a variable and what can I do to improve?
I've got a bash script I'm working to improve, and have put on a great fix thanks to Dennis Williamson. Unfortunately, one of the lines no longer echoes into a variable I can manipulate, rather dumps ...
3
votes
2
answers
10k
views
HTTP downloads stop after some time, resuming is not possible
When I try to download a file via HTTP, the downloads sometimes stop after around 30 MB. The download rates goes down to 0 B/s and no data keeps coming. When I stop the download and resume again, the ...
3
votes
1
answer
15k
views
multiple post requests using wget and same base-url
wget has nice option that lets you allow downloading multiple files from same location
(I mean combination of --base and --input-file)
Advantage of this, is that if possible wget tries to reuse ...
3
votes
3
answers
14k
views
why i cannot download jdk from oracle web site directly without AuthParam?
that is download with the following command, why it fails to download that file?
wget http://download.oracle.com/otn-pub/java/jdk/6u35-b10/jdk-6u35-linux-i586.bin
the following command works, but ...
3
votes
3
answers
5k
views
Download Git Zipball in Unix
I'm trying to download a zipball of a git repo:
e.g.
wget https://github.com/zeromq/jzmq/zipball/master
This works fine in a web browser but on unix the file gets a weird name...how do I do this?
3
votes
3
answers
5k
views
Reboot if tomcat7 service is not responding
I am running a web app on a Tomcat server. There is a hard-to-detect problem within the server code that causes it to crash once or twice everyday. I will dig in to correct it when I have time. But ...
2
votes
3
answers
565
views
wget application replacement
Is there any other application similar to wget?
If yes can you post a link?
2
votes
1
answer
9k
views
How would I write a terminal command to download a folder with wget from a Media Temple (gs) server?
I'm trying to download a folder using wget on the Terminal (I'm usin a Mac if that matters) because my ftp client sucks and keeps timing out. It doesn't stay connected for long. So I was wondering if ...
2
votes
1
answer
3k
views
wget download only certain folders from site
I want to download this site with wget, and to do this I'm using this command:
wget.exe --no-clobber --no-parent -r --convert-links --page-requisites -U Mozilla "http://www.s-manuals.com/smd/"
That'...
2
votes
1
answer
134
views
Wget Minimum Download Rate
Can anyone help with the parameter that ensures that wget will fetch the file at a minimum specified download rate?
2
votes
1
answer
770
views
Stream tar.gz file from FTP server
Here is the situation: I have a tar.gz file on a FTP server which can contain an arbitrary number of files.
Now what I'm trying to accomplish is have this file streamed and uploaded to HDFS through a ...
2
votes
2
answers
311
views
how can i use apache log files to recreate usage scenario
Recently i installed a website that had too many requests and it was too slow. Many improvements have been made to the web site code and we've also bought a new server.
I want to test the new server ...
2
votes
1
answer
277
views
Why does the time it takes to run this script differ so much between user and system and how I might correct the discrepancy?
My application sits behind a load balancer, and every once in a while I like to do a status check on each machine to get an idea of the time it takes to return an index.html document on each machine.
...
2
votes
1
answer
342
views
How to prevent external website from being downloaded in wget?
I'm using wget to download useful website recursively:
wget -k -m -r -q -t 1 http://www.web.com/
But external website also downloaded along with the website I want.
How to prevent this external ...
2
votes
2
answers
3k
views
mirror a site with wget and download static media
I'd like to mirror a site with wget and convert all the links to the local copies I've downloaded. So far that's easy all I have to do is wget -mk http://site.com
However all of the static media is ...
2
votes
2
answers
395
views
Can't serve static files with ampersand in name
I have a website mirrored with wget and I'd like to serve it via Apache. The problem is that I can't access files with special chars. For example, I have files with names like this:
$ ls -N components/...
2
votes
1
answer
2k
views
wget hangs/freezes when downloading file into NFS
I'm running some experiments with Amazon EFS (General purpose) and EC2 and I have the issue that EFS seems to act unstable. Commands that involve the mounted file system would hang or freeze.
For ...
2
votes
1
answer
3k
views
how can I download web pages from the command line in windows
Well, I'm trying to do some automation using batch file to speed up my daily routine at work, and I need to solve these little questions:
I want to install wget command, but using only the pure ...
2
votes
1
answer
792
views
APT-GET behind a proxy with Digest Authentication
I'd like to use apt-get (and other ubuntu tools) to download software and keep it updated.
Unfortunately, my company has set a squid proxy to accept digest authentication only. I've seen somewhere (...
2
votes
2
answers
2k
views
nginx can't execute many requests
I have a question about nginx tune.
I have an application which I want execute 200 times every second.
I created bash file and used wget with bqO switch for execute.
But it has a problem.
When the ...
2
votes
1
answer
495
views
Wget and recursive downloads
A directory listings enabled folder has 2 zip files. I am trying to download only one of them (the latest). Problem is the zip file name is a random string.
The command I have so far is
wget -r -nH ...
2
votes
2
answers
18k
views
Wget on busybox v1.9.1
I have busybox 1.9.1 runing in huawei router. There is few commands, one of them wget and I want to use it, but there is some strange syntax.
When I type wget www.google.com like I do normaly on my ...
2
votes
3
answers
2k
views
wget not working with domain on local machine
Basically - I have some PHP scripts that need to be run as cron jobs.
Lets say the script needing to be run is: http://admin.somedomain.com/cron_jobs/get_stats
If I run the script from the local ...
2
votes
1
answer
4k
views
Using lynx or wget in cron to call a php script?
I'm examining a particular setup, and they have their crontab as follows:
0 * * * * lynx http://www.example.com/cron/scriptA.php
Of course, this relies on the 'security-by-obscurity' precept, ...
2
votes
2
answers
1k
views
wget 5xx error code handling
I'm executing a shell script that utilizes wget to retrieve data from the web. I'm wondering if there's a way to instruct wget to terminate the process immediately upon encountering a 500 error code.
2
votes
2
answers
855
views
`wget`-ting a website for "local" browsing on a different domain
I need to mirror a website and deploy the copy under a different domain name. The mirroring procedure should be all automatic, so that I can update the copy on a regular basis with cron.
The mirror ...
2
votes
1
answer
1k
views
How to set index.html to redirect wget query to an other page/file?
If my server is at example.com, running
$ wget example.com
will only download the file index.html. How can I make wget either
download another file instead of index.html
download this file along ...
2
votes
1
answer
9k
views
Download an SSL certificate from a remote website through a proxy
I want a script checking my server's certificate. There are very good answers around here how to get this with openssl s_client or gnutls-cli, which works fine.... but NOT if you are behind a proxy! ...
2
votes
1
answer
1k
views
Under what scenarios would Apache allow a mismatched HTTP_HOST to pass through?
Someone has managed to pass an undefined HTTP_HOST server variable to my application script, triggering a series of errors. I am quite perturbed but am unable to replicate this behaviour. My httpd ...