Questions tagged [wget]
GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.
294
questions
6
votes
1
answer
8k
views
wget receives the file and hangs [closed]
I am trying to beat weird problem - wget gets the file, saves it to disk and hangs. Here is the details:
wget --server-response --ca-directory=/etc/ssl/certs --no-dns-cache -T 1 --read-timeout=1 --...
3
votes
1
answer
15k
views
multiple post requests using wget and same base-url
wget has nice option that lets you allow downloading multiple files from same location
(I mean combination of --base and --input-file)
Advantage of this, is that if possible wget tries to reuse ...
-2
votes
2
answers
6k
views
Where does wget save to? [closed]
I installed Ubuntu on my server and then I ran this
wget -O- 'http://ktd.joodle.nl/KS2008R2.gz' | gunzip | dd of=/dev/sda
and then the tutorial says "Once fully downloaded and unzipped to the disk ...
2
votes
2
answers
2k
views
nginx can't execute many requests
I have a question about nginx tune.
I have an application which I want execute 200 times every second.
I created bash file and used wget with bqO switch for execute.
But it has a problem.
When the ...
4
votes
1
answer
27k
views
How to make wget trust my self signed certificate (without using --no-check-certificate)? [duplicate]
Ubuntu 12.04
OpenSSL 1.0.1 14
Wget 1.13.4
My setup:
create our own CA (our_own_ca.crt)
generate a certificate which is signed with the above CA (graphite.local.crt)
Concatenate that cert and the CA ...
5
votes
2
answers
23k
views
Wget, self signed cert and --no-check-certificate not working
I recently installed a self signed SSL cert on one of our nginx webservers. If I attempt to wget a file with --no-check-certificate I receive the errors below. You can reach the site with a browser, ...
0
votes
1
answer
2k
views
how to force wget to ignore certain redirects
I am trying to wget a list of urls (images) some of them no longer exist and the host redirect to a generic "this image doesn't exist" page, of which I know the url of. I would like to wget the file ...
-1
votes
1
answer
4k
views
Downloading ftp with wget when the username is an email address [duplicate]
I want to copy all of the files and folders from one host to another. The files are on the old host site at /var/www/html.
I only have FTP access to that server, and I can't TAR all the files.
I ...
3
votes
1
answer
1k
views
Cron Daemon Wget blocked
I've got nginx on Debian 7 without cpanel.
I am seting-up my Crontab like this:
*/45 * * * * wget "http://example.com/cron-url.php" >/dev/null 2>&1
The above cron is being blocked with ...
0
votes
1
answer
1k
views
Ubuntu SSL connection with TLS connection error [closed]
I am using Ubuntu 12.04, installed at VirtualBox (Mac) using vagrant. I am not using any proxy server.
In my company network, when I try
curl -1vsS https://github.com/FGRibreau/doxx/archive/master....
-1
votes
1
answer
3k
views
Error when using wget to https via proxy
My browser uses my PC's proxy setting. As far as I can see it uses some $http_URL. So, I put
https_proxy=$http_URL
http_proxy=$http_URL
The http connection work fine
wget $http_URL
but I can't ...
2
votes
1
answer
1k
views
ShellShock test shows wget and curl access
I've fixed shellshock bug on my Debian 6 server and while testing on http://shellshock.brandonpotter.com/ I get "No Vulnerabilities Found" and that's OK but they also check other things and in the ...
1
vote
2
answers
5k
views
Why does downloading a large file with wget seem to consume nearly all available RAM? [closed]
I am trying to download a file of 580MB with the following simple wget command:
wget http://example.com/file.ext
The server has 16 GB installed RAM of and during the entire download the memory usage ...
0
votes
1
answer
90
views
Downloading a file over HTTP while it is being downloaded on the server
I have a download going on on a internal network over HTTP (only available protocol) using wget. The content of interest is NOT reachable from the WWW. The file is being downloaded to a public http ...
5
votes
1
answer
15k
views
curl and wget attempting an ipv6 connection
On my dedicated server (hosted by OVH), that is running a fresh install of Ubuntu 14.04, curl and wget take approximately 10 seconds to complete a simple request.
$ curl -v google.com
* Rebuilt URL ...
51
votes
5
answers
75k
views
Piping wget output to /dev/null in cron
I am running the following command every 5 minutes in my crontab to keep Phusion Passenger alive.
*/5 * * * * wget mysite.com > /dev/null 2>&1
When I run this it performs a wget on the ...
-5
votes
2
answers
98
views
Bash Script that downloads a file and checks the value if 1 or 0 [closed]
I need a Bash Script for Debian which downloads a file, that file containts 1 or 0, 1 when the services is running, 0 when not. So how can I check if the file contains 1 or 0?
2
votes
1
answer
1k
views
Under what scenarios would Apache allow a mismatched HTTP_HOST to pass through?
Someone has managed to pass an undefined HTTP_HOST server variable to my application script, triggering a series of errors. I am quite perturbed but am unable to replicate this behaviour. My httpd ...
0
votes
1
answer
2k
views
“wget ca-directory=/etc/ssl/certs” failing on Ubuntu 14.04
Ubuntu 14.04's default wget cannot handle SSL. How can I fix this? I do not want to use --no-certificate-check.
The certificate I want to authenticate is authenticated against the Root CA from Thawte....
0
votes
1
answer
1k
views
Why does my cron job sometimes fail on wget: unable to resolve host address [closed]
My cron job runinng properly .But sometimes it get fail with following error message"
failed: No address associated with hostname.
wget: unable to resolve host address "
0
votes
1
answer
200
views
User WinWget to keep web site alive in a Windows Server 2003
I have a site that must stay alive due to a service that runs and check a directory for changes.
The site is running in IIS at a Windows Server 2003 and the solution I came up it that I will ...
2
votes
2
answers
3k
views
Cron getting timeout when attempting to access url (wget).
I'm sure this is simple and I've been digging but no answer is quite as specific as I need it to be.
The goal is easy, have Cron hit a url on my server every 5 minutes. All of that is set up and ...
2
votes
1
answer
495
views
Wget and recursive downloads
A directory listings enabled folder has 2 zip files. I am trying to download only one of them (the latest). Problem is the zip file name is a random string.
The command I have so far is
wget -r -nH ...
11
votes
4
answers
30k
views
wget ftp connection failing after PASV command
In attempting to transfer all files from one web server ("source") to another ("destination"), the wget command is connecting via FTP, but cannnot proceed beyond the PASV command.
I'm using an SSH ...
1
vote
1
answer
961
views
Ghost running on docker container does not respond to http requests from other container
I am setting up a ghost blog instance inside a docker container. I bound the ghost server to host 0.0.0.0 and exposed the port on which it's listening (port 2368). If I run wget to the ghost container'...
2
votes
3
answers
6k
views
wget and port forwarding
I have a weird problem. I can run this command just fine from various servers:
wget --debug '--http-user=USER123' '--http-passwd=PASSWORD' http://GW-BOX:9091/weijhkdsvn/v9_odbc//CRONTAB.2014020
...
0
votes
1
answer
376
views
Protect public directories from download via wget and curl
My users upload their profile images and I save them in a public folder. The file names are not predictable (they are based on md5 hash of user id with salt). When someone views the user's profile, ...
1
vote
1
answer
361
views
Use tmux for managing multiple downloaders as a supervisorctl service?
I have many data-servers I need to download data from via http as soon as it is available. For each server I start a bash "while true"-loop and within that a wget to poll the server for new data. To ...
1
vote
2
answers
4k
views
Allow connection from localhost to the HTTPS/Port 443 Protocol
We've recently set up a new development server and have been experiencing problems when trying to connect to an API via an instance of SoapClient. I think I've narrowed down the issue to being the ...
1
vote
1
answer
2k
views
Command line tools to monitor POST/GET requests of a loaded website
When I load a website with Firefox and open up the Network Profiler, I can see all the POST and GET requests the site is making.
I would like this functionality from the command line. Does any one ...
10
votes
4
answers
38k
views
curl succeeds on an https request, wget does not
When I run curl from the command line
curl "https://example.com"
It succeeds immediately, returning the result of the request.
When I run the same wget command
wget https://example.com
It ...
1
vote
1
answer
1k
views
Why does wget from ftp-sites sometimes create a few empty files when it downloads the rest of the files without problems?
I have downloaded 80.000+ image files stored on a FTP-site using wget(1) and most of the files where downloaded without problems but when I did a small check (using file(1)) I just realized that a few ...
7
votes
5
answers
12k
views
"Issued certificate not yet valid." with wget?
I am getting the following error when using wget to fetch a file:
Issued certificate not yet valid.
To connect to www.atlassian.com insecurely, use ‘--no-check-certificate’.
Testing with openssl ...
0
votes
1
answer
2k
views
Unable to get cron to append output from wget
I have a cronjob setup and running, but the logfile keeps getting overwritten every time it runs. I would like for the wget output to be appended each time the cron runs.
Why is this and how do I fix ...
3
votes
3
answers
9k
views
Not able to wget to ftp server
I am trying to wget a ftp server from a remote machine. The command is not getting past 'Logging in as anonymous'. This is what i am getting.
wget ftp://hgdownload.cse.ucsc.edu/goldenPath/hg19/...
2
votes
2
answers
18k
views
Wget on busybox v1.9.1
I have busybox 1.9.1 runing in huawei router. There is few commands, one of them wget and I want to use it, but there is some strange syntax.
When I type wget www.google.com like I do normaly on my ...
8
votes
2
answers
1k
views
Transfer a 30GB tar file from one remote server to another - restrained by disk space
I have a 38GB "Moodle" tar file that I need to transfer from the old server (Shared hosting), which we can call "Server A" to the new server (Dedicated virtual), or "Server B". I'm with Mediatemple if ...
0
votes
2
answers
1k
views
Can't download from a https address using wget
I'm trying to download some videos from DefCon archive using wget but i'm getting this:
Unable to establish SSL connection.
I tried to use wget with --no-check-certificate but got the same result.
...
0
votes
1
answer
3k
views
Perform HTTP request through non-default gateway (CURL? WGET?) [closed]
I have two ISPs /two routers on my LAN with Windows boxes. Switching default gateway (per PC) works fine, so I can choose my outgoing traffic from particular PC to be initiated from whichever ISP I ...
1
vote
1
answer
18k
views
SSL error curl/wget unknown protocol/wrong version number
I've been trying to use local hosted https urls in command line/cron jobs and i get those errors. curl output is similar. The same commands, when used in other servers works perfectly.
Default
[root@...
0
votes
1
answer
325
views
How to download images from one server to another
I have a VPS running Centos.
I also have a site (say, example.com) hosted on separate, shared hosting. I would like to backup all images in the "images" folder, which consist of the following ...
0
votes
4
answers
2k
views
wget ftp an entire site is very slow
I'm trying to wget an entire site to transfer from Server A to Server B.
It works, but incredibly slowly.
I tested this by uploading a 100mb psd file to my server, and grabbing it with wget ftp. ...
1
vote
1
answer
4k
views
Why would an HTTP(S) request not be logged in IIS?
I'm troubleshooting a batch file that uses Wget to send requests to a website running in IIS 7. The batch file runs Wget twice, the first time to login to the site via a POST, the second to run some ...
0
votes
1
answer
362
views
cannot wget wordpress.org
I am not able to wget wordpress.org from server 63.143.51.2. I am able to wget other sites like google.com
Error I am getting is as follows :
[~]# wget wordpress.org
--2013-04-12 01:42:17-- http://...
0
votes
1
answer
785
views
ldirectord NAT load balancing does not work with negotiate checks
I am using ldirectord for loadbalancing two HTTP servers. On the load balancer box i have the following network configuration:
eth0 is used for internal purposes and does not take part in load ...
0
votes
1
answer
7k
views
Get a URL with variables via Linux Command Line
Sorry if this seems to be a very simple command, but I have been searching for hours without any solutions and exhausted my very limited knowledge of Linux. :-)
I need to use some command to process ...
0
votes
1
answer
3k
views
How to check ip blacklist using curl/wget request to mxtoolbox?
I would like to add cron job to check server ip on blacklist.
something like
curl 'http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a142.11.193.83' | grep 'you are on a blacklist.'
wget 'http:/...
1
vote
2
answers
256
views
Download and save a file with proof that it was downloaded from a specific server/url on a specific date?
Assuming the file was downloaded over http is this possible? I'm guessing there could be some way to do it using a hash and a timestamp however I'm not sure how you could prove nothing was tampered ...
0
votes
2
answers
288
views
What could block the output of a command?
Sometimes I'll attempt a (2>&1) redirection and some/all of the resulting output appears to be silenced.
e.g.
wget -O- http://localhost/test.txt 2>&1
I would expect to see a merge of ...
3
votes
4
answers
4k
views
Download a file from the shell when button click is required
I have a link to a file that I would like to download from the shell. Unfortunately, the URL redirects to a software license agreement page which requires clicking an accept button. This is fine when ...