GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.

learn more… | top users | synonyms

2
votes
3answers
488 views

wget and port forwarding

I have a weird problem. I can run this command just fine from various servers: wget --debug '--http-user=USER123' '--http-passwd=PASSWORD' http://GW-BOX:9091/weijhkdsvn/v9_odbc//CRONTAB.2014020 ...
1
vote
3answers
490 views

wget not working with domain on local machine

Basically - I have some PHP scripts that need to be run as cron jobs. Lets say the script needing to be run is: http://admin.somedomain.com/cron_jobs/get_stats If I run the script from the local ...
0
votes
3answers
149 views

input wget exclusions from a file?

i'm crawling a forum and I keep stumbling across certain threads that have been going on for ten years. i can certainly exclude these using wget option: -X /t/41866,/t/314849,/t/335041,/t/356321,/...
1
vote
2answers
969 views

Cron getting timeout when attempting to access url (wget).

I'm sure this is simple and I've been digging but no answer is quite as specific as I need it to be. The goal is easy, have Cron hit a url on my server every 5 minutes. All of that is set up and ...
0
votes
2answers
504 views

“wget ca-directory=/etc/ssl/certs” failing on Ubuntu 14.04

Ubuntu 14.04's default wget cannot handle SSL. How can I fix this? I do not want to use --no-certificate-check. The certificate I want to authenticate is authenticated against the Root CA from Thawte....
0
votes
2answers
422 views

WGet from one site on a server to another site on the same server

Hey all, I've recently been asked to administer a couple ubuntu boxes running web servers. I'm a dev by trade so if this question is fairly noob please forgive. We have about a dozen sites running ...
-1
votes
2answers
1k views

wget downloading index.html instead of tarball?

When I type the following: http://www.archlinux.org/packages/extra/x86_64/enca/download/ wget is downloading index.html instead of the correct file which is: enca-1.13-2-x86_64.pkg.tar.xz When I ...
4
votes
1answer
4k views

curl and wget attempting an ipv6 connection

On my dedicated server (hosted by OVH), that is running a fresh install of Ubuntu 14.04, curl and wget take approximately 10 seconds to complete a simple request. $ curl -v google.com * Rebuilt URL ...
1
vote
1answer
358 views

Why does wget from ftp-sites sometimes create a few empty files when it downloads the rest of the files without problems?

I have downloaded 80.000+ image files stored on a FTP-site using wget(1) and most of the files where downloaded without problems but when I did a small check (using file(1)) I just realized that a few ...
1
vote
1answer
291 views

SElinux stopping LVS from working with https

LVS/piranha is setup and trying to get it to balance https instead of http. Setup https testing with wget - idea from this link. Works when I do it at the command prompt. With SELinux enforcing, ...
1
vote
1answer
114 views

Serve mirrored (static) web-page with original headers

I have a dynamic webpage which I want to create a "frozen" copy of. Typically I would do something like wget -m http://example.com, and then put the files in the document root of the web-server. ...
0
votes
1answer
18 views

wget blocking subdirectiories

I would like to recursively download a page, but block link like this containing "strony": http://domain.com/rozrywka/strony/2/ my command is wget -r -nd --delete-after -e robots=off -X strony ...
0
votes
1answer
422 views

Download an SSL certificate from a remote website through a proxy

I want a script checking my server's certificate. There are very good answers around here how to get this with openssl s_client or gnutls-cli, which works fine.... but NOT if you are behind a proxy! ...
0
votes
1answer
2k views

Using curl with cookies

I hope someone can help me out on this one. I have been trying this for a while now, with all sorts of variations and cannot seem to get this to work. I am trying to script a way to log in to https://...
0
votes
1answer
254 views

how to force wget to ignore certain redirects

I am trying to wget a list of urls (images) some of them no longer exist and the host redirect to a generic "this image doesn't exist" page, of which I know the url of. I would like to wget the file ...
0
votes
1answer
146 views

Protect public directories from download via wget and curl

My users upload their profile images and I save them in a public folder. The file names are not predictable (they are based on md5 hash of user id with salt). When someone views the user's profile, ...
0
votes
1answer
719 views

Unable to get cron to append output from wget

I have a cronjob setup and running, but the logfile keeps getting overwritten every time it runs. I would like for the wget output to be appended each time the cron runs. Why is this and how do I fix ...
0
votes
1answer
105 views

cannot wget wordpress.org

I am not able to wget wordpress.org from server 63.143.51.2. I am able to wget other sites like google.com Error I am getting is as follows : [~]# wget wordpress.org --2013-04-12 01:42:17-- http://...
0
votes
1answer
482 views

Download error for php data using wget

I am trying to download data from my server using wget. I send the login details and store the cookie. I then rotate through 50 numbers to copy the data into new files. The saved files are always ...
-1
votes
1answer
870 views

Error when using wget to https via proxy

My browser uses my PC's proxy setting. As far as I can see it uses some $http_URL. So, I put https_proxy=$http_URL http_proxy=$http_URL The http connection work fine wget $http_URL but I can't ...
-1
votes
1answer
521 views

Prevent webpages from being downloaded by downloadmanagers with NGINX

I want to create a site that offers themes. People will be able to see the themes and use a demo but I want to prevent cheapskates from using wget or any other download manager to download the demo ...
-1
votes
1answer
1k views

Iptables blocks outgoing HTTP connection

I've setup IPTables with the following script... however, I cannot get outgoing HTTP queries to work (I've tried using lynx & curl). Background - server is hosted remotely (and I'm connecting via ...
2
votes
0answers
618 views

Wrong filename encoding/decoding when downloading recursively with wget

I'm downloading files from a remote directory recursively using wget and whoever created the folders and files, used special characters such as è or Ó, when I download a single specificating the full ...
2
votes
0answers
502 views

Recursively get files over FTP with explicit SSL

Is there a way to recursively get a whole directory from another server using FTP with explicit SSL on Linux? I have tried a number of wget combinations including with and without the passive flag ...
2
votes
0answers
327 views

How to use sitemap.xml to create a static mirror of a CMS

Is there a tool to create a static mirror of a content management system (CMS) that provides a sitemap.xml file? Ideally, I would point a tool like wget or curl to a sitemap.xml file and have it ...
1
vote
0answers
15 views

Can wget detect remote duplicate files (with different names) and download only one?

I'm retrieving a dynamic website contents recursively. Unfortunately, files are available from different dynamic URLs. For instance, maybe file http://foo.bar/bla.php?q=xyz and http://foo.bar/bla....
1
vote
0answers
47 views

Server socket sitting in FIN_WAIT_1 whilst network doesn't see its traffic

We've been trying to get a grip on a really weird problem where we can wget a page from Apache 2.2.19 on solaris 10 and some permutations of requests reliably take various fixed lengths of time to ...
1
vote
0answers
230 views

why is a wget to http://graph.facebook.com resolving to my server's IP?

This makes absolutely no sense and I spent 3 hours trying to figure this out. Suddenly all my scripts connecting to Facebook stopped working, so I tried to do this: root@s01 [~]# wget http://graph....
0
votes
0answers
19 views

Download file from server using a command line

I have 2 different servers, I put my mp3 files in server1, and I am testing to download them from server2 using the wget tool: wget "http://www.example.com/Path-to-mp3-file" I have 2 results: 1) ...
0
votes
0answers
21 views

wget truncating files in centos 7

We have used this on Centos 5 and 6 without issues for years: if /usr/bin/wget --timeout=120 --user=$User --password=$Pass -S -O ${Tmp_path}/${Tmp_file}.asc ${Rules_url}/sigs-${Version}.tar.gz.asc 2&...
0
votes
0answers
39 views

Unable to access internet through amazon ec2 instance

I have instance running on Amazon EC2.Basically i am accessing my Amazon EC2 instance through VPN using Vagrant openconnect https://github.com/ericmann/openconnect-vm . I am able to ssh into instance ...