GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.

learn more… | top users | synonyms

1
vote
1answer
4k views

SSL error curl/wget unknown protocol/wrong version number

I've been trying to use local hosted https urls in command line/cron jobs and i get those errors. curl output is similar. The same commands, when used in other servers works perfectly. Default ...
0
votes
1answer
71 views

How to download images from one server to another

I have a VPS running Centos. I also have a site (say, example.com) hosted on separate, shared hosting. I would like to backup all images in the "images" folder, which consist of the following ...
0
votes
4answers
432 views

wget ftp an entire site is very slow

I'm trying to wget an entire site to transfer from Server A to Server B. It works, but incredibly slowly. I tested this by uploading a 100mb psd file to my server, and grabbing it with wget ftp. ...
1
vote
1answer
570 views

Why would an HTTP(S) request not be logged in IIS?

I'm troubleshooting a batch file that uses Wget to send requests to a website running in IIS 7. The batch file runs Wget twice, the first time to login to the site via a POST, the second to run some ...
0
votes
1answer
85 views

cannot wget wordpress.org

I am not able to wget wordpress.org from server 63.143.51.2. I am able to wget other sites like google.com Error I am getting is as follows : [~]# wget wordpress.org --2013-04-12 01:42:17-- ...
0
votes
1answer
261 views

ldirectord NAT load balancing does not work with negotiate checks

I am using ldirectord for loadbalancing two HTTP servers. On the load balancer box i have the following network configuration: eth0 is used for internal purposes and does not take part in load ...
0
votes
1answer
3k views

Get a URL with variables via Linux Command Line

Sorry if this seems to be a very simple command, but I have been searching for hours without any solutions and exhausted my very limited knowledge of Linux. :-) I need to use some command to process ...
0
votes
1answer
920 views

How to check ip blacklist using curl/wget request to mxtoolbox?

I would like to add cron job to check server ip on blacklist. something like curl 'http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a142.11.193.83' | grep 'you are on a blacklist.' wget ...
1
vote
2answers
64 views

Download and save a file with proof that it was downloaded from a specific server/url on a specific date?

Assuming the file was downloaded over http is this possible? I'm guessing there could be some way to do it using a hash and a timestamp however I'm not sure how you could prove nothing was tampered ...
0
votes
1answer
113 views

What could block the output of a command?

Sometimes I'll attempt a (2>&1) redirection and some/all of the resulting output appears to be silenced. e.g. wget -O- http://localhost/test.txt 2>&1 I would expect to see a merge of ...
3
votes
4answers
1k views

Download a file from the shell when button click is required

I have a link to a file that I would like to download from the shell. Unfortunately, the URL redirects to a software license agreement page which requires clicking an accept button. This is fine when ...
8
votes
4answers
28k views

Is there a built-in command-line tool under Windows like wget/curl?

I come from a Linux/Unix background and I have been wondering if Windows has a binary that can download files from the console. I would like to automate a certain process and one of my requirements ...
0
votes
3answers
136 views

input wget exclusions from a file?

i'm crawling a forum and I keep stumbling across certain threads that have been going on for ten years. i can certainly exclude these using wget option: -X ...
1
vote
3answers
622 views

error while executing bash script -> command not found

i made this script here, and get the error IA-exporto.sh: 13: wget: not found i tried changing the " by ` and mixed all together and rearranged, but it just won't do.. #!/bin/bash ...
2
votes
1answer
49 views

Wget Minimum Download Rate

Can anyone help with the parameter that ensures that wget will fetch the file at a minimum specified download rate?
1
vote
3answers
216 views

Problems with wget AWS ubuntu apache2

I am trying to use wget to download from my AWS ubuntu server with apache2. I have tried several different options but they all result in either a file by the directory name, or an index.html file. ...
-1
votes
1answer
430 views

Prevent webpages from being downloaded by downloadmanagers with NGINX

I want to create a site that offers themes. People will be able to see the themes and use a demo but I want to prevent cheapskates from using wget or any other download manager to download the demo ...
-1
votes
3answers
115 views

How to tar folders with specific letter

I have a long list of folders. i want tar only folder which starts with "a". How can i do that? Actually, I am having photo gallery website, which has 1000s of folders. I wish to move another server ...
0
votes
2answers
3k views

What are the possible problems, when wget returns code 500 but same request works in normal browsers?

What should I be looking for, when wget returns 500 but the same URL works fine in my web browser? I don't see any access_log entries that seem to be related to the error. DEBUG output created by ...
3
votes
3answers
8k views

Execute curl requests in parallel in bash

What is the best way to execute 5 curl requests in parallel from a bash script? I can't run them in serial for performance reasons.
2
votes
3answers
2k views

Reboot if tomcat7 service is not responding

I am running a web app on a Tomcat server. There is a hard-to-detect problem within the server code that causes it to crash once or twice everyday. I will dig in to correct it when I have time. But ...
0
votes
3answers
557 views

cron job running repeatedly, why?

Why is this cron job executing repeatedly over time and what can I do to stop it? I have a cron job that is supposed to run at 4 each morning. It hits a php script that executes some daily data ...
-1
votes
1answer
700 views

Iptables blocks outgoing HTTP connection

I've setup IPTables with the following script... however, I cannot get outgoing HTTP queries to work (I've tried using lynx & curl). Background - server is hosted remotely (and I'm connecting via ...
2
votes
1answer
172 views

Find out what's running wget on my server

In my access.log I see requests coming from 127.0.0.1 and retrieving server's "/" and then all of graphics and CSS referenced in it. This happens regularly, every 5 minutes. I want to find out what, ...
0
votes
2answers
119 views

Fast (non-blocking) way to transfer many files to another server [duplicate]

Possible Duplicate: what is the fastest and most reliable way of transferring a lot of files? I am currently attempting to transfer over 1 million files from one server to another. Using ...
1
vote
1answer
227 views

SElinux stopping LVS from working with https

LVS/piranha is setup and trying to get it to balance https instead of http. Setup https testing with wget - idea from this link. Works when I do it at the command prompt. With SELinux enforcing, ...
1
vote
3answers
395 views

wget not working with domain on local machine

Basically - I have some PHP scripts that need to be run as cron jobs. Lets say the script needing to be run is: http://admin.somedomain.com/cron_jobs/get_stats If I run the script from the local ...
1
vote
3answers
7k views

why i cannot download jdk from oracle web site directly without AuthParam?

that is download with the following command, why it fails to download that file? wget http://download.oracle.com/otn-pub/java/jdk/6u35-b10/jdk-6u35-linux-i586.bin the following command works, but ...
7
votes
3answers
5k views

wget starts downloading then stops “cannot write to”

I'm using wget to mirror some files across from one server to another. I'm using the following command: wget -x -N -i http://domain.com/filelist.txt -x = Because I want to keep the directory ...
1
vote
3answers
1k views

Setting wget permission to 755 so users other than root can execute it a big security risk?

I read recently in blogs that by default wget on linux is 750, so only root can execute it. I would like to allow users wget and change it to 755, but I read around the web that it is a big security ...
0
votes
1answer
79 views

wget does not mirror if ip is provided instead of domain name

I am trying to get a site mirroring with wget, and came across a strange behavior. Say I am mirroring an internal site named www.example.com in the following way, all seems to be working fine: wget ...
1
vote
2answers
100 views

Changed DNS, no effect locally

I know it can take up to a few days for DNS changes to take effect, but this has me baffled, maybe somone can offer a plausible explanation.. $ wget http://***OLDIP***/ -O oldserver.html --2012-07-25 ...
6
votes
1answer
9k views

How do I fix certificate errors when running wget on an HTTPS URL in Cygwin Windows?

I have Cygwin installed in my windows system I am trying to execute wget commad for a website but I am gettign the following error message ERROR: The certificate of xxx.com' is not trusted. ERROR: ...
0
votes
2answers
835 views

wget works for all sites on the web but not the one hosted on that server

I currently have 2 Ubuntu 12.04 servers which are load balanced. If I go to anyone on them from the shell and type: wget stackoverflow.com The page is fetched into index.html. However, assuming ...
1
vote
1answer
105 views

Serve mirrored (static) web-page with original headers

I have a dynamic webpage which I want to create a "frozen" copy of. Typically I would do something like wget -m http://example.com, and then put the files in the document root of the web-server. ...
2
votes
1answer
262 views

Stream tar.gz file from FTP server

Here is the situation: I have a tar.gz file on a FTP server which can contain an arbitrary number of files. Now what I'm trying to accomplish is have this file streamed and uploaded to HDFS through a ...
4
votes
4answers
652 views

How do I reconcile Network Speed from Network Throughput?

I have been arguing with my ISP about the service they are providing; primarily with regard to throughput speeds. I have set up an edge router (see hardware details listed below) and on it I have ...
-1
votes
2answers
414 views

wget not saving file as the triggered file to disk [closed]

What I am trying to do is download a file from a url like this: http://www.example.com/down/files/425/document-preview? When I visit this url in my browser, it triggers a file called for example: ...
1
vote
1answer
465 views

iptables redirect tcp to checkip.dyndns.org from localhost to 127.0.0.1:8118

I've tried several different combinations of rules and nothing seems to be working. I know that you can't use prerouting table for a request coming from the localhost so I used the output table and ...
1
vote
3answers
2k views

Download Git Zipball in Unix

I'm trying to download a zipball of a git repo: e.g. wget https://github.com/zeromq/jzmq/zipball/master This works fine in a web browser but on unix the file gets a weird name...how do I do this?
-1
votes
2answers
430 views

How download all files, directories and subdirectoires with wget [closed]

I have some links such as: http:/foo.com/1/1/1/1.jpg http:/foo.com/1/2/2/2.jpg http:/foo.com/1/3/3/3.jpg ... How I can I download all files and directories with wget?
1
vote
2answers
473 views

Using wget and Awk to count similar expressions

I am trying to create a script that uses wget to download a data set and then awk to sort though the file and tell you the most common filter used which is $14 column. So far I have the wget function ...
0
votes
2answers
1k views

406 Not Acceptable error when runing wget

I have this command in crontab: wget --quiet --delete-after http://boms.ro/admincp/cron/s/9abf0f42c1e4f55fdb87d8237cdde And when I run it with the --debug argument I get the following response: ...
12
votes
1answer
36k views

Accessing a windows shared folder from Linux?

I am trying to copy a (.html) file from a Windows XP Professional shared folder onto a server running Ubuntu Linux 10.04 LTS. As it's a shared folder the usual 'wget' doesn't seem to work. I suspect ...
0
votes
2answers
748 views

Curl and Wget returns different response code

I am trying to wget a link, this wget works fine on my local machine but it doesn't do so on the server. i tried to check the response header and i got the following on my local laptop curl -I ...
0
votes
2answers
902 views

Use number of downloaded file in wget as variable in bash script

Can I get the number of downloaded files with wget -r and use that as a variable? I want to write a script that runs multiple wget commands in it (using -q so i can control output) and then at the ...
3
votes
2answers
9k views

Ubuntu 11.10, using wget/curl fails with ssl

On a completely new install of Ubuntu I'm getting the following errors when using wget: wget https://test.sagepay.com --2012-03-27 12:55:12-- https://test.sagepay.com/ Resolving test.sagepay.com... ...
1
vote
2answers
683 views

How to untar a single file from an archive with randomly generated directory names?

I'm trying write a script that will download a tarball from github and extract a single file from it. However the top level directory inside the tarball has some random characters in it, which I ...
1
vote
2answers
1k views

Cron not running commands as root

I have a file called /scripts/checkInternet and it contains: #!/bin/bash WGET="/usr/bin/wget" rm /tmp/index.google $WGET -q --tries=10 --timeout=5 http://www.google.com -O ...
1
vote
1answer
617 views

How to block some robots and referer pages in Apache root level

When I was using Lighttpd I can easily achieve this by such entries. So all websites were protected. Wget robots: $HTTP["useragent"] =~ "Wget" { $HTTP["url"] =~ "^/tagi(.*)" { # ...