GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.
0
votes
1answer
32 views
How to download images from one server to another
I have a VPS running Centos.
I also have a site (say, example.com) hosted on separate, shared hosting. I would like to backup all images in the "images" folder, which consist of the following ...
0
votes
4answers
49 views
wget ftp an entire site is very slow
I'm trying to wget an entire site to transfer from Server A to Server B.
It works, but incredibly slowly.
I tested this by uploading a 100mb psd file to my server, and grabbing it with wget ftp. ...
0
votes
1answer
39 views
Why would an HTTP(S) request not be logged in IIS?
I'm troubleshooting a batch file that uses Wget to send requests to a website running in IIS 7. The batch file runs Wget twice, the first time to login to the site via a POST, the second to run some ...
0
votes
1answer
33 views
cannot wget wordpress.org
I am not able to wget wordpress.org from server 63.143.51.2. I am able to wget other sites like google.com
Error I am getting is as follows :
[~]# wget wordpress.org
--2013-04-12 01:42:17-- ...
0
votes
1answer
38 views
ldirectord NAT load balancing does not work with negotiate checks
I am using ldirectord for loadbalancing two HTTP servers. On the load balancer box i have the following network configuration:
eth0 is used for internal purposes and does not take part in load ...
0
votes
1answer
57 views
Get a URL with variables via Linux Command Line
Sorry if this seems to be a very simple command, but I have been searching for hours without any solutions and exhausted my very limited knowledge of Linux. :-)
I need to use some command to process ...
0
votes
1answer
88 views
How to check ip blacklist using curl/wget request to mxtoolbox?
I would like to add cron job to check server ip on blacklist.
something like
curl 'http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a142.11.193.83' | grep 'you are on a blacklist.'
wget ...
1
vote
2answers
45 views
Download and save a file with proof that it was downloaded from a specific server/url on a specific date?
Assuming the file was downloaded over http is this possible? I'm guessing there could be some way to do it using a hash and a timestamp however I'm not sure how you could prove nothing was tampered ...
0
votes
1answer
47 views
What could block the output of a command?
Sometimes I'll attempt a (2>&1) redirection and some/all of the resulting output appears to be silenced.
e.g.
wget -O- http://localhost/test.txt 2>&1
I would expect to see a merge of ...
3
votes
4answers
108 views
Download a file from the shell when button click is required
I have a link to a file that I would like to download from the shell. Unfortunately, the URL redirects to a software license agreement page which requires clicking an accept button. This is fine when ...
1
vote
2answers
207 views
Is there a built-in command-line tool under Windows like wget/curl?
I come from a Linux/Unix background and I have been wondering if Windows has a binary that can download files from the console.
I would like to automate a certain process and one of my requirements ...
0
votes
3answers
38 views
input wget exclusions from a file?
i'm crawling a forum and I keep stumbling across certain threads that have been going on for ten years.
i can certainly exclude these using wget option:
-X ...
1
vote
3answers
182 views
error while executing bash script -> command not found
i made this script here, and get the error
IA-exporto.sh: 13: wget: not found
i tried changing the " by ` and mixed all together and rearranged, but it just won't do..
#!/bin/bash
...
2
votes
1answer
30 views
Wget Minimum Download Rate
Can anyone help with the parameter that ensures that wget will fetch the file at a minimum specified download rate?
1
vote
3answers
112 views
Problems with wget AWS ubuntu apache2
I am trying to use wget to download from my AWS ubuntu server with apache2. I have tried several different options but they all result in either a file by the directory name, or an index.html file.
...
-1
votes
1answer
116 views
Prevent webpages from being downloaded by downloadmanagers with NGINX
I want to create a site that offers themes. People will be able to see the themes and use a demo but I want to prevent cheapskates from using wget or any other download manager to download the demo ...
0
votes
3answers
56 views
How to tar folders with specific letter
I have a long list of folders. i want tar only folder which starts with "a". How can i do that?
Actually, I am having photo gallery website, which has 1000s of folders. I wish to move another server ...
0
votes
2answers
504 views
What are the possible problems, when wget returns code 500 but same request works in normal browsers?
What should I be looking for, when wget returns 500 but the same URL works fine in my web browser? I don't see any access_log entries that seem to be related to the error.
DEBUG output created by ...
0
votes
2answers
402 views
Execute curl requests in parallel in bash
What is the best way to execute 5 curl requests in parallel from a bash script? I can't run them in serial for performance reasons.
2
votes
1answer
204 views
Reboot if tomcat7 service is not responding
I am running a web app on a Tomcat server. There is a hard-to-detect problem within the server code that causes it to crash once or twice everyday. I will dig in to correct it when I have time. But ...
0
votes
3answers
105 views
cron job running repeatedly, why?
Why is this cron job executing repeatedly over time and what can I do to stop it?
I have a cron job that is supposed to run at 4 each morning. It hits a php script that executes some daily data ...
-1
votes
1answer
164 views
Iptables blocks outgoing HTTP connection
I've setup IPTables with the following script... however, I cannot get outgoing HTTP queries to work (I've tried using lynx & curl).
Background - server is hosted remotely (and I'm connecting via ...
2
votes
1answer
87 views
Find out what's running wget on my server
In my access.log I see requests coming from 127.0.0.1 and retrieving server's "/" and then all of graphics and CSS referenced in it. This happens regularly, every 5 minutes.
I want to find out what, ...
0
votes
2answers
90 views
Fast (non-blocking) way to transfer many files to another server [duplicate]
Possible Duplicate:
what is the fastest and most reliable way of transferring a lot of files?
I am currently attempting to transfer over 1 million files from one server to another. Using ...
1
vote
1answer
62 views
SElinux stopping LVS from working with https
LVS/piranha is setup and trying to get it to balance https instead of http. Setup https testing with wget - idea from this link. Works when I do it at the command prompt. With SELinux enforcing, ...
1
vote
3answers
159 views
wget not working with domain on local machine
Basically - I have some PHP scripts that need to be run as cron jobs.
Lets say the script needing to be run is: http://admin.somedomain.com/cron_jobs/get_stats
If I run the script from the local ...
0
votes
1answer
2k views
why i cannot download jdk from oracle web site directly without AuthParam?
that is download with the following command, why it fails to download that file?
wget http://download.oracle.com/otn-pub/java/jdk/6u35-b10/jdk-6u35-linux-i586.bin
the following command works, but ...
4
votes
2answers
363 views
wget starts downloading then stops “cannot write to”
I'm using wget to mirror some files across from one server to another. I'm using the following command:
wget -x -N -i http://domain.com/filelist.txt
-x = Because I want to keep the directory ...
0
votes
3answers
283 views
Setting wget permission to 755 so users other than root can execute it a big security risk?
I read recently in blogs that by default wget on linux is 750, so only root can execute it. I would like to allow users wget and change it to 755, but I read around the web that it is a big security ...
0
votes
0answers
87 views
why iftop and wget's transfer status varied so much?
the transfer speed is 2147kb/s now
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
29 ...
0
votes
1answer
35 views
wget does not mirror if ip is provided instead of domain name
I am trying to get a site mirroring with wget, and came across a strange behavior.
Say I am mirroring an internal site named www.example.com in the following way, all seems to be working fine:
wget ...
1
vote
2answers
94 views
Changed DNS, no effect locally
I know it can take up to a few days for DNS changes to take effect, but this has me baffled, maybe somone can offer a plausible explanation..
$ wget http://***OLDIP***/ -O oldserver.html
--2012-07-25 ...
3
votes
1answer
2k views
How do I fix certificate errors when running wget on an HTTPS URL in Cygwin Windows?
I have Cygwin installed in my windows system
I am trying to execute wget commad for a website but I am gettign the following error message
ERROR: The certificate of xxx.com' is not trusted.
ERROR: ...
0
votes
2answers
151 views
wget works for all sites on the web but not the one hosted on that server
I currently have 2 Ubuntu 12.04 servers which are load balanced. If I go to anyone on them from the shell and type:
wget stackoverflow.com
The page is fetched into index.html. However, assuming ...
1
vote
1answer
79 views
Serve mirrored (static) web-page with original headers
I have a dynamic webpage which I want to create a "frozen" copy of.
Typically I would do something like wget -m http://example.com, and then put the files in the document root of the web-server.
...
1
vote
1answer
105 views
Stream tar.gz file from FTP server
Here is the situation: I have a tar.gz file on a FTP server which can contain an arbitrary number of files.
Now what I'm trying to accomplish is have this file streamed and uploaded to HDFS through a ...
4
votes
4answers
187 views
How do I reconcile Network Speed from Network Throughput?
I have been arguing with my ISP about the service they are providing; primarily with regard to throughput speeds.
I have set up an edge router (see hardware details listed below) and on it I have ...
0
votes
0answers
329 views
wget hangs in http request sent awaiting response in some sites
Using Ubuntu 12.04. wget hangs in http request sent, awaiting response... in some sites. Browser's are not opening sites that are failed in wget. But in WinXP everything works.
This works
...
-1
votes
2answers
136 views
wget not saving file as the triggered file to disk [closed]
What I am trying to do is download a file from a url like this:
http://www.example.com/down/files/425/document-preview?
When I visit this url in my browser, it triggers a file called for example: ...
1
vote
1answer
229 views
iptables redirect tcp to checkip.dyndns.org from localhost to 127.0.0.1:8118
I've tried several different combinations of rules and nothing seems to be working. I know that you can't use prerouting table for a request coming from the localhost so I used the output table and ...
1
vote
3answers
593 views
Download Git Zipball in Unix
I'm trying to download a zipball of a git repo:
e.g.
wget https://github.com/zeromq/jzmq/zipball/master
This works fine in a web browser but on unix the file gets a weird name...how do I do this?
-1
votes
2answers
195 views
How download all files, directories and subdirectoires with wget [closed]
I have some links such as:
http:/foo.com/1/1/1/1.jpg
http:/foo.com/1/2/2/2.jpg
http:/foo.com/1/3/3/3.jpg
...
How I can I download all files and directories with wget?
1
vote
2answers
206 views
Using wget and Awk to count similar expressions
I am trying to create a script that uses wget to download a data set and then awk to sort though the file and tell you the most common filter used which is $14 column. So far I have the wget function ...
0
votes
2answers
336 views
406 Not Acceptable error when runing wget
I have this command in crontab:
wget --quiet --delete-after http://boms.ro/admincp/cron/s/9abf0f42c1e4f55fdb87d8237cdde
And when I run it with the --debug argument I get the following response:
...
4
votes
1answer
8k views
Accessing a windows shared folder from Linux?
I am trying to copy a (.html) file from a Windows XP Professional shared folder onto a server running Ubuntu Linux 10.04 LTS.
As it's a shared folder the usual 'wget' doesn't seem to work. I suspect ...
0
votes
2answers
258 views
Curl and Wget returns different response code
I am trying to wget a link, this wget works fine on my local machine but it doesn't do so on the server.
i tried to check the response header and i got the following on my local laptop
curl -I ...
0
votes
2answers
367 views
Use number of downloaded file in wget as variable in bash script
Can I get the number of downloaded files with wget -r and use that as a variable? I want to write a script that runs multiple wget commands in it (using -q so i can control output) and then at the ...
2
votes
2answers
4k views
Ubuntu 11.10, using wget/curl fails with ssl
On a completely new install of Ubuntu I'm getting the following errors when using wget:
wget https://test.sagepay.com
--2012-03-27 12:55:12-- https://test.sagepay.com/
Resolving test.sagepay.com... ...
1
vote
2answers
273 views
How to untar a single file from an archive with randomly generated directory names?
I'm trying write a script that will download a tarball from github and extract a single file from it.
However the top level directory inside the tarball has some random characters in it, which I ...
1
vote
2answers
530 views
Cron not running commands as root
I have a file called /scripts/checkInternet and it contains:
#!/bin/bash
WGET="/usr/bin/wget"
rm /tmp/index.google
$WGET -q --tries=10
--timeout=5 http://www.google.com -O ...
