GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.
1
vote
2answers
524 views
what is the default rate unit in wget?
When I use wget to download a large file, it gives me "56M/s". I want to know if it means 56 megabits per second or 56 megabytes per second. Thanks!
Like this:
231,997,440 61.4M/s in 4.0s
1
vote
3answers
182 views
error while executing bash script -> command not found
i made this script here, and get the error
IA-exporto.sh: 13: wget: not found
i tried changing the " by ` and mixed all together and rearranged, but it just won't do..
#!/bin/bash
...
1
vote
5answers
1k views
Multithreaded downloading with shell script
Let's say I have a file with lots of URLs and I want to download them in parallel using arbitrary number of processes. How can I do it with bash?
1
vote
1answer
107 views
wget only works for some users
Currently have a strange situation: wget only works for one user on a system -- and root isn't it.
There are three user accounts, user A, B and root. User A can use wget with no problems. User B and ...
1
vote
2answers
530 views
Cron not running commands as root
I have a file called /scripts/checkInternet and it contains:
#!/bin/bash
WGET="/usr/bin/wget"
rm /tmp/index.google
$WGET -q --tries=10
--timeout=5 http://www.google.com -O ...
1
vote
2answers
206 views
Using wget and Awk to count similar expressions
I am trying to create a script that uses wget to download a data set and then awk to sort though the file and tell you the most common filter used which is $14 column. So far I have the wget function ...
1
vote
4answers
196 views
Internet speed in Linux via command
I am running several sites and I was trying wget and I noticed slow speed. So I suspect that my sites are taking considerable amount of traffic.
Question is, how I can really detect the total ...
1
vote
2answers
120 views
Has anyone experienced unintentional traffic overload due to clients' use of wget?
I am wondering if it is wise to indiscriminately block wget on servers that I maintain.
Has anyone experienced unintentional traffic overload due to clients' use of wget?
1
vote
3answers
123 views
Limiting specific process to just one per user
My scripts are using wget to retrieve data from Internet. When many users are using this script I got very high load (about 20.00), because of disk I/O. Wget is automatically started each hour by ...
1
vote
2answers
206 views
Can someone explain to me what “wget -O - -q icanhazip.com” mean?
I've used this command to find the IP of the server. Can someone explain what the command means? I want to learn.
So far I know:
"wget" is a free utility for non-interactive download of files from ...
1
vote
2answers
363 views
On Debian server, downloads stall out after one or two seconds using apt-get
I have a Debian server that is used primarily as a very low-traffic web server. Recently it began having some trouble regarding its network connection. I am able to use SFTP to upload a file to this ...
1
vote
2answers
206 views
Is there a built-in command-line tool under Windows like wget/curl?
I come from a Linux/Unix background and I have been wondering if Windows has a binary that can download files from the console.
I would like to automate a certain process and one of my requirements ...
1
vote
1answer
984 views
Wget from HTTPS url using user/password -still getting 400 error
I'm trying to run Wget for a certain HTTP url, also using http-user and http-password command line options (tried the --user and --password, same results).
Using the --no-check-certificate options as ...
1
vote
4answers
2k views
Why is wget hanging?
wget to a single specific url from one of my servers keeps getting timeouts. All other urls from this box work fine. This url works OK from any other boxes I have. Here's the output:
wget -T 10 ...
1
vote
4answers
211 views
wget an remote website index directory
Is it possible to wget an entire index/parent directory with wget? Because I am trying to transfer images to a VPS of mine from my old shared host.
When I say an index/parent directory, I meant like ...
1
vote
2answers
862 views
wget is working only when used with sudo
I'm having quite a strange behavior with wget since yesterday.
I can download files by using sudo wget, but when I try the same file with only wget, I can get this error:
yusufh@ubuntu-yuh:~$ wget ...
1
vote
3answers
11k views
How to install wget on this?
I did download RubyStack 2.0.3 for VMWare (Ubuntu 9.10) but I cannot download anything on it! It appears that all basic utilities are missing/screwed:
bitnami@linux:/var/tmp$ wget
-bash: wget: ...
1
vote
1answer
105 views
Stream tar.gz file from FTP server
Here is the situation: I have a tar.gz file on a FTP server which can contain an arbitrary number of files.
Now what I'm trying to accomplish is have this file streamed and uploaded to HDFS through a ...
1
vote
2answers
273 views
How to untar a single file from an archive with randomly generated directory names?
I'm trying write a script that will download a tarball from github and extract a single file from it.
However the top level directory inside the tarball has some random characters in it, which I ...
1
vote
2answers
238 views
Download file in Linux when file location is unknown
Sorry if this is unclear, but I'm trying to set up a script that downloads a file. Currently, my method of downloading the file is by clicking on a link like so:
...
1
vote
3answers
2k views
wget: download a website which is password protected using a form
I have a problem with downloading a website using wget.
First I need to login to the website with a username and password:
wget --save-cookies cookies.txt --post-data --cookies=on ...
1
vote
1answer
1k views
Using lynx or wget in cron to call a php script?
I'm examining a particular setup, and they have their crontab as follows:
0 * * * * lynx http://www.example.com/cron/scriptA.php
Of course, this relies on the 'security-by-obscurity' precept, ...
1
vote
3answers
298 views
File timestamp not changing even with touch
I have the following problem. I have a bash script that uses wget to get a few files from the build server, and then scp's them over to the production system: Here is the relevant snippet:
#Several ...
1
vote
1answer
224 views
Hourly CRON task running more frequently than one hour
I have a cron task that calls a special PHP script via wget. Here is the crontab entry:
0 * * * * wget http://www....
It will work perfect for several days, running on the hour. However, after a few ...
1
vote
2answers
45 views
Download and save a file with proof that it was downloaded from a specific server/url on a specific date?
Assuming the file was downloaded over http is this possible? I'm guessing there could be some way to do it using a hash and a timestamp however I'm not sure how you could prove nothing was tampered ...
1
vote
1answer
229 views
iptables redirect tcp to checkip.dyndns.org from localhost to 127.0.0.1:8118
I've tried several different combinations of rules and nothing seems to be working. I know that you can't use prerouting table for a request coming from the localhost so I used the output table and ...
1
vote
3answers
593 views
Download Git Zipball in Unix
I'm trying to download a zipball of a git repo:
e.g.
wget https://github.com/zeromq/jzmq/zipball/master
This works fine in a web browser but on unix the file gets a weird name...how do I do this?
1
vote
1answer
325 views
Translate web page from shell
I'm trying to do translate a web page automatically with a shell script.
Here is the interesting part:
wget http://www.google.com/translate?langpair=en%7Ces&u=http%3A%2F%2Fasdf.com
I'm getting ...
1
vote
2answers
95 views
wget has different response times for same fetch
Any ideas why one fetch would fetch a file (big picture in this example) fast while another fetch would fetch same file slow? Not seeing this issue at home. Not seeing this issue through ip ...
1
vote
2answers
1k views
WGet or cURL: Mirror Site from http://site.com And No Internal Access
I have tried wget -m wget -r and a whole bunch of variations. I am getting some of the images on http://site.com, one of the scripts, and none of the CSS, even with the fscking -p parameter. The ...
1
vote
1answer
404 views
wget not getting all files
I have issues using wget,
basically I use it like:
wget -m ftp://username:password@hostname.com:/path/to/dirs
After everything's complete (it says xxx files in xx MB for xx seconds), however, when ...
1
vote
1answer
669 views
wget mirroring, subdomains and directories and cookies
I have an account on a web page that is now "full" (ie I have used up all my allocated space) and I would like to make a mirror of that site. wget seems like the thing to use.
The problem is that I ...
1
vote
3answers
112 views
Problems with wget AWS ubuntu apache2
I am trying to use wget to download from my AWS ubuntu server with apache2. I have tried several different options but they all result in either a file by the directory name, or an index.html file.
...
1
vote
1answer
62 views
SElinux stopping LVS from working with https
LVS/piranha is setup and trying to get it to balance https instead of http. Setup https testing with wget - idea from this link. Works when I do it at the command prompt. With SELinux enforcing, ...
1
vote
3answers
158 views
wget not working with domain on local machine
Basically - I have some PHP scripts that need to be run as cron jobs.
Lets say the script needing to be run is: http://admin.somedomain.com/cron_jobs/get_stats
If I run the script from the local ...
1
vote
2answers
94 views
Changed DNS, no effect locally
I know it can take up to a few days for DNS changes to take effect, but this has me baffled, maybe somone can offer a plausible explanation..
$ wget http://***OLDIP***/ -O oldserver.html
--2012-07-25 ...
1
vote
1answer
79 views
Serve mirrored (static) web-page with original headers
I have a dynamic webpage which I want to create a "frozen" copy of.
Typically I would do something like wget -m http://example.com, and then put the files in the document root of the web-server.
...
1
vote
6answers
2k views
Wget, Curl, Yum Fail but Ping works - CentOS 5
We currently have three webservers.
Servers one and two behave, but I'm having real problems with the third.
wget, curl and yum all fail to make their connections - that is, they all hang after ...
1
vote
0answers
180 views
How to use sitemap.xml to create a static mirror of a CMS
Is there a tool to create a static mirror of a content management system (CMS) that provides a sitemap.xml file?
Ideally, I would point a tool like wget or curl to a sitemap.xml file and have it ...
0
votes
1answer
379 views
Does crontab file automatically bash a text file?
I have a crontab job setup. In the crontab file, I have a path to a text file. My text file has a wget command (which in turn, executes a PHP file). If the crontab file just has the path to the text ...
0
votes
5answers
454 views
is there a problem in disabling curl and wget?
Is it recommended to disable curl and wget to disallow anyone from accessing my websites using those commands?
Or is it needed for some search engines to access my websites?
What are your ...
0
votes
3answers
105 views
cron job running repeatedly, why?
Why is this cron job executing repeatedly over time and what can I do to stop it?
I have a cron job that is supposed to run at 4 each morning. It hits a php script that executes some daily data ...
0
votes
3answers
56 views
How to tar folders with specific letter
I have a long list of folders. i want tar only folder which starts with "a". How can i do that?
Actually, I am having photo gallery website, which has 1000s of folders. I wish to move another server ...
0
votes
3answers
283 views
Setting wget permission to 755 so users other than root can execute it a big security risk?
I read recently in blogs that by default wget on linux is 750, so only root can execute it. I would like to allow users wget and change it to 755, but I read around the web that it is a big security ...
0
votes
2answers
151 views
wget works for all sites on the web but not the one hosted on that server
I currently have 2 Ubuntu 12.04 servers which are load balanced. If I go to anyone on them from the shell and type:
wget stackoverflow.com
The page is fetched into index.html. However, assuming ...
0
votes
2answers
2k views
Using wget to save sequential files as well as renaming the file extension
I run a cron job that requests a snapshot from a remote webcam at a local address:
wget http://user:pass@10.0.0.50/snapshot.cgi
This creates the files snapshot.cgi, snapshot.cgi.1, snapshot.cgi.2, ...
0
votes
4answers
3k views
Why can't I wget specific cacti graph images?
I'm trying to wget a specific cacti graph image in order to make a daily report. We have htpasswd for the cacti website, and I've added a user for that. I'm usign wget's --http-user/password to pass ...
0
votes
2answers
336 views
406 Not Acceptable error when runing wget
I have this command in crontab:
wget --quiet --delete-after http://boms.ro/admincp/cron/s/9abf0f42c1e4f55fdb87d8237cdde
And when I run it with the --debug argument I get the following response:
...
0
votes
3answers
658 views
Crontab repeats every 15 min
My cronjob is being repeated every 15 minutes and I don't have any idea why.
Here is the crontab file with real commands replaced:
# m h dom mon dow command
# Comment
0 9 * * * command arg1
0 14 ...
0
votes
1answer
32 views
How to download images from one server to another
I have a VPS running Centos.
I also have a site (say, example.com) hosted on separate, shared hosting. I would like to backup all images in the "images" folder, which consist of the following ...
