GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.

learn more… | top users | synonyms

0
votes
3answers
2k views

download only new files from ftp using WGET/cUrl

i have Windows 2008 Server on which i store DB backups on daily basis. I want to be able to download new files only using wget, curl or windows built-in FTP doesn't matter. Can you help me with ...
0
votes
1answer
261 views

Am I doing ubuntu 10.0.4 software installation correctly (packages and none-packages)?

Background: I first installed everything using a terminal and "sudo su", i.e. as root. Then I installed jenkins, and then the jenkins user could not run any android sdk tools in /root/androidsdk/tools ...
1
vote
2answers
375 views

Download file in Linux when file location is unknown

Sorry if this is unclear, but I'm trying to set up a script that downloads a file. Currently, my method of downloading the file is by clicking on a link like so: ...
14
votes
2answers
2k views

Wget: Is there a way to pass username and password from a file?

Is there a way to pass the username and password from a file instead of the command line via --user and --password? Background: I want to run wget via cron and don't want the username/password show ...
1
vote
1answer
418 views

Translate web page from shell

I'm trying to do translate a web page automatically with a shell script. Here is the interesting part: wget http://www.google.com/translate?langpair=en%7Ces&u=http%3A%2F%2Fasdf.com I'm getting ...
2
votes
3answers
319 views

wget application replacement

Is there any other application similar to wget? If yes can you post a link?
1
vote
4answers
578 views

wget an remote website index directory

Is it possible to wget an entire index/parent directory with wget? Because I am trying to transfer images to a VPS of mine from my old shared host. When I say an index/parent directory, I meant like ...
1
vote
2answers
121 views

wget has different response times for same fetch

Any ideas why one fetch would fetch a file (big picture in this example) fast while another fetch would fetch same file slow? Not seeing this issue at home. Not seeing this issue through ip ...
0
votes
2answers
3k views

Using wget to save sequential files as well as renaming the file extension

I run a cron job that requests a snapshot from a remote webcam at a local address: wget http://user:pass@10.0.0.50/snapshot.cgi This creates the files snapshot.cgi, snapshot.cgi.1, snapshot.cgi.2, ...
1
vote
4answers
239 views

Internet speed in Linux via command

I am running several sites and I was trying wget and I noticed slow speed. So I suspect that my sites are taking considerable amount of traffic. Question is, how I can really detect the total ...
1
vote
3answers
4k views

wget: download a website which is password protected using a form

I have a problem with downloading a website using wget. First I need to login to the website with a username and password: wget --save-cookies cookies.txt --post-data --cookies=on ...
2
votes
5answers
2k views

Multithreaded downloading with shell script

Let's say I have a file with lots of URLs and I want to download them in parallel using arbitrary number of processes. How can I do it with bash?
2
votes
2answers
5k views

Alternatives to CURL, WGET

Is anyone aware of any alternatives to curl and wget? The key functional requirement I'm looking for is to be able to execute an HTTPS GET request against a known URL. The reason I'm looking for an ...
0
votes
3answers
318 views

How can I use wget to get files recursively if the username contains an @?

Firstly my main goal is to try and get FTP files recursively in Centos. However the actual FTP login contains an @ which ruins the syntax. Is there some way to place the username in "double quotes" ...
2
votes
2answers
1k views

Why is this bash command not echoing into a variable and what can I do to improve?

I've got a bash script I'm working to improve, and have put on a great fix thanks to Dennis Williamson. Unfortunately, one of the lines no longer echoes into a variable I can manipulate, rather dumps ...
2
votes
1answer
225 views

Why does the time it takes to run this script differ so much between user and system and how I might correct the discrepancy?

My application sits behind a load balancer, and every once in a while I like to do a status check on each machine to get an idea of the time it takes to return an index.html document on each machine. ...
2
votes
2answers
2k views

WGet or cURL: Mirror Site from http://site.com And No Internal Access

I have tried wget -m wget -r and a whole bunch of variations. I am getting some of the images on http://site.com, one of the scripts, and none of the CSS, even with the fscking -p parameter. The ...
-1
votes
1answer
295 views

cpanel cron job not working

I have created a cron job in cpanel. The result is this: * * * * * /usr/local/bin/wget http://example.com/facturi-restante/cron The problem is that is not working. Can anyone give me a ...
7
votes
3answers
4k views

Why is only one wget command working in my crontab?

I wish to fetch content from a PHP script on my server two times a day, altering a query variable lang to set what language we want, and save this content in two language specific files. This is my ...
3
votes
1answer
2k views

using wget through .pac config based proxy server

I want to use wget through a proxy which uses .pac config. When I googled a bit, I found that .pac is a javascript file and wget cannot parse it based on the following url ...
6
votes
4answers
5k views

Wget a script and run it

I want to download a script from: http://dl.dropbox.com/u/11210438/flockonus-stack.sh and execute it. My guess is, to use wget renaming it, chmod it, and execute. What are the commands for doing ...
5
votes
1answer
534 views

What to do if wget's progress:mega doen't cut it anymore?

I have script for downloading large files in a queue. Now I would really appreciate it, if I had a decent log. Specifying the -o or -a option is apparently specially made for that, however as of the ...
1
vote
1answer
661 views

wget not getting all files

I have issues using wget, basically I use it like: wget -m ftp://username:password@hostname.com:/path/to/dirs After everything's complete (it says xxx files in xx MB for xx seconds), however, when ...
0
votes
3answers
386 views

Download Sun Studio via CLI

Can anybody please guide me how to download the sun studio from CLI. I was using wget and lynx programs but not worked. As I have only SSH access to my server and I cant not download it on local ...
0
votes
3answers
174 views

transfer code from one server to other server

I wanted to transfer new code into my new production server. I have code files on my development server. Instead of uploading files using FTP from my local machine, there is other way to transfer ...
3
votes
2answers
330 views

pgrep wget: what is the details of process id?

when typing pgrep wget, it show process id 10144 but how to know what is the details of this process id
0
votes
2answers
369 views

DNS issue on Fedora 12? wget wordpress.org fails where wget www.google.com works

I'm administering a Fedora 12 box, but am quite new to networking specifics. Recently one of our WordPress apps hosted on our server has stopped being able to perform its auto-update or auto-download ...
1
vote
3answers
1k views

wget is working only when used with sudo

I'm having quite a strange behavior with wget since yesterday. I can download files by using sudo wget, but when I try the same file with only wget, I can get this error: yusufh@ubuntu-yuh:~$ wget ...
0
votes
1answer
415 views

Download error for php data using wget

I am trying to download data from my server using wget. I send the login details and store the cookie. I then rotate through 50 numbers to copy the data into new files. The saved files are always ...
2
votes
2answers
1k views

Retrieve FTP directory tree from the command line

What would be the fastest way to recursively retrieve entire directory listing from an ftp server using wget/curl/whatever? I don't need to download any files, just directory and file names. Basically ...
1
vote
1answer
1k views

Using lynx or wget in cron to call a php script?

I'm examining a particular setup, and they have their crontab as follows: 0 * * * * lynx http://www.example.com/cron/scriptA.php Of course, this relies on the 'security-by-obscurity' precept, ...
0
votes
2answers
275 views

WGet from one site on a server to another site on the same server

Hey all, I've recently been asked to administer a couple ubuntu boxes running web servers. I'm a dev by trade so if this question is fairly noob please forgive. We have about a dozen sites running ...
0
votes
2answers
39 views

URL arguments and Webserver configuration

I am trying to archive an old wordpress blog of mine. So far the best way I have found to do this is via wget. The problem is the name of the files it generates, like ...
0
votes
6answers
726 views

is there a problem in disabling curl and wget?

Is it recommended to disable curl and wget to disallow anyone from accessing my websites using those commands? Or is it needed for some search engines to access my websites? What are your ...
0
votes
2answers
116 views

I'm seeing a lot of wget requests, should I be worried

I've got a web based application that I've developed and I'm seeing a lot of wget requests on the site. Should I be worried that someone is trying to hack/rip the site. I don't know what they would ...
0
votes
2answers
3k views

How to tell wget to use the name of the target file behind the HTTP redirect?

Let's a consider link such as htttp://www.mysite.net/download.php?id=267 that redirects to htttp://www.mysite.net/downloads/myfile.tar.gz If I execute 'wget ...
1
vote
3answers
346 views

File timestamp not changing even with touch

I have the following problem. I have a bash script that uses wget to get a few files from the build server, and then scp's them over to the production system: Here is the relevant snippet: #Several ...
2
votes
7answers
3k views

Wget, Curl, Yum Fail but Ping works - CentOS 5

We currently have three webservers. Servers one and two behave, but I'm having real problems with the third. wget, curl and yum all fail to make their connections - that is, they all hang after ...
2
votes
1answer
250 views

How to prevent external website from being downloaded in wget?

I'm using wget to download useful website recursively: wget -k -m -r -q -t 1 http://www.web.com/ But external website also downloaded along with the website I want. How to prevent this external ...
1
vote
0answers
222 views

How to use sitemap.xml to create a static mirror of a CMS

Is there a tool to create a static mirror of a content management system (CMS) that provides a sitemap.xml file? Ideally, I would point a tool like wget or curl to a sitemap.xml file and have it ...
9
votes
2answers
15k views

Where can I (safely) get libintl3.dll?

I am having problems running the Windows version of wget on a Windows 7 machine. When I run the command I get this error: The program can't start because libintl3.dll is missing from your ...
0
votes
1answer
1k views

wget - specify directory and rename the file

I'm trying to download multiple files and need to rename as I download, how can I do that and specify the directory I want them to download to? I know i need to be using -P and -O to do this but it ...
1
vote
1answer
128 views

wget only works for some users

Currently have a strange situation: wget only works for one user on a system -- and root isn't it. There are three user accounts, user A, B and root. User A can use wget with no problems. User B and ...
0
votes
4answers
2k views

wget doesn't take variable http_proxy

If I run: echo $http_proxy It output: http://user:pass@host:port But when I run: wget www.google.com It output: Resolviendo www.google.com... 209.85.195.104 Caching www.google.com => ...
4
votes
6answers
5k views

How to download with wget without following links with parameters

I'm trying to download two sites for inclusion on a CD: http://boinc.berkeley.edu/trac/wiki http://www.boinc-wiki.info The problem I'm having is that these are both wikis. So when downloading with ...
6
votes
4answers
2k views

How to refresh an online website mirror created with `wget --mirror`?

One month ago, I used "wget --mirror" to create a mirror of our public website for temporary use during an upcoming scheduled maintenance window. Our primary website runs HTML, PHP & MySQL, but ...
-1
votes
2answers
2k views

What does -q and -O mean?

What does -q and -O mean in the following cronjob? */30 * * * * /usr/bin/wget -q -O /dev/null http://example.com/moodle/admin/cron.php Thanks in advance.
2
votes
2answers
498 views

wget 5xx error code handling

i'm running a shell script that uses wget and pulls something over the web, my question is is there a way to tell wget to abort as soon as it hits 500 error code?
2
votes
4answers
2k views

Escape a ! in the password parameter of wget

I'm trying to execute something like this: wget --user=foo --password=bar! url The ! in the password is causing problems. I've tried escaping it with \, as in --password=bar\! I've tried ...
1
vote
2answers
129 views

Has anyone experienced unintentional traffic overload due to clients' use of wget?

I am wondering if it is wise to indiscriminately block wget on servers that I maintain. Has anyone experienced unintentional traffic overload due to clients' use of wget?