Questions tagged [wget]
GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.
294
questions
55
votes
4
answers
280k
views
Is there a built-in command-line tool under Windows like wget/curl?
I come from a Linux/Unix background and I have been wondering if Windows has a binary that can download files from the console.
I would like to automate a certain process and one of my requirements ...
0
votes
3
answers
600
views
input wget exclusions from a file?
i'm crawling a forum and I keep stumbling across certain threads that have been going on for ten years.
i can certainly exclude these using wget option:
-X /t/41866,/t/314849,/t/335041,/t/356321,/...
1
vote
3
answers
1k
views
error while executing bash script -> command not found
i made this script here, and get the error
IA-exporto.sh: 13: wget: not found
i tried changing the " by ` and mixed all together and rearranged, but it just won't do..
#!/bin/bash
UNAME="...
2
votes
1
answer
134
views
Wget Minimum Download Rate
Can anyone help with the parameter that ensures that wget will fetch the file at a minimum specified download rate?
1
vote
3
answers
497
views
Problems with wget AWS ubuntu apache2
I am trying to use wget to download from my AWS ubuntu server with apache2. I have tried several different options but they all result in either a file by the directory name, or an index.html file.
...
0
votes
1
answer
1k
views
Prevent webpages from being downloaded by downloadmanagers with NGINX
I want to create a site that offers themes. People will be able to see the themes and use a demo but I want to prevent cheapskates from using wget or any other download manager to download the demo ...
-1
votes
3
answers
1k
views
How to tar folders with specific letter
I have a long list of folders. i want tar only folder which starts with "a". How can i do that?
Actually, I am having photo gallery website, which has 1000s of folders. I wish to move another server ...
0
votes
2
answers
12k
views
What are the possible problems, when wget returns code 500 but same request works in normal browsers?
What should I be looking for, when wget returns 500 but the same URL works fine in my web browser? I don't see any access_log entries that seem to be related to the error.
DEBUG output created by ...
22
votes
4
answers
129k
views
Execute curl requests in parallel in bash
What is the best way to execute 5 curl requests in parallel from a bash script? I can't run them in serial for performance reasons.
3
votes
3
answers
5k
views
Reboot if tomcat7 service is not responding
I am running a web app on a Tomcat server. There is a hard-to-detect problem within the server code that causes it to crash once or twice everyday. I will dig in to correct it when I have time. But ...
1
vote
3
answers
2k
views
cron job running repeatedly, why?
Why is this cron job executing repeatedly over time and what can I do to stop it?
I have a cron job that is supposed to run at 4 each morning. It hits a php script that executes some daily data ...
0
votes
1
answer
2k
views
Iptables blocks outgoing HTTP connection
I've setup IPTables with the following script... however, I cannot get outgoing HTTP queries to work (I've tried using lynx & curl).
Background - server is hosted remotely (and I'm connecting via ...
2
votes
1
answer
219
views
Find out what's running wget on my server
In my access.log I see requests coming from 127.0.0.1 and retrieving server's "/" and then all of graphics and CSS referenced in it. This happens regularly, every 5 minutes.
I want to find out what, ...
0
votes
2
answers
404
views
Fast (non-blocking) way to transfer many files to another server [duplicate]
Possible Duplicate:
what is the fastest and most reliable way of transferring a lot of files?
I am currently attempting to transfer over 1 million files from one server to another. Using wget, it ...
1
vote
1
answer
666
views
SElinux stopping LVS from working with https
LVS/piranha is setup and trying to get it to balance https instead of http. Setup https testing with wget - idea from this link. Works when I do it at the command prompt. With SELinux enforcing, ...
2
votes
3
answers
2k
views
wget not working with domain on local machine
Basically - I have some PHP scripts that need to be run as cron jobs.
Lets say the script needing to be run is: http://admin.somedomain.com/cron_jobs/get_stats
If I run the script from the local ...
3
votes
3
answers
14k
views
why i cannot download jdk from oracle web site directly without AuthParam?
that is download with the following command, why it fails to download that file?
wget http://download.oracle.com/otn-pub/java/jdk/6u35-b10/jdk-6u35-linux-i586.bin
the following command works, but ...
16
votes
9
answers
61k
views
wget starts downloading then stops "cannot write to"
I'm using wget to mirror some files across from one server to another. I'm using the following command:
wget -x -N -i http://domain.com/filelist.txt
-x = Because I want to keep the directory ...
1
vote
3
answers
5k
views
Setting wget permission to 755 so users other than root can execute it a big security risk?
I read recently in blogs that by default wget on linux is 750, so only root can execute it. I would like to allow users wget and change it to 755, but I read around the web that it is a big security ...
0
votes
1
answer
320
views
wget does not mirror if ip is provided instead of domain name
I am trying to get a site mirroring with wget, and came across a strange behavior.
Say I am mirroring an internal site named www.example.com in the following way, all seems to be working fine:
wget -...
1
vote
2
answers
121
views
Changed DNS, no effect locally
I know it can take up to a few days for DNS changes to take effect, but this has me baffled, maybe somone can offer a plausible explanation..
$ wget http://***OLDIP***/ -O oldserver.html
--2012-07-25 ...
8
votes
1
answer
32k
views
How do I fix certificate errors when running wget on an HTTPS URL in Cygwin Windows?
I have Cygwin installed in my windows system.
I am trying to execute wget command for a website but I am getting the following error message:
ERROR: The certificate of `example.com' is not trusted.
...
0
votes
2
answers
4k
views
wget works for all sites on the web but not the one hosted on that server
I currently have 2 Ubuntu 12.04 servers which are load balanced. If I go to anyone on them from the shell and type:
wget stackoverflow.com
The page is fetched into index.html. However, assuming ...
1
vote
1
answer
251
views
Serve mirrored (static) web-page with original headers
I have a dynamic webpage which I want to create a "frozen" copy of.
Typically I would do something like wget -m http://example.com, and then put the files in the document root of the web-...
2
votes
1
answer
770
views
Stream tar.gz file from FTP server
Here is the situation: I have a tar.gz file on a FTP server which can contain an arbitrary number of files.
Now what I'm trying to accomplish is have this file streamed and uploaded to HDFS through a ...
4
votes
4
answers
1k
views
How do I reconcile Network Speed from Network Throughput?
I have been arguing with my ISP about the service they are providing; primarily with regard to throughput speeds.
I have set up an edge router (see hardware details listed below) and on it I have ...
-1
votes
2
answers
1k
views
wget not saving file as the triggered file to disk [closed]
What I am trying to do is download a file from a url like this:
http://www.example.com/down/files/425/document-preview?
When I visit this url in my browser, it triggers a file called for example: ...
1
vote
1
answer
733
views
iptables redirect tcp to checkip.dyndns.org from localhost to 127.0.0.1:8118
I've tried several different combinations of rules and nothing seems to be working. I know that you can't use prerouting table for a request coming from the localhost so I used the output table and ...
3
votes
3
answers
5k
views
Download Git Zipball in Unix
I'm trying to download a zipball of a git repo:
e.g.
wget https://github.com/zeromq/jzmq/zipball/master
This works fine in a web browser but on unix the file gets a weird name...how do I do this?
-1
votes
2
answers
997
views
How download all files, directories and subdirectoires with wget [closed]
I have some links such as:
http:/foo.com/1/1/1/1.jpg
http:/foo.com/1/2/2/2.jpg
http:/foo.com/1/3/3/3.jpg
...
How I can I download all files and directories with wget?
1
vote
2
answers
1k
views
Using wget and Awk to count similar expressions
I am trying to create a script that uses wget to download a data set and then awk to sort though the file and tell you the most common filter used which is $14 column. So far I have the wget function ...
1
vote
2
answers
3k
views
406 Not Acceptable error when runing wget
I have this command in crontab:
wget --quiet --delete-after http://boms.ro/admincp/cron/s/9abf0f42c1e4f55fdb87d8237cdde
And when I run it with the --debug argument I get the following response:
...
13
votes
1
answer
57k
views
Accessing a windows shared folder from Linux?
I am trying to copy a (.html) file from a Windows XP Professional shared folder onto a server running Ubuntu Linux 10.04 LTS.
As it's a shared folder the usual 'wget' doesn't seem to work. I suspect ...
0
votes
2
answers
2k
views
Curl and Wget returns different response code
I am trying to wget a link, this wget works fine on my local machine but it doesn't do so on the server.
i tried to check the response header and i got the following on my local laptop
curl -I http://...
0
votes
2
answers
2k
views
Use number of downloaded file in wget as variable in bash script
Can I get the number of downloaded files with wget -r and use that as a variable? I want to write a script that runs multiple wget commands in it (using -q so i can control output) and then at the end,...
5
votes
2
answers
13k
views
Ubuntu 11.10, using wget/curl fails with ssl
On a completely new install of Ubuntu I'm getting the following errors when using wget:
wget https://test.sagepay.com
--2012-03-27 12:55:12-- https://test.sagepay.com/
Resolving test.sagepay.com... ...
1
vote
2
answers
1k
views
How to untar a single file from an archive with randomly generated directory names?
I'm trying write a script that will download a tarball from github and extract a single file from it.
However the top level directory inside the tarball has some random characters in it, which I ...
1
vote
2
answers
3k
views
Cron not running commands as root
I have a file called /scripts/checkInternet and it contains:
#!/bin/bash
WGET="/usr/bin/wget"
rm /tmp/index.google
$WGET -q --tries=10
--timeout=5 http://www.google.com -O /tmp/index....
1
vote
1
answer
866
views
How to block some robots and referer pages in Apache root level
When I was using Lighttpd I can easily achieve this by such entries. So all websites were protected.
Wget robots:
$HTTP["useragent"] =~ "Wget" {
$HTTP["url"] =~ "^/tagi(.*)" {
# $HTTP[...
-1
votes
2
answers
3k
views
wget downloading index.html instead of tarball?
When I type the following:
http://www.archlinux.org/packages/extra/x86_64/enca/download/
wget is downloading index.html instead of the correct file which is: enca-1.13-2-x86_64.pkg.tar.xz
When I ...
1
vote
1
answer
13k
views
Wget - download all links from a http location (not recursivly)
I have a link to an http page that has a structure like this:
Parent Directory -
[DIR] _OLD/ 01-Feb-2012 06:05 -
[DIR] _Jan/ ...
1
vote
3
answers
411
views
Limiting specific process to just one per user
My scripts are using wget to retrieve data from Internet. When many users are using this script I got very high load (about 20.00), because of disk I/O. Wget is automatically started each hour by cron....
6
votes
2
answers
58k
views
FTP file download using Wget
I am using the wget command for download FTP files, when i download the FTP file its showing error "Event not fount". Here i use the password like some below charater ! so its showing this error
bash:...
11
votes
3
answers
10k
views
How can i make wget download only pages not css images etc?
I want to download an entire website using wget but I don't want wget to download images, videos etc.
I tried
wget -bqre robots=off -A.html example.com –user-agent=”Mozilla/5.0 (Windows; U; Windows ...
4
votes
1
answer
2k
views
Downloading from wget - folder issue
I have a website - http://myradio1.localhost and the directory is currently empty.
The virtual host works fine, no issues there.
This is my command I'm using to download the site via wget:
wget --...
1
vote
1
answer
11k
views
Wget from HTTPS url using user/password -still getting 400 error
I'm trying to run Wget for a certain HTTP url, also using http-user and http-password command line options (tried the --user and --password, same results).
Using the --no-check-certificate options as ...
9
votes
3
answers
5k
views
How to serve a wget --mirror'ed directory of files with questionmarks in them
I'm trying to create a static mirror of a php application (an old php Gallery installation, specifically). The app produces URLs such as:
view_album.php?set_albumName=MyAlbum
wget downloads these ...
-1
votes
1
answer
2k
views
Using wget to get count of pages below a link?
I've been using a sitemapping tool to get a simple count of links below a specific url. The free trial has ended, so I figure that rather than paying $70 for what is very simple functionality, I ...
3
votes
2
answers
4k
views
Can someone explain to me what "wget -O - -q icanhazip.com" mean?
I've used this command to find the IP of the server. Can someone explain what the command means? I want to learn.
So far I know:
"wget" is a free utility for non-interactive download of files from ...
0
votes
1
answer
97
views
How to download files that are dynamically served
I've run into this a few times when a site serves a variable file i try to download
wget http://trac-hacks.org/changeset/latest/tracajaxcommentsplugin?old_path=/&filename=tracajaxcommentsplugin&...