Questions tagged [wget]
GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.
294
questions
2
votes
3
answers
6k
views
wget and port forwarding
I have a weird problem. I can run this command just fine from various servers:
wget --debug '--http-user=USER123' '--http-passwd=PASSWORD' http://GW-BOX:9091/weijhkdsvn/v9_odbc//CRONTAB.2014020
...
- 264
2
votes
1
answer
220
views
Find out what's running wget on my server
In my access.log I see requests coming from 127.0.0.1 and retrieving server's "/" and then all of graphics and CSS referenced in it. This happens regularly, every 5 minutes.
I want to find out what, ...
2
votes
0
answers
160
views
Is there a tool I can use to test out multiple vpn tunnels in one machine?
I use wget -q -O https://checkmyip.com/csv to determine if I am connected to a VPN but I want to verify multiple VPN tunnels on the same machine. Is there a tool that can allow me to send a command ...
- 181
2
votes
0
answers
2k
views
Recursively get files over FTP with explicit SSL
Is there a way to recursively get a whole directory from another server using FTP with explicit SSL on Linux? I have tried a number of wget combinations including with and without the passive flag ...
- 183
2
votes
1
answer
1k
views
ShellShock test shows wget and curl access
I've fixed shellshock bug on my Debian 6 server and while testing on http://shellshock.brandonpotter.com/ I get "No Vulnerabilities Found" and that's OK but they also check other things and in the ...
- 23
2
votes
2
answers
9k
views
WGet or cURL: Mirror Site from http://site.com And No Internal Access
I have tried wget -m wget -r and a whole bunch of variations. I am getting some of the images on http://site.com, one of the scripts, and none of the CSS, even with the fscking -p parameter. The ...
- 1,944
2
votes
0
answers
556
views
How to use sitemap.xml to create a static mirror of a CMS
Is there a tool to create a static mirror of a content management system (CMS) that provides a sitemap.xml file?
Ideally, I would point a tool like wget or curl to a sitemap.xml file and have it ...
- 21
2
votes
3
answers
2k
views
The --reject feature in wget doesn't seem to work
I am trying to mirror most of a ftp sever using wget.
This particular sever keeps old copies of the data files in a folder called 'backup' inside every subfolder. e.g. '/MaffiaOffShore/backup' '/...
WgetMonkey
2
votes
1
answer
770
views
Automated export of phpMyAdmin SQL download with wget
I'm using the following script to attempt to download an SQL export from a phpMyAdmin installation (adapted from this Stackoverflow question)
read -p "Username: " USERNAME
read -p "Password: " ...
- 641
2
votes
2
answers
3k
views
Cron getting timeout when attempting to access url (wget).
I'm sure this is simple and I've been digging but no answer is quite as specific as I need it to be.
The goal is easy, have Cron hit a url on my server every 5 minutes. All of that is set up and ...
- 19
1
vote
2
answers
20k
views
What does -q and -O mean?
What does -q and -O mean in the following cronjob?
*/30 * * * * /usr/bin/wget -q -O /dev/null http://example.com/moodle/admin/cron.php
- 333
1
vote
1
answer
13k
views
Wget - download all links from a http location (not recursivly)
I have a link to an http page that has a structure like this:
Parent Directory -
[DIR] _OLD/ 01-Feb-2012 06:05 -
[DIR] _Jan/ ...
- 255
1
vote
4
answers
2k
views
download files from text file that contain files list under linux
i have a text file that contains a list of urls for files of my word
for example :
http://domain.com/file1.zip
http://domain.com/file2.zip
http://domain.com/file3.zip
http://domain.com/file4.zip
......
- 113
1
vote
3
answers
2k
views
cron job running repeatedly, why?
Why is this cron job executing repeatedly over time and what can I do to stop it?
I have a cron job that is supposed to run at 4 each morning. It hits a php script that executes some daily data ...
- 1,677
1
vote
3
answers
4k
views
Downloading php files from python simple http server
I started python -m SimpleHTTPServer on one computer on lan and used wget to download php files from it to another. As far as i see, they seem to be downloaded correctly - i got php sources instead of ...
- 2,019
1
vote
1
answer
6k
views
wget and pipe to tar in the background
I am trying to download a large file which is .tar.gz and (> 2GB) in size. The bash script also does lots of other things, so I want to kick off the download and then continue processing other ...
- 5,438
1
vote
3
answers
5k
views
Setting wget permission to 755 so users other than root can execute it a big security risk?
I read recently in blogs that by default wget on linux is 750, so only root can execute it. I would like to allow users wget and change it to 755, but I read around the web that it is a big security ...
- 1,877
1
vote
3
answers
1k
views
error while executing bash script -> command not found
i made this script here, and get the error
IA-exporto.sh: 13: wget: not found
i tried changing the " by ` and mixed all together and rearranged, but it just won't do..
#!/bin/bash
UNAME="...
- 402
1
vote
1
answer
655
views
wget only works for some users
Currently have a strange situation: wget only works for one user on a system -- and root isn't it.
There are three user accounts, user A, B and root. User A can use wget with no problems. User B and ...
- 340
1
vote
3
answers
7k
views
download only new files from ftp using WGET/cUrl
i have Windows 2008 Server on which i store DB backups on daily basis. I want to be able to download new files only using wget, curl or windows built-in FTP doesn't matter.
Can you help me with ...
- 410
1
vote
2
answers
218
views
Has anyone experienced unintentional traffic overload due to clients' use of wget?
I am wondering if it is wise to indiscriminately block wget on servers that I maintain.
Has anyone experienced unintentional traffic overload due to clients' use of wget?
- 119
1
vote
3
answers
2k
views
Crontab repeats every 15 min
My cronjob is being repeated every 15 minutes and I don't have any idea why.
Here is the crontab file with real commands replaced:
# m h dom mon dow command
# Comment
0 9 * * * command arg1
0 14 *...
- 145
1
vote
2
answers
1k
views
Using wget and Awk to count similar expressions
I am trying to create a script that uses wget to download a data set and then awk to sort though the file and tell you the most common filter used which is $14 column. So far I have the wget function ...
- 11
1
vote
2
answers
3k
views
406 Not Acceptable error when runing wget
I have this command in crontab:
wget --quiet --delete-after http://boms.ro/admincp/cron/s/9abf0f42c1e4f55fdb87d8237cdde
And when I run it with the --debug argument I get the following response:
...
- 105
1
vote
2
answers
3k
views
Cron not running commands as root
I have a file called /scripts/checkInternet and it contains:
#!/bin/bash
WGET="/usr/bin/wget"
rm /tmp/index.google
$WGET -q --tries=10
--timeout=5 http://www.google.com -O /tmp/index....
user74078
1
vote
4
answers
280
views
Internet speed in Linux via command
I am running several sites and I was trying wget and I noticed slow speed. So I suspect that my sites are taking considerable amount of traffic.
Question is, how I can really detect the total ...
TDSii
1
vote
1
answer
3k
views
Changing all HTML absolute links to relative [closed]
I have a website that has a bunch of absolute addresses, and I need to move it (the whole contents of that website) up a level, so all of the absolute links need to be converted to relative.
I know ...
- 13
1
vote
1
answer
18k
views
SSL error curl/wget unknown protocol/wrong version number
I've been trying to use local hosted https urls in command line/cron jobs and i get those errors. curl output is similar. The same commands, when used in other servers works perfectly.
Default
[root@...
1
vote
3
answers
416
views
Limiting specific process to just one per user
My scripts are using wget to retrieve data from Internet. When many users are using this script I got very high load (about 20.00), because of disk I/O. Wget is automatically started each hour by cron....
- 568
1
vote
4
answers
3k
views
wget an remote website index directory
Is it possible to wget an entire index/parent directory with wget? Because I am trying to transfer images to a VPS of mine from my old shared host.
When I say an index/parent directory, I meant like ...
- 2,141
1
vote
1
answer
2k
views
PASV FTP fails for wget/ftp but works for curl/lftp
I have a server running Centos 7 and vsftpd-3.0.3 (installed from source to get the latest version, though everything I describe was also applicable to the vsftpd-3.0.2 version that was previously ...
- 333
1
vote
2
answers
5k
views
Why does downloading a large file with wget seem to consume nearly all available RAM? [closed]
I am trying to download a file of 580MB with the following simple wget command:
wget http://example.com/file.ext
The server has 16 GB installed RAM of and during the entire download the memory usage ...
- 29
1
vote
1
answer
11k
views
Wget from HTTPS url using user/password -still getting 400 error
I'm trying to run Wget for a certain HTTP url, also using http-user and http-password command line options (tried the --user and --password, same results).
Using the --no-check-certificate options as ...
- 111
1
vote
3
answers
6k
views
wget is working only when used with sudo
I'm having quite a strange behavior with wget since yesterday.
I can download files by using sudo wget, but when I try the same file with only wget, I can get this error:
yusufh@ubuntu-yuh:~$ wget ...
- 205
1
vote
2
answers
5k
views
How to tell wget to use the name of the target file behind the HTTP redirect?
Let's a consider link such as htttp://www.mysite.net/download.php?id=267 that redirects to htttp://www.mysite.net/downloads/myfile.tar.gz
If I execute 'wget "htttp://www.mysite.net/download.php?id=...
1
vote
3
answers
524
views
Cannot download some files from a given server
I'm working on a Fedora Core release 6, and whenever I want to download a given file from a FTP site, the connection is refused:
$ curl --ftp-pasv "ftp://ftp.ensembl.org/pub/current_mysql/...
- 429
1
vote
2
answers
2k
views
Trying to mirror my website with wget, but nofollow attribute is found , and I cannot download anything more than index.html
I am running a wordpress site on a Ubuntu 20.04 based LEMP server.
I have the pagespeed plugin enabled, and in order to force it to cache my website, I am using wget from a different box to mirror the ...
- 93
1
vote
2
answers
6k
views
vshpere wget: bad address 'releases.ubuntu.com'
on a raw vsphere esxi 6 without any vm installed i have to setup vm's and ... remotely so i decided to open ssh to esxi and download an Ubuntu iso using wget this way:
wget http://releases.ubuntu....
- 31
1
vote
1
answer
807
views
How do I ensure that a CoreOS cloud-config service is able to download files?
I am defining a one-shot service in my CoreOS cloud-config, but it is failing due to not being able to download files from Google Cloud Storage (via wget):
Apr 13 11:09:56 staging-node-ys9y.c....
- 2,145
1
vote
1
answer
7k
views
wget giving Segmentation fault
I recently did apt-get update followed by apt-get upgrade on my Debian VPS server. Suddenly (at least) wget stopped working with error Segmentation fault. It's not even trying to do something. So ...
- 11
1
vote
2
answers
1k
views
How to untar a single file from an archive with randomly generated directory names?
I'm trying write a script that will download a tarball from github and extract a single file from it.
However the top level directory inside the tarball has some random characters in it, which I ...
- 121
1
vote
2
answers
1k
views
Download file in Linux when file location is unknown
Sorry if this is unclear, but I'm trying to set up a script that downloads a file. Currently, my method of downloading the file is by clicking on a link like so:
https://www.URL.com/view?downloadFile=...
- 325
1
vote
3
answers
9k
views
wget: download a website which is password protected using a form
I have a problem with downloading a website using wget.
First I need to login to the website with a username and password:
wget --save-cookies cookies.txt --post-data --cookies=on --keep-session-...
- 11
1
vote
1
answer
3k
views
wget not getting all files
I have issues using wget,
basically I use it like:
wget -m ftp://username:[email protected]:/path/to/dirs
After everything's complete (it says xxx files in xx MB for xx seconds), however, when ...
- 219
1
vote
3
answers
828
views
File timestamp not changing even with touch
I have the following problem. I have a bash script that uses wget to get a few files from the build server, and then scp's them over to the production system: Here is the relevant snippet:
#Several ...
- 708
1
vote
1
answer
662
views
Hourly CRON task running more frequently than one hour
I have a cron task that calls a special PHP script via wget. Here is the crontab entry:
0 * * * * wget http://www....
It will work perfect for several days, running on the hour. However, after a few ...
- 915
1
vote
1
answer
482
views
On Debian server, downloads stall out after one or two seconds using apt-get
I have a Debian server that is used primarily as a very low-traffic web server. Recently it began having some trouble regarding its network connection. I am able to use SFTP to upload a file to this ...
- 21
1
vote
1
answer
3k
views
curl and wget fail to verify wildcard Let's Encrypt certificate, but browsers work fine
We've obtained a wildcard certificate from Let's Encrypt for civility.social and *.civility.social, using certbot. This works fine on all browsers and with curl and wget when verifying https://...
1
vote
1
answer
1k
views
How To Use Wget Command To Check Server Cache Settings
I'm having trouble with a server side cache with an OpenLiteSpeed server. I've tried the following .htaccess trick to try and turn this cache off. Can you tell me a wget command I can use to test my ...
- 1,518
1
vote
1
answer
6k
views
Upgrading wget on Ubuntu 16.04 server
I'm trying to upgrade my copy of wget on the server. Currently its running:
1.17.1-1ubuntu1.4
The problem is that I need 1.19 so I can get a new feature:
http://tomszilagyi.github.io/2017/02/Wget-...
- 1,184