2022 Developer Survey is open! Take survey.

Questions tagged [wget]

GNU Wget (or just Wget, formerly Geturl) is a computer program that retrieves content from web servers, and is part of the GNU Project. Its name is derived from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP protocols.

Filter by
Sorted by
Tagged with
0 votes
1 answer
22 views

Trying to mirror my website with wget, but nofollow attribute is found , and I cannot download anything more than index.html

I am running a wordpress site on a Ubuntu 20.04 based LEMP server. I have the pagespeed plugin enabled, and in order to force it to cache my website, I am using wget from a different box to mirror the ...
user avatar
  • 93
0 votes
0 answers
20 views

wget timeout after updating Debian from buster to bullseye

After the update from Debian Buster to Bullseye I can't use wget anymore. Curl is working fine. I have timeouts and the file couldn't be downloaded or it takes a very long time or retries. root@...
user avatar
0 votes
1 answer
41 views

Move files from a remote ftp server to a local

I need to pull files from an ftp server regularly and have found that I can get the files easily enough using wget wget -m --user=yyy --password=xxxx ftps://host.com.au and that works really nicely. ...
user avatar
0 votes
1 answer
373 views

wget is not able to get the correct ssl certifiicate expiration date

When I execute this wget -O /root/installer.php https://www.grscripts.com/install/installer.deluxe I receive this error ERROR: cannot verify https://www.grscripts.com's/ certificate, issued by ‘/C=...
user avatar
  • 113
0 votes
0 answers
24 views

wget: Edit broken links while crawling before visiting them

Situation: I want to mirror an old website. This website is on https://example.com/website/. The website uses absolute links to http://www.example.com/website/. Problem: For whatever reason, wget ...
user avatar
0 votes
2 answers
228 views

Downloading a file using Windows CMD line with curl/wget

I have a client [Windows 10 VM] and a server [say a linux based VM]. I have Apache running on the Linux Server. I have a file on the linux server that I want to download on my windows client. I want ...
user avatar
0 votes
1 answer
79 views

ICS Calendar File Does not Exist

I'm trying to load this calendar: https://bushnellbeacons.com/calendar.ics to another internal websystem. The calendar feed is fine - I can click the link above and download the feed. The problem is ...
user avatar
  • 656
0 votes
1 answer
55 views

wget does not recurse when piping the output to stdout [closed]

I want to download webpages recursively and pipe the output to a filter. I am using: wget -qm -O- http://mywebsite.com/initialpath.php | ./filter But wget stops downloading after the first page and ...
user avatar
  • 111
0 votes
1 answer
156 views

How can I tell if wget can support LetsEncrypt's new ISRG Root X1 certificate after DST Root CA X3 Expiration

A few weeks ago (Sept 2021), LetsEncrypt changed how their certificates were signed, which can affect some older programmes and clients. I have a standard Apache webserver (stock v2.4.41 from ubuntu ...
user avatar
  • 29.9k
0 votes
0 answers
88 views

Can wget ignore certificate issues on redirects (or for specific domains)?

We use a redirector service so that requests act like this: https://realsite1.redirector.ourdomain --302--> https://realsite1.anotherdomain1 https://realsite2.redirector.ourdomain --302--> https:...
user avatar
  • 2,571
0 votes
1 answer
193 views

PXE boot pmagic with wget or TFTP after kernel

I wish to boot parted magic by PXE. I have read this article on the topic: https://partedmagic.com/pxe/ where it states that you can either load the entire iso, prepare the squashfs file and load it ...
user avatar
  • 111
0 votes
1 answer
482 views

wget failed: Connection timed out

I have the following command to copy the website, as it tried to hit sun.com it got connection timed out. I would like the wget to exclude the sun.com so that wget would proceed to the next thing. ...
user avatar
  • 1
0 votes
1 answer
333 views

Troubleshooting nginx requests not reaching server

When I browse from other servers using wget -U "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" --spider http://server_ip/page --no-hsts This is the response Spider mode enabled. ...
user avatar
0 votes
0 answers
85 views

wget or curl timeout after connection established, before download begins

I need to download data from a server in batch operations. The server shows the following behaviour with approximately 30 % of the downloads: DNS works, host gets resolved. Connection gets ...
user avatar
-1 votes
2 answers
65 views

Backup my webspace to NAS [closed]

First of all: I don't know if this is the correct place to ask, please tell me where I can go with this before you remove my post, that would be great, thank you! ​ What I want to do (and what my ...
user avatar
  • 113
0 votes
1 answer
159 views

Download with Wget only if new version

Good morning, I have a custom software that updates with a custom script.sh. Part of the file goes something like this: if [[ $software == A ]] then echo "downloading package..." ...
user avatar
0 votes
1 answer
358 views

wget -m delete local files that are not existing anymore on the server

I use wget -m to mirror my webspace to my nas. So far it's working great, but I noticed that when a file on the server got deleted, it will be still there locally. Is there any way to delete the local ...
user avatar
  • 113
0 votes
1 answer
731 views

Can I make curl (or wget) ignore a specific TLS error? (bypassing expired certificate without disabling cert validation)

I'm working on a script which depends on a remote API endpoint which I do not control. Today, my script stopped working, because the endpoint's SSL certificate expired today and they haven't yet fixed ...
user avatar
  • 151
0 votes
0 answers
54 views

Wget- Skip hosts that are down

I'm using wget to bulk download a website, and it grabs files from other servers, however, some hosts are down. All wget does is just hang there, so I'd like it to just skip these websites. Is there ...
user avatar
  • 1
2 votes
0 answers
41 views

Is there a tool I can use to test out multiple vpn tunnels in one machine?

I use wget -q -O https://checkmyip.com/csv to determine if I am connected to a VPN but I want to verify multiple VPN tunnels on the same machine. Is there a tool that can allow me to send a command ...
user avatar
  • 171
2 votes
2 answers
129 views

Can't serve static files with ampersand in name

I have a website mirrored with wget and I'd like to serve it via Apache. The problem is that I can't access files with special chars. For example, I have files with names like this: $ ls -N components/...
user avatar
0 votes
1 answer
120 views

wget website recusively from localhost without using bandwith

i'm looking to download recusively my wordpress website into static using wget, the problem is that whenever i do that, it's using too much bandwith (3.5gb) even though i end up downloading 20mb which ...
user avatar
  • 99
0 votes
0 answers
2k views

Read error (Connection reset by peer) in headers on Vagrant

Trying to install aaPanel into a vagrant machine. vagrant box used is Scotch box 3.5 which preloaded with php v7.0. I've updated php to v7.4 using sury's ppa and working fine. I've used given ...
user avatar
  • 101
1 vote
1 answer
438 views

How to save output file to /dev/null with axel command?

I want to download a file from another server and then save it to /dev/null in order to increase monthly traffic of the first server. As man axel says, I should either use -o file or --output=file, ...
user avatar
  • 163
0 votes
0 answers
277 views

How to download all files from an FTP directory?

I am not experienced in unix or downloading files from FTP, html etc. so haven't been sure if I can safely use some other examples/questions given here. I am looking to download files from an ftp ...
user avatar
  • 101
0 votes
0 answers
547 views

curl + wget cannot make https connections

My Ubuntu 16.04 system has the following problem: wget and curl hang indefinitely when connecting to a server with HTTPS Both programs work fine with HTTP Example for curl: curl -vv https://google....
user avatar
  • 51
1 vote
1 answer
2k views

curl and wget fail to verify wildcard Let's Encrypt certificate, but browsers work fine

We've obtained a wildcard certificate from Let's Encrypt for civility.social and *.civility.social, using certbot. This works fine on all browsers and with curl and wget when verifying https://...
user avatar
0 votes
0 answers
38 views

scripting download of teradata files

I'm trying to automate the build of a server and am having troubles scripting out the download of the teradata tools and utilities for linux from this site https://downloads.teradata.com/download/...
user avatar
1 vote
1 answer
619 views

How To Use Wget Command To Check Server Cache Settings

I'm having trouble with a server side cache with an OpenLiteSpeed server. I've tried the following .htaccess trick to try and turn this cache off. Can you tell me a wget command I can use to test my ...
user avatar
0 votes
1 answer
49 views

wget specific directory with downloading whole path

I try to download data using this command. wget -r --no-parent http://myserver/username/data/ This leaves me with three folders on my server, namely: myserver/username/data/ How do I download only ...
user avatar
0 votes
2 answers
344 views

Use wget on a cluster with ssh-tunnel [closed]

Normally I can sercure copy files from one machine to another using > scp -oProxyJump=user@login.node.org ssh user@main.node.org:/home/user/my_files/* . which is very slow for large data sets. I ...
user avatar
0 votes
2 answers
146 views

Why ping statment doesn't work on this Linux Ubuntu VM? [closed]

I am not so into networking and I have the following doubt. They provide me a Linux Ubuntu 18.04.3 LTS VM on which I have to install some tools. The first thing that I tried was related to the ...
user avatar
0 votes
1 answer
265 views

Connect to ubuntu port from docker container

I have a application running on port 8180 on my local machine When I try to wget http://192.168.1.12:8180/auth from docker container, I get following error --2019-10-10 08:17:03-- http://192.168.1....
user avatar
0 votes
1 answer
120 views

How can I force `wget` to clobber a previously-downloaded file?

I'm trying to force wget to clobber a download and I haven't been able to make it do it with what I can find in the man pages. I'd like something cleaner than writing to and redirecting STDOUT. Anyone ...
user avatar
  • 26
1 vote
1 answer
1k views

PASV FTP fails for wget/ftp but works for curl/lftp

I have a server running Centos 7 and vsftpd-3.0.3 (installed from source to get the latest version, though everything I describe was also applicable to the vsftpd-3.0.2 version that was previously ...
user avatar
  • 323
7 votes
1 answer
833 views

Why does Amazon Linux 2 have older version of wget vs. Amazon Linux 1?

We noticed that wget on Amazon Linux 2 is v1.14, but on the older version of Amazon Linux, it was v1.18. Any idea why this would be? I'm fairly certain we installed from yum in both cases. Is there ...
user avatar
0 votes
1 answer
802 views

Git clone, wget, ping not working on GCP server all of a sudden

My Google Cloud Platform server is suddenly unable to download anything via wget, git clone, and unable to apt-get install anything. --2019-05-07 14:48:02-- http://google.com/ Resolving google.com (...
user avatar
  • 101
-3 votes
1 answer
3k views

Can I assign the special IP address when I use wget?

I have a multi-IP addresses Server, a main IP 102.1.1.1 and a C segment 103.1.1.0/24 IP addresses on it. when I use wget www.google.com can I assign my special ip address in my Server like ping? ...
user avatar
  • 268
0 votes
0 answers
5k views

CentOS 7: curl and wget not working

My problem is that curl and wget usually fail to retrieve more than just the first part of any given file, and often nothing at all. Both have worked previously on the same server, and nothing else on ...
user avatar
1 vote
1 answer
3k views

Upgrading wget on Ubuntu 16.04 server

I'm trying to upgrade my copy of wget on the server. Currently its running: 1.17.1-1ubuntu1.4 The problem is that I need 1.19 so I can get a new feature: http://tomszilagyi.github.io/2017/02/Wget-...
user avatar
19 votes
6 answers
11k views

Why the discrepancy between Speedtest and Wget?

My customer complains about low internet speeds. When measured with Speedtest.net speeds are acceptable. Periodic measured downloads are 10% to 30% of the nominal speed. I cannot explain that. Some ...
user avatar
4 votes
2 answers
4k views

Why does curl work with a specific https site, but wget has problems with certificates?

Centos 6. After updating recently ca-certificates I run into some troubles. I have the most recent versions of curl for centos 6: -bash-4.1$ curl -V -v curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7....
user avatar
  • 231
0 votes
0 answers
106 views

create Linux user to give only wget permission

I own a dedicated server and I'm working on my server along some colleagues. I want to create some users for my colleagues which only be able to use wget command. I mean I don't want to give them ...
user avatar
  • 163
2 votes
2 answers
540 views

`wget`-ting a website for "local" browsing on a different domain

I need to mirror a website and deploy the copy under a different domain name. The mirroring procedure should be all automatic, so that I can update the copy on a regular basis with cron. The mirror ...
user avatar
0 votes
2 answers
2k views

What does the -c in wget -c Linux command mean?

I am installing Artifactory on an Ubuntu server, and trying to add JFrog repository key. The command is: wget -c -O- "https://bintray.com/user/downloadSubjectPublicKey?username=jfrog" | sudo apt-key ...
user avatar
2 votes
1 answer
621 views

Automated export of phpMyAdmin SQL download with wget

I'm using the following script to attempt to download an SQL export from a phpMyAdmin installation (adapted from this Stackoverflow question) read -p "Username: " USERNAME read -p "Password: " ...
user avatar
4 votes
0 answers
11k views

How to Enable JavaScript for wget in linux for grabbing a website?

I use wget like this for save site : wget --page-requisites --no-parent --mirror http://example.com/index.html -P /home/ in Some cases It does NOT work, Error is : This site requires Javascript to ...
user avatar
4 votes
5 answers
22k views

Fixing wget certificates

Basically, I was testing different versions of OpenSSL, and somewhere in the process broke wget's path to certificates. wget https://www.google.com Results now in: ERROR: cannot verify www.google....
user avatar
  • 143
0 votes
1 answer
2k views

wget download only certain folders from site

I want to download this site with wget, and to do this I'm using this command: wget.exe --no-clobber --no-parent -r --convert-links --page-requisites -U Mozilla "http://www.s-manuals.com/smd/" That'...
user avatar
  • 121
0 votes
1 answer
5k views

Why can't I get wget to pipe standard output to /dev/null?

When I run this command it works as expected: wget -p https://www.google.com/ After running this command I check the error code with echo $? This returns 0, indicating that last command ran ...
user avatar

1
2 3 4 5 6