1
vote
1answer
60 views

Wget and recursive downloads

A directory listings enabled folder has 2 zip files. I am trying to download only one of them (the latest). Problem is the zip file name is a random string. The command I have so far is wget -r -nH ...
0
votes
0answers
23 views

Use tmux for managing multiple downloaders as a supervisorctl service?

I have many data-servers I need to download data from via http as soon as it is available. For each server I start a bash "while true"-loop and within that a wget to poll the server for new data. To ...
1
vote
1answer
81 views

Why does wget from ftp-sites sometimes create a few empty files when it downloads the rest of the files without problems?

I have downloaded 80.000+ image files stored on a FTP-site using wget(1) and most of the files where downloaded without problems but when I did a small check (using file(1)) I just realized that a few ...
0
votes
2answers
253 views

Not able to wget to ftp server

I am trying to wget a ftp server from a remote machine. The command is not getting past 'Logging in as anonymous'. This is what i am getting. wget ...
0
votes
1answer
130 views

ldirectord NAT load balancing does not work with negotiate checks

I am using ldirectord for loadbalancing two HTTP servers. On the load balancer box i have the following network configuration: eth0 is used for internal purposes and does not take part in load ...
0
votes
1answer
409 views

How to check ip blacklist using curl/wget request to mxtoolbox?

I would like to add cron job to check server ip on blacklist. something like curl 'http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a142.11.193.83' | grep 'you are on a blacklist.' wget ...
3
votes
4answers
490 views

Download a file from the shell when button click is required

I have a link to a file that I would like to download from the shell. Unfortunately, the URL redirects to a software license agreement page which requires clicking an accept button. This is fine when ...
-1
votes
3answers
83 views

How to tar folders with specific letter

I have a long list of folders. i want tar only folder which starts with "a". How can i do that? Actually, I am having photo gallery website, which has 1000s of folders. I wish to move another server ...
0
votes
2answers
113 views

Fast (non-blocking) way to transfer many files to another server [duplicate]

Possible Duplicate: what is the fastest and most reliable way of transferring a lot of files? I am currently attempting to transfer over 1 million files from one server to another. Using ...
5
votes
3answers
2k views

wget starts downloading then stops “cannot write to”

I'm using wget to mirror some files across from one server to another. I'm using the following command: wget -x -N -i http://domain.com/filelist.txt -x = Because I want to keep the directory ...
1
vote
3answers
667 views

Setting wget permission to 755 so users other than root can execute it a big security risk?

I read recently in blogs that by default wget on linux is 750, so only root can execute it. I would like to allow users wget and change it to 755, but I read around the web that it is a big security ...
1
vote
2answers
96 views

Changed DNS, no effect locally

I know it can take up to a few days for DNS changes to take effect, but this has me baffled, maybe somone can offer a plausible explanation.. $ wget http://***OLDIP***/ -O oldserver.html --2012-07-25 ...
-1
votes
2answers
260 views

wget not saving file as the triggered file to disk [closed]

What I am trying to do is download a file from a url like this: http://www.example.com/down/files/425/document-preview? When I visit this url in my browser, it triggers a file called for example: ...
2
votes
2answers
597 views

How can i make wget download only pages not css images etc?

I want to download an entire website using wget but I don't want wget to download images, videos etc. I tried wget -bqre robots=off -A.html example.com –user-agent=”Mozilla/5.0 (Windows; U; Windows ...
1
vote
2answers
356 views

Can someone explain to me what “wget -O - -q icanhazip.com” mean?

I've used this command to find the IP of the server. Can someone explain what the command means? I want to learn. So far I know: "wget" is a free utility for non-interactive download of files from ...
0
votes
1answer
72 views

How to download files that are dynamically served

I've run into this a few times when a site serves a variable file i try to download wget ...
5
votes
3answers
18k views

Why wget doesn't verify SSL certificates?

I have a problem with my Fedora 8 installation. It looks that wget doesn't know how to verify SSL certificates any more. It's strange because I have another Fedora 8 box which I believe has the same ...
3
votes
2answers
3k views

How do I pipe the output of uptime/df to curl?

I want to send the output of uptime and df commands to a web app using cURL. I tried: uptime | curl http://someurl.com -T - But that didn't seem to send the data. What is the proper way to send ...
0
votes
1answer
462 views

wget and connection errors / timeouts

I was using wget the in the last week to recursively download a whole website of html pages. I used it this way: wget --recursive --no-clobber --page-requisites --html-extension --convert-links ...
-1
votes
1answer
660 views

wget --timeout doesn't time out an active connection

When using --timeout with wget, I was expecting that if the transaction hasn't completed within the timeout period, then it would abort. However, it seems to mean (with respect to read-timeout) that ...
2
votes
3answers
310 views

wget application replacement

Is there any other application similar to wget? If yes can you post a link?
1
vote
4answers
234 views

Internet speed in Linux via command

I am running several sites and I was trying wget and I noticed slow speed. So I suspect that my sites are taking considerable amount of traffic. Question is, how I can really detect the total ...
5
votes
1answer
519 views

What to do if wget's progress:mega doen't cut it anymore?

I have script for downloading large files in a queue. Now I would really appreciate it, if I had a decent log. Specifying the -o or -a option is apparently specially made for that, however as of the ...
0
votes
2answers
363 views

DNS issue on Fedora 12? wget wordpress.org fails where wget www.google.com works

I'm administering a Fedora 12 box, but am quite new to networking specifics. Recently one of our WordPress apps hosted on our server has stopped being able to perform its auto-update or auto-download ...
3
votes
6answers
5k views

How to download with wget without following links with parameters

I'm trying to download two sites for inclusion on a CD: http://boinc.berkeley.edu/trac/wiki http://www.boinc-wiki.info The problem I'm having is that these are both wikis. So when downloading with ...
1
vote
1answer
256 views

Hourly CRON task running more frequently than one hour

I have a cron task that calls a special PHP script via wget. Here is the crontab entry: 0 * * * * wget http://www.... It will work perfect for several days, running on the hour. However, after a few ...
0
votes
1answer
396 views

Does crontab file automatically bash a text file?

I have a crontab job setup. In the crontab file, I have a path to a text file. My text file has a wget command (which in turn, executes a PHP file). If the crontab file just has the path to the text ...
0
votes
2answers
215 views

Downloading file with different unique name

How to download file with different unique name each time wget -O image.jpg "http://localhost/cgi-bin/nph-zms?mode=single&monitor=1" Using wget's -O option overwrites file each time. What I ...
2
votes
1answer
3k views

HTTP downloads stop after some time, resuming is not possible

When I try to download a file via HTTP, the downloads sometimes stop after around 30 MB. The download rates goes down to 0 B/s and no data keeps coming. When I stop the download and resume again, the ...
1
vote
4answers
417 views

download files from text file that contain files list under linux

i have a text file that contains a list of urls for files of my word for example : http://domain.com/file1.zip http://domain.com/file2.zip http://domain.com/file3.zip http://domain.com/file4.zip ...
1
vote
6answers
25k views

WGET can't resolve host

I've got 2 machines on the same sub-net running Debian 5.0. They use the same DNS (/etc/resolv.conf), they route to the same gateway (#route) and they have the same iptables settings (#iptables -L). ...
7
votes
6answers
5k views

wget recursive download, but I don't want to follow all links

I'm trying to mirror a website using wget, but I don't want to download lots of files, so I'm using wget's --reject option to not save all the files. However wget will still download all the files and ...
1
vote
3answers
2k views

Downloading php files from python simple http server

I started python -m SimpleHTTPServer on one computer on lan and used wget to download php files from it to another. As far as i see, they seem to be downloaded correctly - i got php sources instead of ...
2
votes
3answers
1k views

The --reject feature in wget doesn't seem to work

I am trying to mirror most of a ftp sever using wget. This particular sever keeps old copies of the data files in a folder called 'backup' inside every subfolder. e.g. '/MaffiaOffShore/backup' ...
11
votes
5answers
14k views

How do I pipe a downloaded file to standard output in bash?

How is it possible to pipe out wget's downloaded file? If not what alternatives should I use? UPD: Thanks all for your answers, I managed to tune my downloader (curl) to send files to stout. However ...