0
votes
0answers
18 views

Use tmux for managing multiple downloaders as a supervisorctl service?

I have many data-servers I need to download data from via http as soon as it is available. For each server I start a bash "while true"-loop and within that a wget to poll the server for new data. To ...
0
votes
1answer
52 views

Unable to get cron to append output from wget

I have a cronjob setup and running, but the logfile keeps getting overwritten every time it runs. I would like for the wget output to be appended each time the cron runs. Why is this and how do I fix ...
0
votes
1answer
71 views

What could block the output of a command?

Sometimes I'll attempt a (2>&1) redirection and some/all of the resulting output appears to be silenced. e.g. wget -O- http://localhost/test.txt 2>&1 I would expect to see a merge of ...
1
vote
3answers
307 views

error while executing bash script -> command not found

i made this script here, and get the error IA-exporto.sh: 13: wget: not found i tried changing the " by ` and mixed all together and rearranged, but it just won't do.. #!/bin/bash ...
1
vote
2answers
280 views

Using wget and Awk to count similar expressions

I am trying to create a script that uses wget to download a data set and then awk to sort though the file and tell you the most common filter used which is $14 column. So far I have the wget function ...
0
votes
2answers
553 views

Use number of downloaded file in wget as variable in bash script

Can I get the number of downloaded files with wget -r and use that as a variable? I want to write a script that runs multiple wget commands in it (using -q so i can control output) and then at the ...
1
vote
2answers
793 views

Cron not running commands as root

I have a file called /scripts/checkInternet and it contains: #!/bin/bash WGET="/usr/bin/wget" rm /tmp/index.google $WGET -q --tries=10 --timeout=5 http://www.google.com -O ...
1
vote
5answers
2k views

Multithreaded downloading with shell script

Let's say I have a file with lots of URLs and I want to download them in parallel using arbitrary number of processes. How can I do it with bash?
2
votes
2answers
954 views

Why is this bash command not echoing into a variable and what can I do to improve?

I've got a bash script I'm working to improve, and have put on a great fix thanks to Dennis Williamson. Unfortunately, one of the lines no longer echoes into a variable I can manipulate, rather dumps ...
2
votes
1answer
216 views

Why does the time it takes to run this script differ so much between user and system and how I might correct the discrepancy?

My application sits behind a load balancer, and every once in a while I like to do a status check on each machine to get an idea of the time it takes to return an index.html document on each machine. ...
5
votes
1answer
476 views

What to do if wget's progress:mega doen't cut it anymore?

I have script for downloading large files in a queue. Now I would really appreciate it, if I had a decent log. Specifying the -o or -a option is apparently specially made for that, however as of the ...
2
votes
4answers
2k views

Escape a ! in the password parameter of wget

I'm trying to execute something like this: wget --user=foo --password=bar! url The ! in the password is causing problems. I've tried escaping it with \, as in --password=bar\! I've tried ...
0
votes
1answer
393 views

Does crontab file automatically bash a text file?

I have a crontab job setup. In the crontab file, I have a path to a text file. My text file has a wget command (which in turn, executes a PHP file). If the crontab file just has the path to the text ...
11
votes
5answers
13k views

How do I pipe a downloaded file to standard output in bash?

How is it possible to pipe out wget's downloaded file? If not what alternatives should I use? UPD: Thanks all for your answers, I managed to tune my downloader (curl) to send files to stout. However ...