0
votes
1answer
52 views

Protect public directories from download via wget and curl

My users upload their profile images and I save them in a public folder. The file names are not predictable (they are based on md5 hash of user id with salt). When someone views the user's profile, ...
0
votes
1answer
64 views

Command line tools to monitor POST/GET requests of a loaded website

When I load a website with Firefox and open up the Network Profiler, I can see all the POST and GET requests the site is making. I would like this functionality from the command line. Does any one ...
1
vote
2answers
877 views

curl succeeds on an https request, wget does not

When I run curl from the command line curl "https://example.com" It succeeds immediately, returning the result of the request. When I run the same wget command wget https://example.com ...
-1
votes
1answer
153 views

Perform HTTP request through non-default gateway (CURL? WGET?) [closed]

I have two ISPs /two routers on my LAN with Windows boxes. Switching default gateway (per PC) works fine, so I can choose my outgoing traffic from particular PC to be initiated from whichever ISP I ...
0
votes
1answer
696 views

SSL error curl/wget unknown protocol/wrong version number

I've been trying to use local hosted https urls in command line/cron jobs and i get those errors. curl output is similar. The same commands, when used in other servers works perfectly. Default ...
0
votes
1answer
817 views

Get a URL with variables via Linux Command Line

Sorry if this seems to be a very simple command, but I have been searching for hours without any solutions and exhausted my very limited knowledge of Linux. :-) I need to use some command to process ...
0
votes
1answer
405 views

How to check ip blacklist using curl/wget request to mxtoolbox?

I would like to add cron job to check server ip on blacklist. something like curl 'http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a142.11.193.83' | grep 'you are on a blacklist.' wget ...
1
vote
2answers
54 views

Download and save a file with proof that it was downloaded from a specific server/url on a specific date?

Assuming the file was downloaded over http is this possible? I'm guessing there could be some way to do it using a hash and a timestamp however I'm not sure how you could prove nothing was tampered ...
3
votes
4answers
486 views

Download a file from the shell when button click is required

I have a link to a file that I would like to download from the shell. Unfortunately, the URL redirects to a software license agreement page which requires clicking an accept button. This is fine when ...
1
vote
3answers
9k views

Is there a built-in command-line tool under Windows like wget/curl?

I come from a Linux/Unix background and I have been wondering if Windows has a binary that can download files from the console. I would like to automate a certain process and one of my requirements ...
1
vote
3answers
2k views

Execute curl requests in parallel in bash

What is the best way to execute 5 curl requests in parallel from a bash script? I can't run them in serial for performance reasons.
-1
votes
1answer
246 views

Iptables blocks outgoing HTTP connection

I've setup IPTables with the following script... however, I cannot get outgoing HTTP queries to work (I've tried using lynx & curl). Background - server is hosted remotely (and I'm connecting via ...
0
votes
2answers
416 views

Curl and Wget returns different response code

I am trying to wget a link, this wget works fine on my local machine but it doesn't do so on the server. i tried to check the response header and i got the following on my local laptop curl -I ...
2
votes
2answers
6k views

Ubuntu 11.10, using wget/curl fails with ssl

On a completely new install of Ubuntu I'm getting the following errors when using wget: wget https://test.sagepay.com --2012-03-27 12:55:12-- https://test.sagepay.com/ Resolving test.sagepay.com... ...
3
votes
2answers
3k views

How do I pipe the output of uptime/df to curl?

I want to send the output of uptime and df commands to a web app using cURL. I tried: uptime | curl http://someurl.com -T - But that didn't seem to send the data. What is the proper way to send ...
3
votes
2answers
612 views

Curl always returns same 404 page

No matter what URL I specify for curl I always get the same HTML 404 Error page back. If I use the --verbose option, it looks like curl always connects to the same IP address. $ curl --verbose ...
0
votes
3answers
2k views

download only new files from ftp using WGET/cUrl

i have Windows 2008 Server on which i store DB backups on daily basis. I want to be able to download new files only using wget, curl or windows built-in FTP doesn't matter. Can you help me with ...
2
votes
5answers
2k views

Multithreaded downloading with shell script

Let's say I have a file with lots of URLs and I want to download them in parallel using arbitrary number of processes. How can I do it with bash?
2
votes
2answers
5k views

Alternatives to CURL, WGET

Is anyone aware of any alternatives to curl and wget? The key functional requirement I'm looking for is to be able to execute an HTTPS GET request against a known URL. The reason I'm looking for an ...
2
votes
2answers
2k views

WGet or cURL: Mirror Site from http://site.com And No Internal Access

I have tried wget -m wget -r and a whole bunch of variations. I am getting some of the images on http://site.com, one of the scripts, and none of the CSS, even with the fscking -p parameter. The ...
2
votes
2answers
956 views

Retrieve FTP directory tree from the command line

What would be the fastest way to recursively retrieve entire directory listing from an ftp server using wget/curl/whatever? I don't need to download any files, just directory and file names. Basically ...
0
votes
6answers
684 views

is there a problem in disabling curl and wget?

Is it recommended to disable curl and wget to disallow anyone from accessing my websites using those commands? Or is it needed for some search engines to access my websites? What are your ...
1
vote
0answers
214 views

How to use sitemap.xml to create a static mirror of a CMS

Is there a tool to create a static mirror of a content management system (CMS) that provides a sitemap.xml file? Ideally, I would point a tool like wget or curl to a sitemap.xml file and have it ...
0
votes
3answers
190 views

Cannot download some files from a given server

I'm working on a Fedora Core release 6, and whenever I want to download a given file from a FTP site, the connection is refused: $ curl --ftp-pasv ...
1
vote
2answers
392 views

On Debian server, downloads stall out after one or two seconds using apt-get

I have a Debian server that is used primarily as a very low-traffic web server. Recently it began having some trouble regarding its network connection. I am able to use SFTP to upload a file to this ...