Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm new to cURL and I'm trying to do something reasonably simple. I want to login to a server and download data at regular intervals over a few hours.

First, I login to the server - which I've managed by posting the login values like so:

curl -v -c I:\ccookiejar.txt -d "user=me&password=whatever&Login=submit" http://www.website.com/login_p.asp -o I:\curltest.htm

second, I want to download a html page of data (I'm not worried about the code to do this repeatedly; at the moment, just once would be good) with I thought something like this including the cookies for the session to save me logging in every time.

curl -v -c I:\ccookiejar.txt "http://www.website.com/equities/quotes.asp?refRate=&levels=&Symbol=abc&type=2&v.x=33&v.y=16" -o I:\curltest2.htm

But this doesn't work as a separate command. It seems because cURL closes the connection?

When I include the url for the data I want to download in the same command as the login url it works ok and I get the two htm files output with the kind of data I expect.

Have I made a mistake in my second example? Is there some other way to save me having to login to the site hundreds of times to download my data using the first method?

share|improve this question
    
THanks very much Anagio and Kaji. OH DOH! Yep I was overwriting the cookie file by mistakenly using -c when issuing the curl command to save the html data file. Using -b as written clearly in the doc curl.pdf works so I only have to login once to make multiple downloads Thanks again. Gary –  Gary Jan 25 '12 at 4:55

2 Answers 2

up vote 0 down vote accepted

Is the quotes page in the password protected area of the site? If so isn't -c creating a new cookie file? Wouldn't you want to load the cookie file with -b? I'm not to familiar with curl command line but have had a few scripts developed to use curl multi. You could probably have a simple php script made for a couple of bucks literally that lets you hard code your login and pass then just set it as a cron script to run however often you need.

share|improve this answer
    
Quite right see my comment above –  Gary Jan 25 '12 at 4:56

If you mean the username and password are entered in a form on a login page, then cURL can "submit" that form like:

curl -d "username=yourusername&password=yourpassword" http://yourweb.com/login

and if you want to store the cookie that comes back you do so by specifying a cookie file:

curl -c cookies.txt -d "username=yourusername&password=yourpassword" http://yourweb.com/login

and to use those cookie in later requests you do:

curl -b cookies.txt -d "username=yourusername&password=yourpassword" http://yourweb.com/login

or do both if you want to both send and receive cookies:

curl -b cookies.txt -c cookies.txt -d "username=yourusername&password=yourpassword" http://yourweb.com/login
share|improve this answer
    
Quite right see my comment above –  Gary Jan 25 '12 at 4:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.