[lug] Using curl to get at pages that require a cookie

[Chip Atkinson]

You probably have to use the -c option in conjunction with the -b.  I
believe that -b puts the retrieved cookies in the file but the -c tells
curl to retrieve and use the cookies from the same file.  

If this doesn't work, you may need to analyze the transaction more
closely.  I have used webscarab in the past and found it quite useful.  It
allows you to see what is being sent to the web server.

The curl command can get quite long when you include all the stuff that is
sent.  Here's an example that I put together for an automated web browsing
script:

  curl $QUIET -k -b ${RESULTS_DIR}/cookiejar -c ${RESULTS_DIR}/cookiejar
--create-dirs -o "$FILE" -H 'Accept: image/gif, image/x-xbitmap,
image/jpeg, image/pjpeg, application/vnd.ms-excel,
application/vnd.ms-powerpoint, application/msword,
application/x-shockwave-flash, */*' -e
"https://$HOST/index.jsp?action=ruleswin" -H 'Accept-Language: en-us' -H
'Connection: Keep-Alive' -A 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.0; .NET CLR 1.1.4322)' -H "Host: $HOST" -H 'X-Forwarded-For: 10.1.54.71'
"https://$HOST:443/index.jsp?action=actionwin&reset_session_timeout=true"

Chip

On Thu, 2 Nov 2006, Bill Thoen wrote:

> I'm trying to use curl to download a series of HTTP pages that require a 
> password at a top-level page and then I guess they use cookies to allow 
> access to the lower pages. When I log in via my browser and enter the 
> password, I can get to all the pages. But when I try curl 
> -b"/.../firefox/.../cookies.txt" http://my.url.com/?page=A it's not 
> working. (I substitute the real filename for cookies.txt and the full 
> sub-page URL when I do this for real.) Also I'm logged in via my browser 
> and have entered the password, so I think my cookie for this site is 
> up-to-date.
> 
> Am I using the -b option incorrectly?  Can anyone provide some help with 
> this?
> 
> - Bill Thoen
> 
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: lug.boulder.co.us port=6667 channel=#colug
>