[lug] Help on scripting interactions with a web site

[bdoctor at ps-ax.com]

I would look into LWP if you are comfortable with Perl:

http://search.cpan.org/search?mode=module&query=lwp

-brad

> I want to acquire some datasets from a web site. 
> 
> Unfortunately, the web site is designed so that in order to acquire
> the approximately 500 text files containing the data one must interact
> with a server at that site repeatedly, and then finally cut and paste
> the displayed data from a browser window into a file. I suspect the
> web site is using CGI scripts in the procedure because the final
> dataset does not show an html address that changes. Things stay the
> same for the last 4 or so interactive choices.
> 
> I am frustrated by the whole thing. It is a waste of my (or a
> support persons) time to have to do this, and the opportunity for
> mistakes is very high.
> 
> I have contacted both the webmaster for the site, and the
> investigators, and my sense is that they dont want to make it easy to
> acquire the data. They are however contractually constrained to make
> the data publicly available. They just dont have to make it easy.
> 
> So I am looking for a way around my problems. I want to script the exchange. So
> I just enter the relevant info in the script (e.g. the years, the
> stations, the destination, etc) and the whole thing goes on automatically
> from my end. As far as the web server is concerned somebody is sitting
> at the my end. But in reality a program is handling the transaction.
> 
> Can anybody make a suggestion on the right tool?
> 
> Thanks
> 
> Phil
> 
> -- 
> Phil Rasch, Climate Modeling Section, National Center for Atmospheric Research
> Mail     --> P.O. Box 3000, Boulder CO 80307  
> Shipping --> 1850 Table Mesa Dr, Boulder, CO 80305
> email: pjr at ucar.edu, Web: http://www.cgd.ucar.edu/cms/pjr Phone: 303-497-1368, FAX: 303-497-1324 
> 
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>