[lug] Help on scripting interactions with a web site

[Phil Rasch]

I want to acquire some datasets from a web site. 

Unfortunately, the web site is designed so that in order to acquire
the approximately 500 text files containing the data one must interact
with a server at that site repeatedly, and then finally cut and paste
the displayed data from a browser window into a file. I suspect the
web site is using CGI scripts in the procedure because the final
dataset does not show an html address that changes. Things stay the
same for the last 4 or so interactive choices.

I am frustrated by the whole thing. It is a waste of my (or a
support persons) time to have to do this, and the opportunity for
mistakes is very high.

I have contacted both the webmaster for the site, and the
investigators, and my sense is that they dont want to make it easy to
acquire the data. They are however contractually constrained to make
the data publicly available. They just dont have to make it easy.

So I am looking for a way around my problems. I want to script the exchange. So
I just enter the relevant info in the script (e.g. the years, the
stations, the destination, etc) and the whole thing goes on automatically
from my end. As far as the web server is concerned somebody is sitting
at the my end. But in reality a program is handling the transaction.

Can anybody make a suggestion on the right tool?

Thanks

Phil

-- 
Phil Rasch, Climate Modeling Section, National Center for Atmospheric Research
Mail     --> P.O. Box 3000, Boulder CO 80307  
Shipping --> 1850 Table Mesa Dr, Boulder, CO 80305
email: pjr at ucar.edu, Web: http://www.cgd.ucar.edu/cms/pjr Phone: 303-497-1368, FAX: 303-497-1324