[lug] Help on scripting interactions with a web site

[Chip Atkinson]

I believe that Perl has some web client/browser tools/modules available. 
  I haven't used them but remember seeing them as I was going through 
the list of available modules.
Try looking at the web page to find the name of the variables and see if 
you can put together a url using get.  You might be able to make it 
something simple and regular if the names for the reports are regular.

Hope that helps a tiny bit.

Chip

Phil Rasch wrote:

> I want to acquire some datasets from a web site. 
> 
> Unfortunately, the web site is designed so that in order to acquire
> the approximately 500 text files containing the data one must interact
> with a server at that site repeatedly, and then finally cut and paste
> the displayed data from a browser window into a file. I suspect the
> web site is using CGI scripts in the procedure because the final
> dataset does not show an html address that changes. Things stay the
> same for the last 4 or so interactive choices.
> 
> I am frustrated by the whole thing. It is a waste of my (or a
> support persons) time to have to do this, and the opportunity for
> mistakes is very high.
> 
> I have contacted both the webmaster for the site, and the
> investigators, and my sense is that they dont want to make it easy to
> acquire the data. They are however contractually constrained to make
> the data publicly available. They just dont have to make it easy.
> 
> So I am looking for a way around my problems. I want to script the exchange. So
> I just enter the relevant info in the script (e.g. the years, the
> stations, the destination, etc) and the whole thing goes on automatically
> from my end. As far as the web server is concerned somebody is sitting
> at the my end. But in reality a program is handling the transaction.
> 
> Can anybody make a suggestion on the right tool?
> 
> Thanks
> 
> Phil