[lug] Help on scripting interactions with a web site

[Walter Pienciak]

On Tue, 18 Sep 2001, Chip Atkinson wrote:

> I believe that Perl has some web client/browser tools/modules available.
>   I haven't used them but remember seeing them as I was going through
> the list of available modules.
> Try looking at the web page to find the name of the variables and see if
> you can put together a url using get.  You might be able to make it
> something simple and regular if the names for the reports are regular.
>
> Hope that helps a tiny bit.
>
> Chip
>
> Phil Rasch wrote:
>
> > I want to acquire some datasets from a web site.
> >
> > Unfortunately, the web site is designed so that in order to acquire
> > the approximately 500 text files containing the data one must interact
> > with a server at that site repeatedly, and then finally cut and paste
> > the displayed data from a browser window into a file. I suspect the
> > web site is using CGI scripts in the procedure because the final
> > dataset does not show an html address that changes. Things stay the
> > same for the last 4 or so interactive choices.
> >
> > I am frustrated by the whole thing. It is a waste of my (or a
> > support persons) time to have to do this, and the opportunity for
> > mistakes is very high.
> >
> > I have contacted both the webmaster for the site, and the
> > investigators, and my sense is that they dont want to make it easy to
> > acquire the data. They are however contractually constrained to make
> > the data publicly available. They just dont have to make it easy.
> >
> > So I am looking for a way around my problems. I want to script the exchange. So
> > I just enter the relevant info in the script (e.g. the years, the
> > stations, the destination, etc) and the whole thing goes on automatically
> > from my end. As far as the web server is concerned somebody is sitting
> > at the my end. But in reality a program is handling the transaction.
> >
> > Can anybody make a suggestion on the right tool?
> >
> > Thanks
> >
> > Phil

Perl's LWP library.