[lug] Managing work queue for scientific processing

[Glenn Murray]

Hi Vince,

If you can keep your requirements simple, then Maui is probably the way to
go, if you can stand it.

There are several big projects such as

Gobus: http://www.globus.org/
Taverna: http://www.taverna.org.uk/

but these are not "relatively simple".

My two cents:

My experience (I've built several systems, mostly for NREL) is that bringing
some software engineering to your existing system may be the way to go.  The
systems I've built, in Python and Java, have often gotten mileage out of
using an enterprise service bus (e.g., Mule) to gain
platform/language/protocol independence and to use as "glue" to bring legacy
systems together.

One problem in this area is that it turns out there are so many kinds of
scientific workflows that one-size-fits-all solutions are necessarily
complex or diverse (e.g., Kepler).  What you have may be (mostly) what you
need.

Cheers,
Glenn

On Tue, May 3, 2011 at 5:50 PM, Vince Dean <vincedean at frii.com> wrote:

> Dear BLUG folks:
>
> I'm looking for tools to run the data processing for
> an atmospheric research instrument: perhaps a "batch
> queue manager", "resource manager" or "scientific
> workflow system."
>
> One recommendation has been:
> - the Maui scheduler, which runs on the
> - Torque resource manager, a fork of the
> - Portable Batch Manager (PBM)
>
> Do you have thoughts or personal experience with this
> or any competing systems?
>
> We have four Linux boxes--dual quad-core Xeon systems
> running Scientific Linux.  The team is small: a few
> developers and scientists.
>
> We use cron and Perl scripts to launch the jobs today.
> This works for us, but the CPU utilization is not as
> high as I would like and the system is hard to understand
> and manage. The system is big enough that is has grown
> complex, but this is not what I would call high-performance
> computing.
>
> There ought to be a better way.  This is a well-studied
> problem and I expect there are some standard solutions.
>
> Desirable:
> - cheap/free
> - relatively simple to install and experiment with; we
>  cannot afford to make this a large project
> - scheduling based on priority and resource availability:
>  - CPUs
>  - memory
> - ability to monitor and manage the queue
> - distribute jobs to multiple compute servers
> - command-line interface for ease of integration
> - modular/light-weight  enough to be adapted to our existing
>  structure
>
> I'd be grateful for any comments or suggestions.
>
> Thanks,
> Vince Dean
>
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20110504/cb24ff6b/attachment.html>