[lug] Managing work queue for scientific processing

[Will]

I've had good luck with CA's Autosys which meets most of
your requirements but fails the first one as it is no where near cheap.

On Tue, May 3, 2011 at 5:50 PM, Vince Dean <vincedean at frii.com> wrote:

> Dear BLUG folks:
>
> I'm looking for tools to run the data processing for
> an atmospheric research instrument: perhaps a "batch
> queue manager", "resource manager" or "scientific
> workflow system."
>
> One recommendation has been:
> - the Maui scheduler, which runs on the
> - Torque resource manager, a fork of the
> - Portable Batch Manager (PBM)
>
> Do you have thoughts or personal experience with this
> or any competing systems?
>
> We have four Linux boxes--dual quad-core Xeon systems
> running Scientific Linux.  The team is small: a few
> developers and scientists.
>
> We use cron and Perl scripts to launch the jobs today.
> This works for us, but the CPU utilization is not as
> high as I would like and the system is hard to understand
> and manage. The system is big enough that is has grown
> complex, but this is not what I would call high-performance
> computing.
>
> There ought to be a better way.  This is a well-studied
> problem and I expect there are some standard solutions.
>
> Desirable:
> - cheap/free
> - relatively simple to install and experiment with; we
>  cannot afford to make this a large project
> - scheduling based on priority and resource availability:
>  - CPUs
>  - memory
> - ability to monitor and manage the queue
> - distribute jobs to multiple compute servers
> - command-line interface for ease of integration
> - modular/light-weight  enough to be adapted to our existing
>  structure
>
> I'd be grateful for any comments or suggestions.
>
> Thanks,
> Vince Dean
>
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20110503/cf267b77/attachment.html>