[lug] Managing work queue for scientific processing

[Moore, Kevin L.]

Maui/Torque are still free and open source. Cluster Resources had done
some value add to Maui and they call it Moab for the scheduler, but
Torque is still the resource manager underneath. Torque has a scheduler,
but it's not as full featured as Maui, so I think people use the two
together as they integrate well. Maui/Torque is one of the more common
schedulers for HPC clusters, and is what we use here on our cluster with
good results. YMMV...Kevin

 

From: lug-bounces at lug.boulder.co.us
[mailto:lug-bounces at lug.boulder.co.us] On Behalf Of Glenn Murray
Sent: Wednesday, May 04, 2011 12:16 PM
To: Boulder (Colorado) Linux Users Group -- General Mailing List
Subject: Re: [lug] Managing work queue for scientific processing

 

Hi Vince,

If you can keep your requirements simple, then Maui is probably the way
to go, if you can stand it.

There are several big projects such as 

Gobus: http://www.globus.org/
Taverna: http://www.taverna.org.uk/

but these are not "relatively simple".  

My two cents:

My experience (I've built several systems, mostly for NREL) is that
bringing some software engineering to your existing system may be the
way to go.  The systems I've built, in Python and Java, have often
gotten mileage out of using an enterprise service bus (e.g., Mule) to
gain platform/language/protocol independence and to use as "glue" to
bring legacy systems together.

One problem in this area is that it turns out there are so many kinds of
scientific workflows that one-size-fits-all solutions are necessarily
complex or diverse (e.g., Kepler).  What you have may be (mostly) what
you need.  

Cheers,
Glenn

On Tue, May 3, 2011 at 5:50 PM, Vince Dean <vincedean at frii.com> wrote:

Dear BLUG folks:

I'm looking for tools to run the data processing for
an atmospheric research instrument: perhaps a "batch
queue manager", "resource manager" or "scientific
workflow system."

One recommendation has been:
- the Maui scheduler, which runs on the
- Torque resource manager, a fork of the
- Portable Batch Manager (PBM)

Do you have thoughts or personal experience with this
or any competing systems?

We have four Linux boxes--dual quad-core Xeon systems
running Scientific Linux.  The team is small: a few
developers and scientists.

We use cron and Perl scripts to launch the jobs today.
This works for us, but the CPU utilization is not as
high as I would like and the system is hard to understand
and manage. The system is big enough that is has grown
complex, but this is not what I would call high-performance
computing.

There ought to be a better way.  This is a well-studied
problem and I expect there are some standard solutions.

Desirable:
- cheap/free
- relatively simple to install and experiment with; we
 cannot afford to make this a large project
- scheduling based on priority and resource availability:
 - CPUs
 - memory
- ability to monitor and manage the queue
- distribute jobs to multiple compute servers
- command-line interface for ease of integration
- modular/light-weight  enough to be adapted to our existing
 structure

I'd be grateful for any comments or suggestions.

Thanks,
Vince Dean

_______________________________________________
Web Page:  http://lug.boulder.co.us
Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety

 




This message and any enclosures are intended only for the addressee.  Please  
notify the sender by email if you are not the intended recipient.  If you are  
not the intended recipient, you may not use, copy, disclose, or distribute this  
message or its contents or enclosures to any other person and any such actions  
may be unlawful.  Ball reserves the right to monitor and review all messages  
and enclosures sent to or from this email address.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20110504/932947c0/attachment.html>