[lug] Clustering and GFS

[D. Stimits]

Jeff Schroeder wrote:
> Alan wrote:
> 
> 
>>PVS is very fast, very cheap, but not robust with respect to crashes.
>> It depends on what your needs are.  For scientific clustering in
>>non-critical environments, it's the most popular choice.
> 
> 
> Thanks to everyone who sent links and suggestions.  I like the look of 
> PVFS, but run into two problems: (1) the client insists on GFS, and (2) 
> the shared disk has to support a *sustained* data transfer rate in 
> excess of 100 MB/s... ouch.

That kind of sustained rate means the use of RAID. If your files are 
large, the XFS filesystem overlaid with something other system would be 
a good choice, e.g., OpenAFS I believe is a layer on top of the 
underlying system, similar to NFS being a network layer on top of 
anything else. I know you are not allowed to use OpenAFS, I am curious 
though what Alan Robertson thinks of it? In any case, before you can 
figure out how you can best obtain 100 MB/s, you need to know what kind 
of files are used: lots of small files, or a few very large files. And I 
assume that GFS is not a direct filesystem itself, but a layer on top of 
other filesystems to distribute them? No matter what you choose, you 
probably will have to have large SCSI drives (higher cluster density) at 
high rpm, e.g., U160 drives at 10k rpm, 36 GB, in RAID 0 or RAID 10, 
using two U160 channels for each machine if needed. For a cluster, that 
can be expensive. I have not played with network storage, but a single 
network storage device with oodles (think of this as something far 
beyond "kilo, mega, and tera", and OodleByte) of bandwidth and doing 
nothing but serving files and bandwidth over gigabit fiber. Frankly, I 
cannot imagine a cluster of machines with sustained disk throughput of 
100 MB/s without lots of cash.

D. Stimits, stimits AT attbi.com

PS: Has the client stated why the GFS is so important?