[lug] Cluster File Systems

[Steve Webb]

I think that I would need to know a few things before I could answer what a
good cluster filesystem would be for the task.  You already covered the
standard answers: NFS, GlusterFS & Lustre.  Do you need support for dozens,
hundreds or thousands of clients?  Is there a NAS available for shared
storage or is it truly distributed storage (spanning multiple hosts)?  What
is the app that's running that needs access to > 10G files?  video?  Large
databases?  Weather Model?

NFS would be my go-to if it handles the workload.  Pretty stable, runs on
tons of platforms.  Could be a SPOF though in production environments.
GlusterFS would be a good solution if the storage is truely distributed and
you don't want to do much cluster management.
Lustre is the go-to for many clients - good for large parallel computing
platforms.
GFS is awesome, supported by RedHat/CentOS, but requires a SAN or central
storage.
Ceph is awesome, but requires a bunch of machines for infrastructure (is
mainly an object storage system but has a filesystem driver for it)

If this is in AWS, you could look into EFS (AWS's version of NFSv4) or an
S3-based FUSE wrapper.

- Steve Webb

On Fri, Aug 3, 2018 at 10:29 AM Bear Giles <bgiles at coyotesong.com> wrote:

> 1. Do you need full filesystem support or is it enough to be able to read
> and write files programmatically? This could be a relative minor change or
> a huge headache, of course.
>
> 2. Do you only need to read and write files sequentially, or do you need
> to be able to move within the file, update the file in-place, etc.?
>
> If both conditions are met then hadoop (HDFS) could be well-supported
> solution. However it's definitely not a solution if you need to be able to
> treat it like a regular filesystem or can't tweak code.
>
> On Fri, Aug 3, 2018 at 6:46 AM, Davide Del Vento <
> davide.del.vento at gmail.com> wrote:
>
>> You are very welcome.
>> I suspect you can make it work with CentOS and perhaps even with Fedora
>> or other distros, but if you have easier routes....
>> The only other option I know is IBM's spectrum scale, which is
>> proprietary and so expensive you do not even want to know the price....
>> Keep us posted!
>> Davide
>>
>> On Fri, Aug 3, 2018 at 1:14 AM, Lee Woodworth <blug-mail at duboulder.com>
>> wrote:
>>
>>> On 08/02/2018 05:33 PM, Davide Del Vento wrote:
>>>
>>>> I'd suggest you take a look at http://lustre.org/
>>>> I don't know how it is from an administrative perspective, but it
>>>> certainly
>>>> can do what you ask. It might be overkill, but that's your call.
>>>>
>>>
>>> Thanks for the suggestion. It appears to require kernel patches and using
>>> RHEL for the server (Lustre Support Matrix only has RHEL for the server).
>>>
>>>   http://lustre.org/getting-started-with-lustre/:
>>>     Metadata and Object Storage Server require the Lustre patched Linux
>>> kernel,
>>>     Lustre modules, Lustre utilities and e2fsprogs installed. The clients
>>>     require the Lustre client modules, client utilities and, optionally,
>>>     the Lustre patched kernel.
>>>
>>>
>>>
>>>> On Thu, Aug 2, 2018 at 5:01 PM, Lee Woodworth <blug-mail at duboulder.com>
>>>> wrote:
>>>>
>>>> Can anyone recommend an open-source cluster file system
>>>>> that can handle doing lots of reads/writes to a remote
>>>>> file, especially >10GB files? Would like to have a
>>>>> mix of host architectures and not have to setup a full
>>>>> cluster + management.
>>>>>
>>>>> A few years ago I used glusterfs with the native fs driver
>>>>> in the kernel. I would get hard kernel-level lockups
>>>>> in the above scenario (client reboot was the only way
>>>>> to recover).
>>>>>
>>>>> Even nfsv4 servers locked up, though I haven't tried doing
>>>>> that in awhile.
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Web Page:  http://lug.boulder.co.us
>>>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>>>> Join us on IRC: irc.hackingsociety.org port=6667
>>>>> channel=#hackingsociety
>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Web Page:  http://lug.boulder.co.us
>>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>>> Join us on IRC: irc.hackingsociety.org port=6667
>>>> channel=#hackingsociety
>>>>
>>>>
>>> _______________________________________________
>>> Web Page:  http://lug.boulder.co.us
>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>>>
>>
>>
>> _______________________________________________
>> Web Page:  http://lug.boulder.co.us
>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>>
>
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20180803/9c618891/attachment.html>