[lug] Cluster File Systems

Fri Aug 3 19:16:06 MDT 2018

Since you and others mentioned a few things I did not know about, while
trying to learn more about them I've found this which seems to be quite
informative regarding BeeGFS (aka fhGFS) as compared to gluster and a
little bit to Lustre.

On Fri, Aug 3, 2018 at 2:55 PM, Steve Webb <bigwebb at gmail.com> wrote:

> ... and swift: https://www.swiftstack.com/product/open-source/
> openstack-swift
>
> On Fri, Aug 3, 2018 at 2:54 PM Steve Webb <bigwebb at gmail.com> wrote:
>
>> I think that I would need to know a few things before I could answer what
>> a good cluster filesystem would be for the task.  You already covered the
>> standard answers: NFS, GlusterFS & Lustre.  Do you need support for dozens,
>> hundreds or thousands of clients?  Is there a NAS available for shared
>> storage or is it truly distributed storage (spanning multiple hosts)?  What
>> is the app that's running that needs access to > 10G files?  video?  Large
>> databases?  Weather Model?
>>
>> NFS would be my go-to if it handles the workload.  Pretty stable, runs on
>> tons of platforms.  Could be a SPOF though in production environments.
>> GlusterFS would be a good solution if the storage is truely distributed
>> and you don't want to do much cluster management.
>> Lustre is the go-to for many clients - good for large parallel computing
>> platforms.
>> GFS is awesome, supported by RedHat/CentOS, but requires a SAN or central
>> storage.
>> Ceph is awesome, but requires a bunch of machines for infrastructure (is
>> mainly an object storage system but has a filesystem driver for it)
>>
>> If this is in AWS, you could look into EFS (AWS's version of NFSv4) or an
>> S3-based FUSE wrapper.
>>
>> - Steve Webb
>>
>> On Fri, Aug 3, 2018 at 10:29 AM Bear Giles <bgiles at coyotesong.com> wrote:
>>
>>> 1. Do you need full filesystem support or is it enough to be able to
>>> read and write files programmatically? This could be a relative minor
>>> change or a huge headache, of course.
>>>
>>> 2. Do you only need to read and write files sequentially, or do you need
>>> to be able to move within the file, update the file in-place, etc.?
>>>
>>> If both conditions are met then hadoop (HDFS) could be well-supported
>>> solution. However it's definitely not a solution if you need to be able to
>>> treat it like a regular filesystem or can't tweak code.
>>>
>>> On Fri, Aug 3, 2018 at 6:46 AM, Davide Del Vento <
>>> davide.del.vento at gmail.com> wrote:
>>>
>>>> You are very welcome.
>>>> I suspect you can make it work with CentOS and perhaps even with Fedora
>>>> or other distros, but if you have easier routes....
>>>> The only other option I know is IBM's spectrum scale, which is
>>>> proprietary and so expensive you do not even want to know the price....
>>>> Keep us posted!
>>>> Davide
>>>>
>>>> On Fri, Aug 3, 2018 at 1:14 AM, Lee Woodworth <blug-mail at duboulder.com>
>>>> wrote:
>>>>
>>>>> On 08/02/2018 05:33 PM, Davide Del Vento wrote:
>>>>>
>>>>>> I'd suggest you take a look at http://lustre.org/
>>>>>> I don't know how it is from an administrative perspective, but it
>>>>>> certainly
>>>>>> can do what you ask. It might be overkill, but that's your call.
>>>>>>
>>>>>
>>>>> Thanks for the suggestion. It appears to require kernel patches and
>>>>> using
>>>>> RHEL for the server (Lustre Support Matrix only has RHEL for the
>>>>> server).
>>>>>
>>>>>   http://lustre.org/getting-started-with-lustre/:
>>>>>     Metadata and Object Storage Server require the Lustre patched
>>>>> Linux kernel,
>>>>>     Lustre modules, Lustre utilities and e2fsprogs installed. The
>>>>> clients
>>>>>     require the Lustre client modules, client utilities and,
>>>>> optionally,
>>>>>     the Lustre patched kernel.
>>>>>
>>>>>
>>>>>
>>>>>> On Thu, Aug 2, 2018 at 5:01 PM, Lee Woodworth <
>>>>>> blug-mail at duboulder.com>
>>>>>> wrote:
>>>>>>
>>>>>> Can anyone recommend an open-source cluster file system
>>>>>>> that can handle doing lots of reads/writes to a remote
>>>>>>> file, especially >10GB files? Would like to have a
>>>>>>> mix of host architectures and not have to setup a full
>>>>>>> cluster + management.
>>>>>>>
>>>>>>> A few years ago I used glusterfs with the native fs driver
>>>>>>> in the kernel. I would get hard kernel-level lockups
>>>>>>> in the above scenario (client reboot was the only way
>>>>>>> to recover).
>>>>>>>
>>>>>>> Even nfsv4 servers locked up, though I haven't tried doing
>>>>>>> that in awhile.
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Web Page:  http://lug.boulder.co.us
>>>>>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>>>>>> Join us on IRC: irc.hackingsociety.org port=6667
>>>>>>> channel=#hackingsociety
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Web Page:  http://lug.boulder.co.us
>>>>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>>>>> Join us on IRC: irc.hackingsociety.org port=6667
>>>>>> channel=#hackingsociety
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Web Page:  http://lug.boulder.co.us
>>>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>>>> Join us on IRC: irc.hackingsociety.org port=6667
>>>>> channel=#hackingsociety
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Web Page:  http://lug.boulder.co.us
>>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>>> Join us on IRC: irc.hackingsociety.org port=6667
>>>> channel=#hackingsociety
>>>>
>>>
>>> _______________________________________________
>>> Web Page:  http://lug.boulder.co.us
>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>>
>>
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20180803/6f4c8f51/attachment-0001.html>