[lug] Cluster File Systems

Sat Aug 4 05:31:38 MDT 2018

Ops. The link: http://moo.nac.uci.edu/~hjm/fhgfs_vs_gluster.html

On Fri, Aug 3, 2018 at 7:16 PM, Davide Del Vento <davide.del.vento at gmail.com
> wrote:

> Since you and others mentioned a few things I did not know about, while
> trying to learn more about them I've found this which seems to be quite
> informative regarding BeeGFS (aka fhGFS) as compared to gluster and a
> little bit to Lustre.
>
> On Fri, Aug 3, 2018 at 2:55 PM, Steve Webb <bigwebb at gmail.com> wrote:
>
>> ... and swift: https://www.swiftstack.com/product/open-source/openst
>> ack-swift
>>
>> On Fri, Aug 3, 2018 at 2:54 PM Steve Webb <bigwebb at gmail.com> wrote:
>>
>>> I think that I would need to know a few things before I could answer
>>> what a good cluster filesystem would be for the task.  You already covered
>>> the standard answers: NFS, GlusterFS & Lustre.  Do you need support for
>>> dozens, hundreds or thousands of clients?  Is there a NAS available for
>>> shared storage or is it truly distributed storage (spanning multiple
>>> hosts)?  What is the app that's running that needs access to > 10G files?
>>> video?  Large databases?  Weather Model?
>>>
>>> NFS would be my go-to if it handles the workload.  Pretty stable, runs
>>> on tons of platforms.  Could be a SPOF though in production environments.
>>> GlusterFS would be a good solution if the storage is truely distributed
>>> and you don't want to do much cluster management.
>>> Lustre is the go-to for many clients - good for large parallel computing
>>> platforms.
>>> GFS is awesome, supported by RedHat/CentOS, but requires a SAN or
>>> central storage.
>>> Ceph is awesome, but requires a bunch of machines for infrastructure (is
>>> mainly an object storage system but has a filesystem driver for it)
>>>
>>> If this is in AWS, you could look into EFS (AWS's version of NFSv4) or
>>> an S3-based FUSE wrapper.
>>>
>>> - Steve Webb
>>>
>>> On Fri, Aug 3, 2018 at 10:29 AM Bear Giles <bgiles at coyotesong.com>
>>> wrote:
>>>
>>>> 1. Do you need full filesystem support or is it enough to be able to
>>>> read and write files programmatically? This could be a relative minor
>>>> change or a huge headache, of course.
>>>>
>>>> 2. Do you only need to read and write files sequentially, or do you
>>>> need to be able to move within the file, update the file in-place, etc.?
>>>>
>>>> If both conditions are met then hadoop (HDFS) could be well-supported
>>>> solution. However it's definitely not a solution if you need to be able to
>>>> treat it like a regular filesystem or can't tweak code.
>>>>
>>>> On Fri, Aug 3, 2018 at 6:46 AM, Davide Del Vento <
>>>> davide.del.vento at gmail.com> wrote:
>>>>
>>>>> You are very welcome.
>>>>> I suspect you can make it work with CentOS and perhaps even with
>>>>> Fedora or other distros, but if you have easier routes....
>>>>> The only other option I know is IBM's spectrum scale, which is
>>>>> proprietary and so expensive you do not even want to know the price....
>>>>> Keep us posted!
>>>>> Davide
>>>>>
>>>>> On Fri, Aug 3, 2018 at 1:14 AM, Lee Woodworth <blug-mail at duboulder.com
>>>>> > wrote:
>>>>>
>>>>>> On 08/02/2018 05:33 PM, Davide Del Vento wrote:
>>>>>>
>>>>>>> I'd suggest you take a look at http://lustre.org/
>>>>>>> I don't know how it is from an administrative perspective, but it
>>>>>>> certainly
>>>>>>> can do what you ask. It might be overkill, but that's your call.
>>>>>>>
>>>>>>
>>>>>> Thanks for the suggestion. It appears to require kernel patches and
>>>>>> using
>>>>>> RHEL for the server (Lustre Support Matrix only has RHEL for the
>>>>>> server).
>>>>>>
>>>>>>   http://lustre.org/getting-started-with-lustre/:
>>>>>>     Metadata and Object Storage Server require the Lustre patched
>>>>>> Linux kernel,
>>>>>>     Lustre modules, Lustre utilities and e2fsprogs installed. The
>>>>>> clients
>>>>>>     require the Lustre client modules, client utilities and,
>>>>>> optionally,
>>>>>>     the Lustre patched kernel.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Thu, Aug 2, 2018 at 5:01 PM, Lee Woodworth <
>>>>>>> blug-mail at duboulder.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Can anyone recommend an open-source cluster file system
>>>>>>>> that can handle doing lots of reads/writes to a remote
>>>>>>>> file, especially >10GB files? Would like to have a
>>>>>>>> mix of host architectures and not have to setup a full
>>>>>>>> cluster + management.
>>>>>>>>
>>>>>>>> A few years ago I used glusterfs with the native fs driver
>>>>>>>> in the kernel. I would get hard kernel-level lockups
>>>>>>>> in the above scenario (client reboot was the only way
>>>>>>>> to recover).
>>>>>>>>
>>>>>>>> Even nfsv4 servers locked up, though I haven't tried doing
>>>>>>>> that in awhile.
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Web Page:  http://lug.boulder.co.us
>>>>>>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>>>>>>> Join us on IRC: irc.hackingsociety.org port=6667
>>>>>>>> channel=#hackingsociety
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Web Page:  http://lug.boulder.co.us
>>>>>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>>>>>> Join us on IRC: irc.hackingsociety.org port=6667
>>>>>>> channel=#hackingsociety
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Web Page:  http://lug.boulder.co.us
>>>>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>>>>> Join us on IRC: irc.hackingsociety.org port=6667
>>>>>> channel=#hackingsociety
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Web Page:  http://lug.boulder.co.us
>>>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>>>> Join us on IRC: irc.hackingsociety.org port=6667
>>>>> channel=#hackingsociety
>>>>>
>>>>
>>>> _______________________________________________
>>>> Web Page:  http://lug.boulder.co.us
>>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>>> Join us on IRC: irc.hackingsociety.org port=6667
>>>> channel=#hackingsociety
>>>
>>>
>> _______________________________________________
>> Web Page:  http://lug.boulder.co.us
>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20180804/e88f3125/attachment.html>