[lug] Cluster File Systems

[Bear Giles]

1. Do you need full filesystem support or is it enough to be able to read
and write files programmatically? This could be a relative minor change or
a huge headache, of course.

2. Do you only need to read and write files sequentially, or do you need to
be able to move within the file, update the file in-place, etc.?

If both conditions are met then hadoop (HDFS) could be well-supported
solution. However it's definitely not a solution if you need to be able to
treat it like a regular filesystem or can't tweak code.

On Fri, Aug 3, 2018 at 6:46 AM, Davide Del Vento <davide.del.vento at gmail.com
> wrote:

> You are very welcome.
> I suspect you can make it work with CentOS and perhaps even with Fedora or
> other distros, but if you have easier routes....
> The only other option I know is IBM's spectrum scale, which is proprietary
> and so expensive you do not even want to know the price....
> Keep us posted!
> Davide
>
> On Fri, Aug 3, 2018 at 1:14 AM, Lee Woodworth <blug-mail at duboulder.com>
> wrote:
>
>> On 08/02/2018 05:33 PM, Davide Del Vento wrote:
>>
>>> I'd suggest you take a look at http://lustre.org/
>>> I don't know how it is from an administrative perspective, but it
>>> certainly
>>> can do what you ask. It might be overkill, but that's your call.
>>>
>>
>> Thanks for the suggestion. It appears to require kernel patches and using
>> RHEL for the server (Lustre Support Matrix only has RHEL for the server).
>>
>>   http://lustre.org/getting-started-with-lustre/:
>>     Metadata and Object Storage Server require the Lustre patched Linux
>> kernel,
>>     Lustre modules, Lustre utilities and e2fsprogs installed. The clients
>>     require the Lustre client modules, client utilities and, optionally,
>>     the Lustre patched kernel.
>>
>>
>>
>>> On Thu, Aug 2, 2018 at 5:01 PM, Lee Woodworth <blug-mail at duboulder.com>
>>> wrote:
>>>
>>> Can anyone recommend an open-source cluster file system
>>>> that can handle doing lots of reads/writes to a remote
>>>> file, especially >10GB files? Would like to have a
>>>> mix of host architectures and not have to setup a full
>>>> cluster + management.
>>>>
>>>> A few years ago I used glusterfs with the native fs driver
>>>> in the kernel. I would get hard kernel-level lockups
>>>> in the above scenario (client reboot was the only way
>>>> to recover).
>>>>
>>>> Even nfsv4 servers locked up, though I haven't tried doing
>>>> that in awhile.
>>>>
>>>>
>>>> _______________________________________________
>>>> Web Page:  http://lug.boulder.co.us
>>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>>> Join us on IRC: irc.hackingsociety.org port=6667
>>>> channel=#hackingsociety
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Web Page:  http://lug.boulder.co.us
>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>>>
>>>
>> _______________________________________________
>> Web Page:  http://lug.boulder.co.us
>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>>
>
>
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20180803/bb0368fc/attachment.html>