[lug] large file management

Wed Apr 30 11:31:33 MDT 2014

Something else to consider is how much data you regularly move at one
time and your network bandwidth. It takes me about 12 minutes to move
8GB video files using all of the bandwidth of a 100Mbit hardwired
network.

On 04/30/2014 10:28 AM, Quentin Hartman wrote:
> For my purposes the amount of space was much more important than speed, and
> I had failure cases covered by RAID. I've been running this setup quite
> happily on a 10 year old dual Xeon (P4 based, 2x2 cores) box with 8GB of
> RAM. The storage is software RAID1 750GB HDDs. It's also doing a bunch of
> miscellaneous other stuff.
> 
> 
> 
> On Wed, Apr 30, 2014 at 10:18 AM, Bethany Hobson <blhobson at gmail.com> wrote:
> 
>> Thanks for the heads up on the gotchas, Quentin.  Care to elaborate on
>> your hardware for the remote server?  Wondering if we should invest in new
>> SSD on a new Mac/PC or new SSD on linux box created from an older machine
>> or all new hardware?  Or am I off the mark/too naive from the start?
>>
>> Thanks again,
>>
>> Bethany
>>
>> On Apr 29, 2014, at 11:24 AM, Quentin Hartman <qhartman at gmail.com> wrote:
>>
>> Sure thing. I'm using Owncloud 6 ( http://owncloud.org/six/) on a remote
>> server, and I usually sync via one of the clients (
>> http://owncloud.org/install/) though the web interface is quite useful as
>> well. There wasn't much to setting it up. I pretty much just installed the
>> packages and setup an apache vhost as directed.
>>
>> The two gotchas I ran into were:
>> - Make sure the location it's storing data has plenty of space. By default
>> it just dumps files into a sub-directory of your webroot, which probably
>> isn't what you want.
>> - When upgrading from 4.x to 5.x the upgrade process trashed pretty much
>> everything. I had it backed up so it wasn't a big deal to install 5.x fresh
>> and restore data, but it was a bit of a hassle. Upgrading from 5.x to 6.x
>> did not do that. I imagine if you don't install via packages that won't
>> happen, but it is something to be aware of. The packages are community
>> maintained and they aren't always as rigorously tested as one might hope.
>>
>> QH
>>
>>
>> On Tue, Apr 29, 2014 at 11:11 AM, Bethany Hobson <blhobson at gmail.com>wrote:
>>
>>> I followed this thread with great interest.  I'm an amateur genealogist
>>> with photo, video, audio, document files in various formats in various
>>> places - websites, online storage, personal computers, etc.  I would like
>>> to engage with my teenagers (still computer-ish, but not always gaming) in
>>> a project this Summer to pull all these data together and organize it.
>>>  Right now, it's pretty much useless to anyone, but me - not good.
>>>
>>> ownCloud looks intriguing and maybe most accessible for my (our) skill
>>> level(s).  The idea of being "in control" of my data locally until I figure
>>> out what to put out there sounds nice.  Being able to tie in and organize
>>> our family music scene would be sweet, too.  Teens may appreciate this
>>> project more.
>>>
>>> Please chime in!  Ideas and suggestions are much appreciated.  Quentin
>>> Hartmen, if you have the time and inclination to elaborate on your private
>>> ownCloud installation, I thank you.
>>>
>>> --
>>> Bethany Hobson
>>>
>>> On Mar 30, 2014, at 9:22 PM, Davide Del Vento <davide.del.vento at gmail.com>
>>> wrote:
>>>
>>>>>> Shortlist of stuff which I will look/consider more:
>>>>>> - Unison
>>>>>> - boar
>>>>>> - OpenKM
>>>>>> - subversion
>>>>>> - git annex
>>>>>
>>>>> Definitely look at git annex.  Your use case is a lot like one Joey
>>>>> considered when he started writing it.  There's also a web front end
>>> (git
>>>>> annex assistant?) to make it easy for "regular" people to use it.
>>>>
>>>> Yes, I'm definitely looking at it. I'm finding it much more complex
>>>> than I thought at first (e.g. have to unlock files before changing,
>>>> having to manually add them to git-annex but not git itself when
>>>> modified before committing) on top of the already complicated git
>>>> workflow (it's no mystery that I like hg better -- by the way I
>>>> checked mercurial-largefiles and that is not what I want). But I am
>>>> not ruling it out, at least not yet.
>>>>
>>>> There is a fundamental thing that I'd like to do with git-annex and I
>>>> can't figure out how: while having only some repos available (possibly
>>>> only one) deciding that some files aren't worth keeping anywhere and
>>>> asking for them to be deleted from all the repos the next time I sync.
>>>> With rsync, I just write a "note to self" in the repo I'm editing,
>>>> which says something like "use --delete the next time you rsync". This
>>>> is unnerving, though, because when I do this I have to be careful that
>>>> I'm not deleting some other file just for space purposes, which is
>>>> something I also like to do (and which git-annex does with the --drop
>>>> option). Basically I'm looking for a suboption of --drop that would
>>>> say --on-all-remotes and can't find it. You may say "but this is not
>>>> how version control is supposed to work, you don't want to completely
>>>> delete something everywhere", and you'd be right. But this is not
>>>> "normal version control", these are pictures which I initially dumped
>>>> from the camera, synched among repos, and then realizing they were
>>>> crap and are not worth keeping. That's my workflow, I seldom have time
>>>> to look at the pictures before synching and therefore I would like an
>>>> easy (but clear and unmistakable) way to delete them from everywhere
>>>> (even when "everywhere" isn't available -- of course the actual
>>>> deletion will happen at the next synch, not "immediately", but I can
>>>> consider them gone "mentally").
>>>>
>>>> Not sure if I was clear enough... I'm already sleeping :-)
>>>>
>>>> Thanks for any suggestions,
>>>> Dav
>>>> _______________________________________________
>>>> Web Page:  http://lug.boulder.co.us
>>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>>> Join us on IRC: irc.hackingsociety.org port=6667
>>> channel=#hackingsociety
>>>
>>> _______________________________________________
>>> Web Page:  http://lug.boulder.co.us
>>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>>> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>>>
>>
>> _______________________________________________
>> Web Page:  http://lug.boulder.co.us
>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>>
>>
>>
>> _______________________________________________
>> Web Page:  http://lug.boulder.co.us
>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>>
> 
> 
> 
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
> 