[lug] large file management

Wed Apr 30 10:28:53 MDT 2014

For my purposes the amount of space was much more important than speed, and
I had failure cases covered by RAID. I've been running this setup quite
happily on a 10 year old dual Xeon (P4 based, 2x2 cores) box with 8GB of
RAM. The storage is software RAID1 750GB HDDs. It's also doing a bunch of
miscellaneous other stuff.

On Wed, Apr 30, 2014 at 10:18 AM, Bethany Hobson <blhobson at gmail.com> wrote:

> Thanks for the heads up on the gotchas, Quentin.  Care to elaborate on
> your hardware for the remote server?  Wondering if we should invest in new
> SSD on a new Mac/PC or new SSD on linux box created from an older machine
> or all new hardware?  Or am I off the mark/too naive from the start?
>
> Thanks again,
>
> Bethany
>
> On Apr 29, 2014, at 11:24 AM, Quentin Hartman <qhartman at gmail.com> wrote:
>
> Sure thing. I'm using Owncloud 6 ( http://owncloud.org/six/) on a remote
> server, and I usually sync via one of the clients (
> http://owncloud.org/install/) though the web interface is quite useful as
> well. There wasn't much to setting it up. I pretty much just installed the
> packages and setup an apache vhost as directed.
>
> The two gotchas I ran into were:
> - Make sure the location it's storing data has plenty of space. By default
> it just dumps files into a sub-directory of your webroot, which probably
> isn't what you want.
> - When upgrading from 4.x to 5.x the upgrade process trashed pretty much
> everything. I had it backed up so it wasn't a big deal to install 5.x fresh
> and restore data, but it was a bit of a hassle. Upgrading from 5.x to 6.x
> did not do that. I imagine if you don't install via packages that won't
> happen, but it is something to be aware of. The packages are community
> maintained and they aren't always as rigorously tested as one might hope.
>
> QH
>
>
> On Tue, Apr 29, 2014 at 11:11 AM, Bethany Hobson <blhobson at gmail.com>wrote:
>
>> I followed this thread with great interest.  I'm an amateur genealogist
>> with photo, video, audio, document files in various formats in various
>> places - websites, online storage, personal computers, etc.  I would like
>> to engage with my teenagers (still computer-ish, but not always gaming) in
>> a project this Summer to pull all these data together and organize it.
>>  Right now, it's pretty much useless to anyone, but me - not good.
>>
>> ownCloud looks intriguing and maybe most accessible for my (our) skill
>> level(s).  The idea of being "in control" of my data locally until I figure
>> out what to put out there sounds nice.  Being able to tie in and organize
>> our family music scene would be sweet, too.  Teens may appreciate this
>> project more.
>>
>> Please chime in!  Ideas and suggestions are much appreciated.  Quentin
>> Hartmen, if you have the time and inclination to elaborate on your private
>> ownCloud installation, I thank you.
>>
>> --
>> Bethany Hobson
>>
>> On Mar 30, 2014, at 9:22 PM, Davide Del Vento <davide.del.vento at gmail.com>
>> wrote:
>>
>> >>> Shortlist of stuff which I will look/consider more:
>> >>> - Unison
>> >>> - boar
>> >>> - OpenKM
>> >>> - subversion
>> >>> - git annex
>> >>
>> >> Definitely look at git annex.  Your use case is a lot like one Joey
>> >> considered when he started writing it.  There's also a web front end
>> (git
>> >> annex assistant?) to make it easy for "regular" people to use it.
>> >
>> > Yes, I'm definitely looking at it. I'm finding it much more complex
>> > than I thought at first (e.g. have to unlock files before changing,
>> > having to manually add them to git-annex but not git itself when
>> > modified before committing) on top of the already complicated git
>> > workflow (it's no mystery that I like hg better -- by the way I
>> > checked mercurial-largefiles and that is not what I want). But I am
>> > not ruling it out, at least not yet.
>> >
>> > There is a fundamental thing that I'd like to do with git-annex and I
>> > can't figure out how: while having only some repos available (possibly
>> > only one) deciding that some files aren't worth keeping anywhere and
>> > asking for them to be deleted from all the repos the next time I sync.
>> > With rsync, I just write a "note to self" in the repo I'm editing,
>> > which says something like "use --delete the next time you rsync". This
>> > is unnerving, though, because when I do this I have to be careful that
>> > I'm not deleting some other file just for space purposes, which is
>> > something I also like to do (and which git-annex does with the --drop
>> > option). Basically I'm looking for a suboption of --drop that would
>> > say --on-all-remotes and can't find it. You may say "but this is not
>> > how version control is supposed to work, you don't want to completely
>> > delete something everywhere", and you'd be right. But this is not
>> > "normal version control", these are pictures which I initially dumped
>> > from the camera, synched among repos, and then realizing they were
>> > crap and are not worth keeping. That's my workflow, I seldom have time
>> > to look at the pictures before synching and therefore I would like an
>> > easy (but clear and unmistakable) way to delete them from everywhere
>> > (even when "everywhere" isn't available -- of course the actual
>> > deletion will happen at the next synch, not "immediately", but I can
>> > consider them gone "mentally").
>> >
>> > Not sure if I was clear enough... I'm already sleeping :-)
>> >
>> > Thanks for any suggestions,
>> > Dav
>> > _______________________________________________
>> > Web Page:  http://lug.boulder.co.us
>> > Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>> > Join us on IRC: irc.hackingsociety.org port=6667
>> channel=#hackingsociety
>>
>> _______________________________________________
>> Web Page:  http://lug.boulder.co.us
>> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>>
>
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>
>
>
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20140430/5db2c1d0/attachment.html>