[lug] large file management

[Davide Del Vento]

Thanks everybody for their suggestions. Here are the ones I looked at
so far and my final shortlist.

> Unison isn't actively supported.
Yes, but unison-users mailing list is low-traffic but active, so
decided to give it a try.

> Would DRBD help?
Too much cumbersome for my needs.

> Subversion
More than its slowness the thing I don't like is the fact that each
working copy is twice the size of its actual content. But maybe that's
not big deal, compared to the other pros you mentioned, including that
I used it for so long that I can probably use it effectively while
sleeping.

> Picasa
LOL! For the records, I used it in the past and it created more
problems than it solved

> Bittorrent Sync and AeroFS
They both seem promising, but if my understanding is right they are
proprietary and therefore subjected to "sudden disappear", so I won't
even consider them. I need something that works and is guaranteed to
keep working for years, before I invest my precious time in learning
it, setting it up and teaching it to other family members....

> Sparkleshare
On their front page they explicitly mention that is not good for
"Storing your photo or music collection". I know nothing about this
project and I don't understand why they wrote so, but I removed it
from the shortlist for this reason.

Shortlist of stuff which I will look/consider more:
- Unison
- boar
- OpenKM
- subversion
- git annex

I'll keep the list updated on my investigation. Thanks again and feel
free to send around other suggestions!

Cheers,
Davide


On Mon, Mar 17, 2014 at 3:24 PM, Davide Del Vento
<davide.del.vento at gmail.com> wrote:
> I have a large number of fairly large files (spoiler alert: the
> digital pictures of a lifetime). The collection is growing, mostly but
> not only by append. I have three copies onsite, and would like to add
> a forth copy offsite. Two of the copies are actively edited
> (occasionally, but the frequency doesn't matter), and often appended.
> The third (and forth) copy is just for backup.
>
> Managing such a collection of files is becoming a nightmare. Consider
> some scenarios:
>
> Some files are added to Copy 1 (my laptop) taken from the SD card of
> the camera in a hurry. No synch is done with the other copies at this
> time, because I have to run out of the door to take pictures of the
> kids.
>
> Some files are deleted in Copy 2 (e.g. by my wife) because they were
> out of focus, blurred, not worth keeping. No synch is done with the
> other copies at this time, either.
>
> Some (old) files are deleted in Copy 2 (e.g. by myself) because we run
> out of space in the laptop.
>
> Then, one is in the process of rsynch'ing the 3 copies and my rsynch
> workflow (discussed on this list some time ago, IIRC) breaks. Not that
> it was particularly good anyway, having to know "off channel" what
> happened to which copy before I could fire a reasonable command....
> Since I'm reading my swag at the BLUG (Cliff Stoll's "Silicon Snake
> Oil", but that's the subject for another post), I'm tempted to say
> "computers are crap", let my mother ship me my old film cameras and
> lenses from the early 80s (or maybe buying them here for cheap) and
> stop at that. Heck those cameras worked even without batteries!
>
> Instead I say: there must be a better way. I mean, we solve this very
> same problem (for text files) zillions of times a day, don't we? But
> for large binaries, I can't just throw [git|hg|svn] at it and be
> happy. As mentioned in the scenario, at least on one machine space is
> tight. Yes, space is getting cheaper and the like, but: a) I don't
> have any money to buy a disk for that laptop and b) there is no reason
> to keep a copy of some pictures from a trip in 2003 on it anyway. And
> I am not even sure [git|hg|svn] is the right tool for the job.
>
> So googling around I found http://code.google.com/p/boar/ and
> http://www.cis.upenn.edu/~bcpierce/unison/index.html and I ask: has
> any of you tried either? Or anything other piece of software worth
> trying for this purpose?
>
> Cheers,
> Davide