[lug] Fwd: Simple counter?

[Paul Condon]

On 05/24/2013 10:16 PM, Anthony Foiani wrote:
> Bear Giles<bgiles at coyotesong.com>  writes:
>
>> It's the concurrency that will get you.
> Oh, don't I know it.  :)
>
> (Current $dayjob is an embedded(ish) linux system, 32+ threads, and
> currently debugging a collision...)
>
>> Even locks might not work if the underlying filesystem doesn't
>> properly support them.
> Advisory locks (flock(2)) should be kernel-based, so not dependent on
> FS.  So long as everyone accesses through the advisory locks,
> everything is good.  (But this quickly becomes a battle of
> specifications and assumptions.)
>
>> That's why I was thinking along the lines of a named pipe - that way
>> you have a single process on the other end of the pipe.
> Interesting... although the persistence is an issue.  Have to make
> sure you save it just once.  Also, no mechanism for "current value" if
> all you have is a pipe.
>
>> The database sequence is another possibility. I wouldn't use mysql
>> though - there are lightweight JVM databases (e.g., H2) and you
>> could also investigate SQLite.
> That seems like sledgehammer-as-flyswatter, but it does serve to
> properly separate out persistence of data from increment and/or
> access.
>
>> Re-reading the same value will always be problematic outside of a
>> database transaction though - how do you know some other process
>> asn't gotten a newer value in the interrim? The only way to be
>> really sure is to cache the information in your own process somehow
>> and again that's adding a lot of weight to the code.
> Right.
>
> My assumption was that the end user would use this wisely, which
> includes dealing with concurrency sanely.
>
> And it's almost necessary that it's the end user that is responsible
> for this; there's no way to predict all possible use cases, and the
> end result of trying to do so is a huge, heavyweight, probably
> difficult-to-deploy mess.
>
> To draw a comparison, the pthread standard doesn't make things safe
> for all users; instead, it outlines rules and assumptions that one can
> make when writing concurrent / threaded programs.  The programmer is
> still ultimately responsible for doing it right.
>
> My concept of "counter" is intended to be such a tool, for use by
> shell scripts.  I'll admit that my interpretation of "seed" and "next
> value" are both fairly loose and more designed for sanity checking
> than rigorous use.
>
> I suspect that the most advanced creatures of this sort will be found
> in cluster filesystems, to get a unique identifier across thousands of
> nodes.  Probably not sequential, but that's one more tradeoff, right?
>
> t.
>
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
In Debian there is a package 'lockfile-progs'. Its package description is:
<quote>
This package includes several programs to safely lock and unlock files 
and mailboxes from the command line. These include:

lockfile-create
lockfile-remove
lockfile-touchlock
mail-lock
mail-unlock
mail-touchlock

These programs use liblockfile to perform the file locking and 
unlocking, so they are guaranteed compatible with Debian's file locking 
policies.

</quote>

I take this as proof that the issues involved in a "Simple Counter" have 
been considered by others before us, and that the full solution will be 
far from "simple" ;-)

I suggest a system based on using email and a single mailing list server 
listening for requests for the 'next' number. When it gets a request, it 
runs the mutex and emails the name of the requester, the number he got, 
and a unique message ID to all participants. Joining and leaving the 
would be handled according to the protocol of the mailing list package 
being used. The message ID is a fairly simple way to limit cheating in 
whatever activities these sequential numbers are put to.

My belief is, further, that liblockfile is in the public domain and can 
be ported to any distribution to which it is not already ported, and 
that this list could be used as a test bed by reserving the subject line 
"test of simple counter" to identify incoming emails that are intended 
for the test. Or maybe set up an entirely separate list, but using the 
existing hardware.