[lug] MEAS Data file format

[Wayde Allen]

On Wed, 16 Aug 2000, David wrote:

> First let me write that I hope that this endeavour is highly
> successful for you.  The negative stuff that follows is meant to be
> devil's advocate rantings.

That was the whole point in posting the idea here.  I appreciate you
taking the time to comment.

> I suspect that, in the long run, this effort will be less than you
> would like it to be.  People have been trying to standardise (computer
> stuff) for 50 years, and they still are doing it EVEN AT the national
> standards organisation!
> 
> The problem you alluded to in your first posting:
>    ... 
>    have tried to make this simple so that it has a chance of being used by
>    someone who would normally just create a row and column set of data.
>    ...
> The value of peoples' "give a !beep!" coefficient varies enormously
> and I doubt that you will be able to achieve standards adherence to
> the degree required to make this really work.
>
> I think that you have done the easy bit, although what you have done
> will work nicely for up to, say, four people.  The hard work you have
> not mentioned.  Some other people may be actually destructive, but,
> ignoring them, also you will have misteaks or well-intentioned
> misunderstandings, and, of course, the all-important "gotta get it
> done now!" and lack of willingness to correct errors "just to make it
> fit that system".  All of these things will cause problems with
> smooth-running, so then you will need a system administrator as things
> get bigger.  Whether or not mis-match problems actually MATTER I do
> not know; but if they do not matter why are you doing it?

Yep, I'm afraid I tend to agree with your assessment of the situation.

> Here are a few questions that come to mind immediately, that is, as
> fast as I can type them.
> 
> ** Why are some keywords terminated with a colon and some with a
> space,

That would be due to a typo on my part.  They should all end with a colon.

> ** What does a blank line mean?

Blank lines should be ignored.  I thought I put that in there, but ...

> ** What happens when a user uses commas instead of spaces to separate
> STANDARDS, as you did?

That is something that still needs to be worked out.  I just recently
added this tag to the file and haven't exactly figured out all of the
details.

> ** Why have two kinds of comment? # and #COMMENT

Lines beginning with # are file comments and intended to describe things
internal to the file itself.  Lines beginning with #COMMENT: are tags
describing comments made by the system operator.  These are measurement
specific things that we might want to keep track of and pass on to the
researchers doing the data analysis.

> * You might have problems with all-numeric dates if you try
>   international data exchange.  Is the Fourth of July represented as
>   4-7-yy or 7-4-yy?  I choose this particular date intentionally, if
>   you do not get it read the sentence very carefully.  New Year's day
>   is easy because it is the same in each system - or is that a
>   trip-wire?

I agree that we should use the ISO standard date format.

> As I wrote at the beginning, I wish you luck; but, then, I believe in
> formats, conventions, no literals in code, comments in code, software
> engineering, formalisms, etc.  Those who are anti-intellectual deny
> the need for such things

If we even get some semblance of similarity between some of our data
files I think we've accomplished something.

> Here are my two suggestions.
> 
> * Write a grammar for your data file format (you do not need to admit
>   to anyone that you did it).  A grammar will aid you ENORMOUSLY as
>   you proceed later.  You will need to parse things, you will need to
>   handle errors, you will need to define semantics, you will need to
>   make enhancements without messing-up what has gone before.  If you
>   believe that a grammar is not necessary, perhaps because what you
>   are suggesting is too simple for that, then see my comment above
>   about anti-intellectualism in computing.

In a sense, I thought that was what I was trying to do.  I think we agree
on this point.

> * I have no idea how practical this is in your setting.  But you might
>   consider an alternative (actually an addition) to defining a
>   file-format: provide a library of (object-oriented?) functions that
>   enable people to write your standard files without even knowing that
>   they are doing it.  If the calls are dead-easy, you might find that
>   people use the functions because it is even simpler than writing
>   code that creates a flat file.  This approach clearly implies an
>   adminstrator (i.e., the code maintainer - for *all* the languages
>   that might be used); but, equally, enables fearsome quality control
>   to satisfy your other objectives.

I've been working in this direction too.  The snippet of code I sent to 
the Python mailing list is an example.  The big problem is that most of
the people in my group don't use object oriented languages. Rocky Mountain
Basic is the typical instrument control language.  I think the closest
thing would be to create a set of common subprograms.

Anyway, since I will be leaving NIST to work for ITS effective August
27th, most of this will become moot.

- Wayde
  (wallen at boulder.nist.gov)