[lug] Testing

[Davide Del Vento]

> Ergo, the progression tests need to find bugs you can ignore, need to find
> bugs you will still find other ways, need to find bugs that aren't in the
> initial set of tests that you do fix, and need to actually find some bugs.

Interesting. I was actually thinking at a different ways of generating
"progression test suites".  It looks like your "progression tests" are
in fact what I call "regression test used for progression purposes",
whereas I was thinking to "unit tests used for progression purposes".

Take weather forecast: I can't crawl the web looking for a day and the
subsequent one's weather. And even if I could (maybe I can get access
to some databases for that) I can't use them as a good test for my
weather forecast program: the program might be right for the wrong
reason, or it might be wrong without having a bug (say, for that
particularly "difficult" day a higher resolution is required, and
that's a manual configuration parameter, not a software bug). I could
use synthetic data for which an analytic solution is possible (and
that's actually done in some test suites), but it is too "expensive"
just for progression and it limits the things that are tested (e.g. my
topology must be analytic as well, which might be true somewhere in
the great plains, but surely it's not when you are close to any real
mountain).

Bye,
;Dav