[lug] can't make this stuff up, folks... My 2 lines of code, errr, I mean, my 2 cents....

[Davide Del Vento]

> Okay, here's a text version of the article. http://alumnus.caltech.edu/~copeland/work/
>  This is before the magazine's
> copy editors patched it up, so you'll find typos, and there will be no fancy
> formatting or pictures with circles and arrows on the back of each one, but
> the content's there.

Cool. But IMO it wouldn't work, for other two reasons besides your

> In our experience, the hardest thing about
> maintaining a progression-test suite is
> convincing your fellow developers not
>  to fix reproduceable bugs for which you
> have perfectly good test cases.

My reasons are:

1) where are you getting those tests? somebody wrote them, and the
writing served a purpose:
1a) reproducing in the suite a bug report from the user (you CANNOT
avoid fixing them)
1b) specifying in the suite a requirement or a design choice,
especially if you are doing test first TDD (these tests CANNOT be
broken by definition)

Now, to create this progression test suite, you are writing a "large
bunch of tests" (but how to decide what to test, and what is the
"correct" outcome?) That's a lot of effort, just to make a
measurement! I believe that it doesn't worth the effort.

2) Even if you get the progression suite, how do you guarantee that's
statistically relevant? You should have a large number of (unfixed!)
bugs, and they must be independent. Now, if you have a large number of
known bug, it makes a little sense not fixing them, otherwise you'll
just know that your code is cripple. But the independent is even
trickier: when people try to design tests, they'll likely design them
in "sections" which will be closely related each other, and they will
be broken (or not) in not-independent groups.

In conclusion, I believe that this concept is not likely to be useful
in practice, even though is theoretically very interesting. It could
be interesting in practice too, if you figure out a way of generating
"random tests" automatically.. They must not be perfect (so developers
would not tempted to fix them, because they might be wrong), but they
must be unbiased and statistically independent. If you do, you'll have
produced a really great result!

Have a good night
;Dav
PS: this series was very nice, what a pity that you stopped writing it!