[lug] Testing

[Jeffrey Haemer]

Davide,
I can't imagine most folks reading the original thread are interested in
this stuff, so I'm going to fork it so they can mute this one. :-)

On Tue, Oct 20, 2009 at 11:08 PM, Davide Del Vento <
davide.del.vento at gmail.com> wrote:

> > Okay, here's a text version of the article.
> http://alumnus.caltech.edu/~copeland/work/
> >  This is before the magazine's
> > copy editors patched it up, so you'll find typos, and there will be no
> fancy
> > formatting or pictures with circles and arrows on the back of each one,
> but
> > the content's there.
>
> Cool. But IMO it wouldn't work, for other two reasons besides your
>
> > In our experience, the hardest thing about
> > maintaining a progression-test suite is
> > convincing your fellow developers not
> >  to fix reproduceable bugs for which you
> > have perfectly good test cases.
>
> My reasons are:
>
> 1) where are you getting those tests? somebody wrote them, and the
> writing served a purpose:
> 1a) reproducing in the suite a bug report from the user (you CANNOT
> avoid fixing them)
> 1b) specifying in the suite a requirement or a design choice,
> especially if you are doing test first TDD (these tests CANNOT be
> broken by definition)
>
> Now, to create this progression test suite, you are writing a "large
> bunch of tests" (but how to decide what to test, and what is the
> "correct" outcome?) That's a lot of effort, just to make a
> measurement! I believe that it doesn't worth the effort.
>
> 2) Even if you get the progression suite, how do you guarantee that's
> statistically relevant? You should have a large number of (unfixed!)
> bugs, and they must be independent. Now, if you have a large number of
> known bug, it makes a little sense not fixing them, otherwise you'll
> just know that your code is cripple. But the independent is even
> trickier: when people try to design tests, they'll likely design them
> in "sections" which will be closely related each other, and they will
> be broken (or not) in not-independent groups.
>
> In conclusion, I believe that this concept is not likely to be useful
> in practice, even though is theoretically very interesting. It could
> be interesting in practice too, if you figure out a way of generating
> "random tests" automatically.. They must not be perfect (so developers
> would not tempted to fix them, because they might be wrong), but they
> must be unbiased and statistically independent. If you do, you'll have
> produced a really great result!
>
> Have a good night
> ;Dav
> PS: this series was very nice, what a pity that you stopped writing it!
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: lug.boulder.co.us port=6667 channel=#hackingsociety
>



-- 
Click to Call Me Now! --
http://seejeffrun.blogspot.com/2009/09/call-me-now.html

Jeffrey Haemer <jeffrey.haemer at gmail.com>
720-837-8908 [cell],  @goyishekop [twitter]
http://www.youtube.com/user/goyishekop [vlog]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20091021/62a67431/attachment.html>