[lug] Testing

[Jeffrey Haemer]

Davide,
Progression testing, even using big subsets of even bigger coverage suites
can go wrong in some interesting ways.

(1) Ethics.

Ignoring bugs in a PostScript interpreter might annoy your VP of Marketing.
 Ignoring bugs your users report is stupid.  Ignoring bugs in life-support
equipment or a missile-guidance system kills people.  You can't always set
aside a subset.

(2) Too much redundancy.

Consider /bin/true.  What might we reasonably test?

- Does it crash?
- Does it run really slowly or hang?
- Does it return the wrong exit status?
- Does "/bin/true --help" give the wrong usage message?
- Does "/bin/true --version" give the wrong information?
- If we set LANG=it_IT.UTF8, does it ignore us?  Or spit out messages in
Vietnamese?

Perhaps there are other modes of failure, but we'll just test for these.

If we have a 5000-test, test suite for /bin/true, and sequester half, the
other half is still going to cover everything.  In a sense, the tests
set-aside tell us what we want to know: fixing the bugs uncovered by the
other set have fixed all the bugs.  Still, going to the work of setting
aside a subset doesn't get us much.

(3) Too much orthogonality.

If you buy a comprehensive test suite for /bin/true that has exactly six
tests -- one for each of the things listed above -- then it could be that
these behaviors are so orthogonal that fixing any subset doesn't affect the
others. If the rest of the tests are the only things you'll ever run, the
set-aside tests tell you nothing, except that there's never progress.
  Setting aside a subset only ignores errors.

(4) Too little relevance.

If your tests are "looking under the lamppost for your keys because the
light's better there," watching their progress in a subset just isn't
interesting.  Nothing ever happens.

Imagine, for example, you have a huge randomly-generated test suite but your
code starts out completely bug-free and you never have to fix anything.
 (It's fun to pretend.)

Ergo, the progression tests need to find bugs you can ignore, need to find
bugs you will still find other ways, need to find bugs that aren't in the
initial set of tests that you do fix, and need to actually find some bugs.

-- 
Click to Call Me Now! --
http://seejeffrun.blogspot.com/2009/09/call-me-now.html

Jeffrey Haemer <jeffrey.haemer at gmail.com>
720-837-8908 [cell],  @goyishekop [twitter]
http://www.youtube.com/user/goyishekop [vlog]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20091021/41ed722c/attachment.html>