Re: automated test? (was Re: Linux 2.6.17.7)

[Matthias Andree]

On Tue, 25 Jul 2006, Arjan van de Ven wrote:

> well you can do such a thing withing statistical bounds; however... if
> the patch already is in -git (as is -stable policy normally).. it should
> have been found there already...

The sad facts I learned from Debian bug #212762 (not kernel related) that
culminated in CVE-2005-2335 (remote root exploit against older
fetchmail) and from various qmail bugs Guninski discovered:

- a bug need not necessarily be found soon after introduction

- a bug report may not convey the hint "look at this NOW, the shit
  already hit the fan"
  (sorry, I meant to write: look NOW, it's urgent and important)

- an automated test to catch non-trivial mistakes is non-trivial in
  itself, and - what I've seen with another project I was involved with,
  and more often than I found amusing - is that the test itself can be
  buggy causing bogus results.

That doesn't mean I object to automated tests, but "it should have been
found by now" (because the source is open, someone could have tested it,
whatever) just doesn't work.

-- 
Matthias Andree
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/