Re: Are there kernel testing suites out there? We need them.

Miles Lane (miles@amazon.com)
Sun, 04 Jul 1999 11:50:54 -0700


Rik van Riel wrote:
>
> On Wed, 30 Jun 1999, Miles Lane wrote:
>
> > One of the things Micro$oft does is stress test Windows NT after
> > every build.
>
> Looking at the quality of Windows NT should be enough to establish
> the usefulness of such a test...

I really must disagree with you on this. Granted, there are
bugs in Windows NT. Many on these bugs are related to the
legacy support of the 16-bit "DOS/Win3.1 compatibility"
subsystem, plus the complexity and fragility of
migrating what was formerly subsystem code into the
microkernel. At the inception of NT, the main subsystem
that hand-tuned assembly code that ran in ring 0 was the
Networking Subsystem (please forgive me if my details are
a bit off, it's been years since I was an NT network
subsystem tester). My understanding is that more and more
NT subsystem code has been allowed to bypass the subsystem
architecture and been integrated into the microkernel.
This has been done to decrease the size of the OS in working
memory, to increase performance and decrease load time.
This is done at the cost of portability (there's more assembly
code) and kernel stability (if you screw up in the
microkernel, you're system is hosed).

Anyhow, I can say from firsthand experience that Micro$oft's
testing systems flush out a lot of bugs. The number of bugs
that are seem by the public are a vanishing fraction of the
ones Micro$oft has fixed or decided to leave in because of
launch date requirements and bug severity/priority. Testing
is a really good thing. So is easily reproducible
regression testing (testing stuff that was broken in the past).

I'm not sure how much testing methodology gets discussed in
this forum. There are many forms of testing:

API "white box"
Application "black box"
Stress
Boundary condition
Usability
Production "burn-in"

My impression is that a lot of the Linux kernel testing is
performed on an ad hoc basis by Linux kernel developers.
Of course, early adopters of development kernels find many
problems, but from what I've seen of this list, many folks
who should know how to generate useful OOPS to report, don't.

Anyhow, using the instability of NT as an excuse to not
develop systems to automate and simplify testing is just silly.
Testing is the right thing to do. The question is what test
systems would be helpful and feasible in the context of the
OSS distributed development model.

I would argue that developing a test script repository that
held tests for the various subsystems (video, networking,
disk, and so on) sorted by CPU, kernel revision, distribution
hardware, and so on, would be useful.

The primary benefit of such a system would be that less
knowledgable or skilled development kernel users could help
find bugs by stressing/testing their system in many ways.

I understand that Linux gets a lot of use, and so many bugs
get found. But systematic methods of finding new bugs is a
good thing.

> Stresstesting a Linux kernel on a known-good piece of hardware
> will only show up the most obvious bugs and not the typical
> race conditions and marginal-hardware cases that bother us most.
>
> Rik -- Open Source: you deserve to be in control of your data.
> +-------------------------------------------------------------------+
> | Le Reseau netwerksystemen BV: http://www.reseau.nl/ |
> | Linux Memory Management site: http://www.linux.eu.org/Linux-MM/ |
> | Nederlandse Linux documentatie: http://www.nl.linux.org/ |
> +-------------------------------------------------------------------+

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/