> > > confidence.
> > >
> > > > But, let me suggest the Posix conformance suites as a good first past
> > > > test for stupid brokeness.
> > >
> > > This was something I was considering, and looking at the redistribution
> > > license (http://www.itl.nist.gov/div897/ctg/softagre.htm), it looks like
> > > the NIST code could be incorporated into a GPLed test suite because it's
> > > produced by the government and therefore not protected by copyright.
> >
> > A sick thought crossed my mind ... What if we doing something
> > similar to what the cryptography folks are doing. Have people run the
> > stress test when they want, how ever long they want, and have it
> > communicate with a main server someplace which will keep track of these
> > things ...
> > What do you people think?
> >
> > -Seth
>
> That it is unnecesarily complex, and you can only report success, anyway.
> If it crashes, you can't expect that machine to report it :-).
> [You need human beings in this process...]
We could use an ACK type of system. If we have an identical test
suite for every system, we can have the suite send something to the effect
of "Starting test 1" and then "Test 1 succeeded", etc. If we get a
"Starting test x", but no "Test x succeeded", we know exactly where things
went wrong. Also, diagnostic messages that get spit up every so often
could contain important information which will get transmitted.
Also, what is a daemon or some such is placed in the rc.M file
when this processes starts. We have the system run and run and run until
it crashes, logging all the way. When the system is rebooted, this file is
sent to the developers ...
Yes, we will need the people in the process to catch things that
we didn't think about ("My monitor exploded on test 4", for example, could
not be reported by the system, same with "My hard disk now talks in
tongues", for obvious reasons). But this will a) ensure that incomplete
information isn't given (i.e., my computer? It's white.) and b)
information is sent for glitches that the user might not see if he didn't
go into the syslog.
-Seth