> Maybe simpler than calculating the variance, why not just track the
> smallest and largest results for each test, and see if these change
> from version to version? How about a "realistic" test, that is,
> something similar to what really gets run a lot, for instance,
> compiling the kernel with -pipe and other things; the variance would
> be high, but it would be interesting to see (and improvements to
> things like paging might show up dramatically).
THe results are there for you to just tract the top/bottom if you wish,
but I think the byte bench marks are much better than kernel compile time,
which changes linearly with each release due to there being more code and
ifdefs to process, and is very heavily disk space and filesystem state
dependant. The byte test do purely cpu intensive, io intensive and mixed
shell script tests (the shell scripts do things like sort | uniq | awk, and
the test runs upto 8 of the scripts in parallel). Just ignore anything in the
+/- 5% range.
Anyway on the 1.3.11 -> 1.3.12 change the pipe based context switch almost
_doubled_ on both the P5/60 and 486/33 I tested it on. Nothing else
significant change. Bit surprising that the shell scripts didn't do better,
but probably shows they are IO/CPU limited rather than context switch limited.
Jon