Re: Benchmarks - 1.3.11

Zygo Blaxell (zblaxell@ezmail.com)
Mon, 24 Jul 1995 15:04:06 -0400 (EDT)


> > Duh... What I do is, *always*, start the benchmark run (4 to 5 hours)
> > after a fresh reboot. I login from the first multiterminal, start the
> > daemon, check that it started OK, log out, and, generally, go to sleep.
> >
> > My machine is not on the net (except when I dial in). From MY point of
> > view, I am being as impartial as possible.
>
> Sounds to me like you are making an good effort to have simular
> testing conditions every time. =)
>
> > Each test is run 20 times (10 for the most time-consuming). If I find the
> > time, I will add the calculation of the standard deviation.
>
> I would be interested in seeing that. Its all fine and well to say
> that task switching times or something has increased + or - 0.82% over
> the previous version, however when there haven't been any changes that
> could make a difference in such a thing, then you have to figre that
> you are actually detecting something else.
>
> Christopher

It's also possible that we are seeing changes that affect his machine's
cache+memory+CPU implementation. Depending on the type of memory
cacheing algorithms, rearranging a few bytes of critical kernel code
here and there might make a 0.1% difference in overall performance.
Such issues become quite obvious when you start comparing inlined vs.
conventional functions, rolled vs. unrolled loops, etc. Large OS vendors
know this--they have tools that do nothing but rearrange code so that it
"fits" on the hardware better.

To do these tests properly would require several different types of
machines, different CPU's (remember that Pentiums have
counterintuitively low BogoMips ratings), bus architectures, memory,
disks...Then there's the possible effects of the compiler version,
libc, and other components of the software run-time environment to
consider.

By benchmarking someone's unknown hardware, we can hack up GCC and
friends to rearrange the kernel binary to run maybe 2% faster on it...

Since I generally favor correctness over performance, I say we fix the
bugs first. ;-)