RE: Performance changes between 2.6.13 and 2.6.23

[Sanders, Rob M.]

-----Original Message-----
From: madrabbit@xxxxxxxxx on behalf of Ray Lee
Sent: Tue 3/25/2008 12:41 PM
To: Sanders, Rob M.
Cc: linux-kernel@xxxxxxxxxxxxxxx; bart.vanassche@xxxxxxxxx
Subject: Re: Performance changes between 2.6.13 and 2.6.23
 
On Tue, Mar 25, 2008 at 5:34 AM, Sanders, Rob M. <sanders-rob@xxxxxxx> wrote:
> Hello all,
>   I've been lurking on the digest for some time (don't want to receive full lkml traffic at work) and saw the
>  posts about Wine performance regressions in 2.6.24.  Some of what I saw there, particularly Andi Kleen's
>  responses, mirror something that I see on my box at home.  I had emailed Andi directly since I only read the
>  digest, and I'm posting this here at his suggestion.  Please CC: rms@xxxxxxx with any replies, as I'm only
>  getting the lkml digest.
>   I'm running on a dual 2GHZ G5 Powermac w/2GB ram.  I recently upgraded from YDL4.0.91 (2.6.13 kernel) to
>  YDL6 (2.6.23 kernel) and noticed that the overall performance of the box seems more sluggish after the upgrade.
>  Of more particular concern, the main application that I build has seen a 4-5X slowdown in performance.  Under
>  YDL4.0.91 I could process roughly 1e8 data points in ~2 seconds, and under YDL6.0 I now process 1e8 data points

So, two processors, and multiple processes passing data back and
forth. Key point seems to be:

>  in ~8 seconds.  Total CPU loading (from top) is about 5%, under both systems.  The application spawns multiple
>  processes and uses semaphores and shared memory to move data between the processes.  When I use 'vmstat 3' the
>  single biggest difference I see between YDL4.0.91 and YDL6 is that the YDL4.0.91 system is the YDL4.0.91 system is
>  context switching about 7000 per interval, whereas the YDl6 system is context switching about 1200 times per interval.

Many more context switches per second, so a lot less work is getting
done each time.

>   I've been talking somewhat with Owen Stampflee (works for the distro maker, TSS) and have rebuilt the kernel on
>  the 2.6 box removing some of the things that I don't need on my box (cell support, etc).  There did seem (subjectively)
>  to be a slight improvement after that.  When I get some more time to play at home I'm going to take the 2.6.13 .config
>  I use and build a 2.6.23 kernal using 'make oldconfig' and try to figure if it is a config issues.
>   I realize that going from 2.6.13 to 2.6.23 is a *huge* change, and that the problems may not be tied to the kernel
>  but to other things, but are there any other suggestions folks have for finding the cause of the performance slowdow

A lot has changed between 2.6.23 and current mainline as well.
Particularly in the scheduler, which I suspect is the issue for your
test. If possible, could you try a 2.6.25-rc-latest kernel, both
before and after an "echo 5 > /proc/sys/kernel/sched_features" and see
if that makes any difference?


=====

Sorry to disappear for a few days, I just hadn't had time to do anything at home.  Finally had some time to 
run some test, and at this point I'm thinking that the problem is not with the kernel.  Not quite sure where to look 
next, but I've done the following tests:

YDL4.0.91 with 2.6.13 kernel - normal speed
YDL6.0    with 2.6.23 kenrel - 4-5x slowdown
YDL4.1    with 2.6.15 kernel - 4-5x slowdown
YDL4.0.91 with 2.6.15 kernel - *normal speed* - kernel config pulled from YDL4.1 install
YDL4.0.91 with 2.6.23 kernel - failed boot

I'm going to try and test a few other distros (Fedora 8/9, possibly OpenSuse) to see if they have similar 
issues.  If anyone has other suggestions (including websites to read) feel free to drop me a private line 
(at rms@xxxxxxx).  Thanks....

Rob

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/