> Suppose: We have a numbercrunching application running mostly on CPU1.
> Cache-1 is "hot", cache-2 is "cold". CPU1 is currently executing some
> triviality, CPU2 is idle. And suppose we have 1M cache on the CPUs.
>
> So suppose we schedule the number crunching application onto CPU2.
>
> DIMMs are 64 bits right? Do they run in something like 6-1-1-1 timing?
> So we get 32 bytes into the cache in 100 ns (rounding a bit). This is
> a 100ns stall, and wether the application "heats up the cache" all at
> once after starting, or wether it happens over a (much) longer period
> of time doesn't matter: in the end it will have waited
>
> 1M bytes cache size / 32 bytes/cache-line-fetch * 100ns = 3.3ms.
>
> for the cache to heat up.
you are assuming the worst case. The above is true only if there is
minimal 'cache-flux' (cachelines moving in and out of cache), and the
whole cache is related to the application. Typical number crunching
applications have either a considerable cache-flux, or have a rather small
cache-footprint. In either case the 3.3ms worst-case 'cache-miss cost' is
(often much) smaller.
nevertheless we do not want processes to ping-pong between CPUs, and the
Linux scheduler very actively tries to avoid it. A CPU-switch every 2
seconds is not good, but even in the worst case costs only 3.3ms/2000 ms =
0.16% slowdown. (also note that this is caused by xosview, a 'pure'
number-crunching system will show no such pingpongs. And note that only
2-CPU systems are affected, on a 4-ways box this effect is much less
probable.) But we do not want to have even this amount of ping-pong, i've
sent some patches to Mathias, and if the problem is resolved i'll send a
summary patch to linux-kernel as well. (there are some other small
scheduling corner-cases we miss on SMP, see the recent PMesa thread)
-- mingo
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/