Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
From: Lin Ming
Date: Thu Sep 04 2008 - 08:42:19 EST
On Thu, 2008-09-04 at 14:26 +0200, Peter Zijlstra wrote:
> On Thu, 2008-09-04 at 20:12 +0800, Lin Ming wrote:
> > > > > Thats bizarre... that just indicates the better clock, which should give
> > > > > better (read fairer) scheduling hurts your workload.
> > > > >
> > > > > Is there anything I can run to see if we can fix the scheduler perhaps?
> > > >
> > > > I observed schedstats of sysbench, there's more
> > > > "nr_failed_migrations_hot"
> > > >
> > > > 2.6.27-rc4: se.nr_failed_migrations_hot 11
> > > > 2.6.27-rc5: se.nr_failed_migrations_hot 95
> > > >
> > > > task migration failed because of task_hot, the system is un-balanced?
> > >
> > > Ah, that makes sense, a more accurate clock could indeed make more tasks
> > > hot.
> > >
> > > Can you try fiddling with: /proc/sys/kernel/sched_migration_cost ?
> >
> > sched_migration_cost regression
> > ---------------------- -------------
> > 50000 ~6%
> > 0 ~8%
> > 500000 (default) ~10%
> > 5000000 ~14%
> > -1 ~19%
>
> at 50000 (~6%), is the predominant difference in schedstats still the
> nr_failed_migrations_hot?
Yes, it's strange that nr_failed_migrations_hot (cost=50000) is larger
than (cost=500000)
sched_migration_cost = 50000
----------------------------------------------------------
se.exec_start : 3475158.689868
se.vruntime : 935215.526989
se.sum_exec_runtime : 14358.431942
se.avg_overlap : 0.000000
se.wait_start : 0.000000
se.sleep_start : 3475158.689868
se.block_start : 0.000000
se.sleep_max : 37.910779
se.block_max : 0.033796
se.exec_max : 0.121028
se.slice_max : 0.000000
se.wait_max : 6.007209
se.wait_sum : 25029.649588
se.wait_count : 1954948
sched_info.bkl_count : 0
se.nr_migrations : 2521
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 147
se.nr_failed_migrations_hot : 123
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 1236
se.nr_wakeups : 1952827
se.nr_wakeups_sync : 1944785
se.nr_wakeups_migrate : 2479
se.nr_wakeups_local : 1930122
se.nr_wakeups_remote : 22705
se.nr_wakeups_affine : 13252
se.nr_wakeups_affine_attempts : 825950
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.007355
avg_per_cpu : 5.695530
nr_switches : 1951994
nr_voluntary_switches : 1949963
nr_involuntary_switches : 2031
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 81
sched_migration_cost = 500000
----------------------------------------------------------
se.exec_start : 3719706.806593
se.vruntime : 1049301.798712
se.sum_exec_runtime : 14636.764384
se.avg_overlap : 0.000121
se.wait_start : 0.000000
se.sleep_start : 3719706.806593
se.block_start : 0.000000
se.sleep_max : 214.989106
se.block_max : 0.036240
se.exec_max : 0.147829
se.slice_max : 0.000000
se.wait_max : 6.475864
se.wait_sum : 24932.530233
se.wait_count : 1922627
sched_info.bkl_count : 0
se.nr_migrations : 2314
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 71
se.nr_failed_migrations_hot : 76
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 1735
se.nr_wakeups : 1920337
se.nr_wakeups_sync : 1908767
se.nr_wakeups_migrate : 2272
se.nr_wakeups_local : 1898194
se.nr_wakeups_remote : 22143
se.nr_wakeups_affine : 1218
se.nr_wakeups_affine_attempts : 820118
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.007624
avg_per_cpu : 6.325308
nr_switches : 1919596
nr_voluntary_switches : 1917396
nr_involuntary_switches : 2200
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 69
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/