High context switch rate, ksoftirqd's chewing cpu

[Zlatko Calusic]

Hello Tejun et al.

This week I spent some hours tracking a regression in 3.7 kernel that 
was producing high context switch rate on one of my machines. I 
carefully bisected between 3.6 and 3.7-rc1 and eventually found this 
commit a culprit:

commit e7c2f967445dd2041f0f8e3179cca22bb8bb7f79
Author: Tejun Heo <tj@xxxxxxxxxx>
Date:   Tue Aug 21 13:18:24 2012 -0700

    workqueue: use mod_delayed_work() instead of __cancel + queue

    Now that mod_delayed_work() is safe to call from IRQ handlers,
    __cancel_delayed_work() followed by queue_delayed_work() can be
    replaced with mod_delayed_work().

    Most conversions are straight-forward except for the following.

    * net/core/link_watch.c: linkwatch_schedule_work() was doing a quite
      elaborate dancing around its delayed_work.  Collapse it such that
      linkwatch_work is queued for immediate execution if LW_URGENT and
      existing timer is kept otherwise.

    Signed-off-by: Tejun Heo <tj@xxxxxxxxxx>
    Cc: "David S. Miller" <davem@xxxxxxxxxxxxx>
    Cc: Tomi Valkeinen <tomi.valkeinen@xxxxxx>

Then I carefully reverted chunk by chunk to find out what exact change 
is responsible for the regression. You can find it attached as wq.patch 
(to preserve whitespace). Very simple modification with wildly different 
behavior on only one of my machines, weird. I'm also attaching ctxt/s 
graph that shows the impact nicely. I'll gladly provide any additional 
info that could help you resolve this.

Please Cc: on reply (not subscribed to lkml).

Regards,
--
Zlatko
diff --git a/block/blk-core.c b/block/blk-core.c
index 4b4dbdf..4b8b606 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -319,10 +319,8 @@ EXPORT_SYMBOL(__blk_run_queue);
  */
 void blk_run_queue_async(struct request_queue *q)
 {
-	if (likely(!blk_queue_stopped(q))) {
-		__cancel_delayed_work(&q->delay_work);
-		queue_delayed_work(kblockd_workqueue, &q->delay_work, 0);
-	}
+	if (likely(!blk_queue_stopped(q)))
+		mod_delayed_work(kblockd_workqueue, &q->delay_work, 0);
 }
 EXPORT_SYMBOL(blk_run_queue_async);

Attachment: context-weekly.png
Description: PNG image