Re: Rescheduling interrupts

[Ingo Molnar]

* S.ÃaÄlar Onur <caglar@xxxxxxxxxxxxx> wrote:

> Top causes for wakeups:
>   59,9% (238,4)      <kernel IPI> : Rescheduling interrupts
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   14,7% ( 58,6)         amarokapp : schedule_timeout (process_timeout)

hm, would be nice to figure out what causes these IPIs. Could you stick 
something like this into arch/x86/kernel/smp_32.c's 
smp_send_reschedule() function [this is the function that generates the 
IPI]:

static void native_smp_send_reschedule(int cpu)
{
        WARN_ON(cpu_is_offline(cpu));
        send_IPI_mask(cpumask_of_cpu(cpu), RESCHEDULE_VECTOR);
	if (panic_timeout > 0) {
		panic_timeout--;
		printk("IPI from task %s:%d on CPU#%d:\n", 
			current->comm, current->pid, cpu);
		dump_stack();
	}
}

NOTE: if you run an SMP kernel then first remove these two lines from 
kernel/printk.c:

        if (!oops_in_progress && waitqueue_active(&log_wait))
                wake_up_interruptible(&log_wait);

otherwise you'll get lockups. (the IPI is sent while holding the 
runqueue lock, so the printks will lock up)

then wait for the bad condition to occur on your system and generate a 
stream of ~10 backtraces, via:

	echo 10 > /proc/sys/kernel/panic

you should be getting 10 immediate backtraces - please send them to us. 
The backtraces should show the place that generates the wakeups. [turn 
on CONFIG_FRAME_POINTERS=y to get high quality backtraces.]
 
If you do _not_ get 10 immediate backtraces, then something in the 
system is generating such IPIs outside of the scheduler's control. That 
would suggest some other sort of borkage.

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/