Re: [PATCH 3/3] ftrace: function tracer with irqs disabled

[Steven Rostedt]

On Tue, 4 Nov 2008, Ingo Molnar wrote:

> 
> * Ingo Molnar <mingo@xxxxxxx> wrote:
> 
> > * Steven Rostedt <rostedt@xxxxxxxxxxx> wrote:
> > 
> > > Running hackbench 3 times with the irqs disabled and 3 times with 
> > > the preempt disabled function tracer yielded:
> > > 
> > > tracing type       times            entries recorded
> > > ------------      --------          ----------------
> > > irq disabled      43.393            166433066
> > >                   43.282            166172618
> > >                   43.298            166256704
> > > 
> > > preempt disabled  38.969            159871710
> > >                   38.943            159972935
> > >                   39.325            161056510
> > 
> > your numbers might be correct, but i found that hackbench is not 
> > reliable boot-to-boot - it can easily produce 10% systematic noise 
> > or more. (perhaps depending on how the various socket data 
> > structures happen to be allocated)
> > 
> > the really conclusive way to test this would be to add a hack that 
> > either does preempt disable or irqs disable, depending on a runtime 
> > flag - and then observe how hackbench performance reacts to the 
> > value of that flag.
> 
> ... which is exactly what your patch implements :-)

Yep ;-)

Those numbers were done without any reboots in between. I even tried it
several times, randomly picking to use irqs_disabled and preempt_disabled,
and everytime preempt_disabled was around 39 secs, and irqs disabled was
around 43.

> 
> > note that preempt-disable will also produce less trace entries, 
> > especially in very irq-rich workloads. Hence it will be "faster".
> 
> this point still holds. Do we have any good guess about the 'captured 
> trace events per second' rate in the two cases, are they the same?

If you look at the end of my change log, I printed stats from a patch I 
added that counted the times that ftrace recursed, but did not record.
Those numbers were quite big with preempt_disabled.

>> With irq disabled: 1,150 times the function tracer did not trace due to
>>   recursion.
>> with preempt disabled: 5,117,718 times.

When we used the preempt disabled version, we lost 5 million traces, as 
suppose to the irq disabled which was only 1,150 traces lost.

Considering that we had 166,256,704 traces total, that 5 million is only 
4% lost of traces. Still quite a lot. But again, this is an extreme,
because we are tracing hackbench.

-- Steve

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/