Re: [RFC GIT PULL] perf/trace/lock optimization/scalabilityimprovements

[Jens Axboe]

On Wed, Feb 03 2010, Frederic Weisbecker wrote:
> On Wed, Feb 03, 2010 at 11:25:41AM +0100, Jens Axboe wrote:
> > On Wed, Feb 03 2010, Frederic Weisbecker wrote:
> > > Hi,
> > > 
> > > There are many things that happen in this patchset, treating
> > > different problems:
> > > 
> > > - remove most of the string copy overhead in fast path
> > > - open the way for lock class oriented profiling (as
> > >   opposite to lock instance profiling. Both can be useful
> > >   in different ways).
> > > - remove the buffers muliplexing (less contention)
> > > - event injection support
> > > - remove violent lock events recursion (only 2 among 3, the remaining
> > >   one is detailed below).
> > > 
> > > Some differences, by running:
> > > 	perf lock record perf sched pipe -l 100000
> > > 
> > > Before the patchset:
> > > 
> > > 	Total time: 91.015 [sec]
> > > 
> > > 	     910.157300 usecs/op
> > > 		   1098 ops/sec
> > > 
> > > After this patchset applied:
> > > 
> > > 	Total time: 43.706 [sec]
> > > 
> > > 	     437.062080 usecs/op
> > > 		   2288 ops/sec
> > 
> > This does a lot better here, even if it isn't exactly stellar
> > performance. It generates a LOT of data:
> > 
> > root@nehalem:/dev/shm # time perf lock rec -fg ls
> > perf.data  perf.data.old
> > [ perf record: Woken up 0 times to write data ]
> > [ perf record: Captured and wrote 137.224 MB perf.data (~5995421
> > samples) ]
> 
> 
> 
> Doh, 137 MB for a single ls :)

Yeah, it's pretty crazy. It varies a lot too, I didn't post the other
run which was ~170MB.

> That said we don't have yet support for callchains in perf lock,
> and callchains can fill the buffer quickly, especially on lock
> events. You can drop the -g option for now.

OK

> > So while this is orders of magnitude better than the previous patchset,
> > it's still not anywhere near lean. But I expect you know that, just
> > consider this a 'I tested it and this is what happened' report :-)
> 
> 
> Ok, thanks a lot, the fact you can test on a 64 threads box is critically
> helpful.

My pleasure, I'd love to have a fast and functional perf lockstat. If my
testing helps getting there, consider me signed up :-)

> I also wonder what happens after this patch applied:
> 
> diff --git a/kernel/perf_event.c b/kernel/perf_event.c
> index 98fd360..254b3d4 100644
> --- a/kernel/perf_event.c
> +++ b/kernel/perf_event.c
> @@ -3094,7 +3094,8 @@ static u32 perf_event_tid(struct perf_event *event, struct task_struct *p)
>         if (event->parent)
>                 event = event->parent;
>  
> -       return task_pid_nr_ns(p, event->ns);
> +       return p->pid;
>  }
> 
> In my box it has increased the speed from 2x this patchset.

Cool, I'll give that a spin in the morning, the box is off atm.

> I wonder if the tool becomes usable for you with that.
> Otherwise, it means we have other things to fix, and
> the result of:
> 
> 	perf record -g -f perf lock record sleep 6
> 	perf report
> 
> would be very nice to have.

I'll package that up for you and put it somewhere.

-- 
Jens Axboe

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/