Re: [RFC] scripts: Include postprocessing script for memory allocation tracing

From: Janani Ravichandran
Date: Thu Oct 20 2016 - 19:10:46 EST


Michal,

> On Oct 18, 2016, at 8:13 AM, Michal Hocko <mhocko@xxxxxxxxxx> wrote:
>
>>
>
> yes, function_graph tracer will give you _some_ information but it will
> not have the context you are looking for, right? See the following
> example
>
> ------------------------------------------
> 0) x-www-b-22756 => x-termi-4083
> ------------------------------------------
>
> 0) | __alloc_pages_nodemask() {
> 0) | /* mm_page_alloc: page=ffffea000411b380 pfn=1066702 order=0 migratetype=0 gfp_flags=GFP_KERNEL */
> 0) 3.328 us | }
> 3) | __alloc_pages_nodemask() {
> 3) | /* mm_page_alloc: page=ffffea0008f1f6c0 pfn=2344923 order=0 migratetype=0 gfp_flags=GFP_KERNEL */
> 3) 1.011 us | }
> 0) | __alloc_pages_nodemask() {
> 0) | /* mm_page_alloc: page=ffffea000411b380 pfn=1066702 order=0 migratetype=0 gfp_flags=GFP_KERNEL */
> 0) 0.587 us | }
> 3) | __alloc_pages_nodemask() {
> 3) | /* mm_page_alloc: page=ffffea0008f1f6c0 pfn=2344923 order=0 migratetype=0 gfp_flags=GFP_KERNEL */
> 3) 1.125 us | }
>
> How do I know which process has performed those allocations? I know that
> CPU0 should be running x-termi-4083 but what is running on other CPUs?
>
> Let me explain my usecase I am very interested in. Say I that a usespace
> application is not performing well. I would like to see some statistics
> about memory allocations performed for that app - are there few outliers
> or the allocation stalls increase gradually? Where do we spend time during
> that allocation? Reclaim LRU pages? Compaction or the slab shrinkers?
>
> To answer those questions I need to track particular events (alocation,
> reclaim, compaction) to the process and know how long each step
> took. Maybe we can reconstruct something from the above output but it is
> a major PITA. If we either hard start/stop pairs for each step (which
> we already do have for reclaim, compaction AFAIR) then this is an easy
> scripting. Another option would be to have only a single tracepoint for
> each step with a timing information.
>
> See my point?

Yes, if we want to know what processes are running on what CPUs,
echo funcgraph-proc > trace_options in the tracing directory should give us
what we want.

The bash script which is part of this patch does this kind of setup for you.
As a result, the output you get is something like what you see here:

https://github.com/Jananiravichandran/Analyzing-tracepoints/blob/master/no_tp_no_threshold.txt

Does this answer your question? Let me know if otherwise.

Janani.

> --
> Michal Hocko
> SUSE Labs