Re: [RFC] Adding Support for Firefox's Gecko Profile Format

[Anup Sharma]

On Tue, Jul 04, 2023 at 11:22:14PM +0530, Anup Sharma wrote:
> On Tue, Jul 04, 2023 at 08:08:11PM +0530, Anup Sharma wrote:
> > On Thu, Jun 29, 2023 at 09:35:22AM +0300, Adrian Hunter wrote:
> > > On 29/06/23 09:26, Namhyung Kim wrote:
> > > > Hi Anup,
> > > > 
> > > > On Wed, Jun 28, 2023 at 1:11 PM Anup Sharma <anupnewsmail@xxxxxxxxx> wrote:
> > > >>
> > > >> On Wed, Jun 14, 2023 at 01:17:52PM -0700, Ian Rogers wrote:
> > > >>> On Wed, Jun 14, 2023 at 1:14 PM Anup Sharma <anupnewsmail@xxxxxxxxx> wrote:
> > > >>>>
> > > >>>> On Thu, Jun 08, 2023 at 06:23:49PM +0530, Anup Sharma wrote:
> > > >>>> I wanted to take a moment to provide you with an update on the progress
> > > >>>> of our Firefox Gecko converter work. While I must emphasize that the code
> > > >>>> I'm sharing is not the final version, I wanted to share the advancements
> > > >>>> I have made thus far.
> > > >>>>
> > > >>>> This script can generate a JSON format from the output of the "perf script" command.
> > > >>>> I attempted to load this JSON file into profile.firefox.com, and although it
> > > >>>> successfully loaded, the call tree are not visible. I'm certain this issue
> > > >>>> is related to the format of the JSON file or if there is another underlying
> > > >>>> cause. I will continue investigating to determine the cause of this problem.
> > > >>>
> > > >>> Great Anup, progress can be frustrating slow at first but it is a good
> > > >>> milestone to be generating output and having firefox profiler consume
> > > >>> it. You can open up the JavaScript console for the firefox profiler
> > > >>> and it will give some debug output. This should hopefully help you
> > > >>> make progress.
> > > >>
> > > >> This week I tried playing with perf-script-python, the first challenge was
> > > >> figuring out how to parse the data coming from the process_events.
> > > >> Understanding the structure and extracting the necessary information
> > > >> proved to be more complex than anticipated. This required me to spend
> > > >> a significant amount of time researching and experimenting with different
> > > >> parsing techniques.
> > > > 
> > > > I'm not sure what kind of parsing you meant.  IIRC the sample data is
> > > > in a dict and all the information should be available there.  Maybe we
> > > > missed some new sample data types though.
> > > 
> > > Most data is there.  There are existing scripts in
> > > tools/perf/scripts/python/ for reference.
> > > 
> > > There is also the dlfilter API:
> > > 
> > > https://perf.wiki.kernel.org/index.php/Latest_Manual_Page_of_perf-dlfilter.1
> > >
> > Hello everyone,
> > 
> > I hope you're all doing well. I have been working on the using process_events function and have made
> > progress in developing the converter script. Currently, I am in the testing phase. However, I
> > have encountered a problem after performing some recent compilations. I am no longer receiving
> > the complete callchains in the output as I used to.
> > For Example, when I ran the command perf script -F +pid, the output would include detailed
> > information like the following example:
> > 
> > perf-exec  132554/132554  171854.674582:        356 cycles:P: 
> > 	ffffffff8fab3fc6 arch_static_branch+0x6 (inlined)
> > 	ffffffff8fab3fc6 static_key_false+0x6 (inlined)
> > 	ffffffff8fab3fc6 native_write_msr+0x6 (/lib/modules/6.4.0-rc1/build/vmlinux)
> > 	ffffffff8fa12ca5 intel_pmu_enable_all+0x15 (/lib/modules/6.4.0-rc1/build/vmlinux)
> > 
> > However, in my current situation, the output is limited to the following:
> > 
> > perf  108107/108107  [000] 67650.031827:          1 cycles:P:  
> > ffffffff83ab3fc4 native_write_msr+0x4 (/lib/modules/6.4.0-rc1/build/vmlinux)
> > perf  108107/108107  [000] 67650.031832:          1 cycles:P:
> > ffffffff83ab3fc4 native_write_msr+0x4 (/lib/modules/6.4.0-rc1/build/vmlinux)
> > 
> > It seems to be issue in perf record. I would appreciate any suggestions or assistance
> > in resolving this issue. Thank you all for your help.
> 
> My apologies, I realized that I had mistakenly not included the '-g' option,
> and it is now working fine."

Dear all,

I want to share a recent development in our project that
I have been working on this week. I have successfully
implemented a new feature that allows users to directly
open the browser and visualize the output of perf.data
on Firefox Profiler.

To enhance user experience, I have made sure this option
is enabled by default. Additionally, I've also added a
handy command line argument, "--save-only," which allows
users to save the output file as needed. This enhancement
replaces the previous standard output method, providing a
much-improved experience.

Moreover, Everything is processed locally, and no data is
sent to any server.

During this implementation, I had the opportunity to delve
into several exciting topics, including threading, HTTP server,
and CORS mechanisms. The learning process has been both
challenging and rewarding.

Currently, I am in the process of performing some clean-up tasks.
Once that is complete, I will be sharing the next version of the
series, which includes this fantastic new feature, with all of
you very soon.

Thank you for your ongoing support and encouragement.

Thanks,
Anup

> > Regards,
> > Anup
> > > > 
> > > >>
> > > >> The second challenge revolved around the usage of event hooks provided with the perf script
> > > >> python event handlers. I found myself deliberating between two approaches. The first
> > > >> approach involved creating custom functions that would be called using the event
> > > >> handlers. These functions would then save the data in an organized format within
> > > >> globally created data structures. The alternative approach was to write the entire
> > > >> logic inside the event handlers themselves.
> > > >>
> > > >> Additionally, I contemplated whether it would be more suitable to handle the creation of
> > > >> a Gecko format for JSON and the profile format within the same script or to separate
> > > >> them into different scripts.
> > > >>
> > > >> I will discuss this points during tomorrow's office hour.
> > > >>
> > > >> However, I have gained a deeper understanding of the problem at hand and will use this
> > > >> knowledge to make more informed decisions and progress more effectively in the coming weeks.
> > > > 
> > > > Sounds like you did a good amount of research on how perf script
> > > > works.  We can talk about the details in the meeting.
> > > > 
> > > > Thanks,
> > > > Namhyung
> > >