Re: [PATCH v1 00/14] Fix perf tools/lib includes

From: Arnaldo Carvalho de Melo
Date: Wed Nov 09 2022 - 13:27:39 EST

Next message: Mark Brown: "Re: [PATCH] regmap-irq: Use the new num_config_regs property in regmap_add_irq_chip_fwnode"
Previous message: syzbot: "Re: [syzbot] memory leak in drm_vma_node_allow"
In reply to: Ian Rogers: "Re: [PATCH v1 00/14] Fix perf tools/lib includes"
Next in thread: Ian Rogers: "Re: [PATCH v1 00/14] Fix perf tools/lib includes"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Em Wed, Nov 09, 2022 at 08:57:01AM -0800, Ian Rogers escreveu:
> On Wed, Nov 9, 2022 at 8:52 AM Ian Rogers <irogers@xxxxxxxxxx> wrote:
> >
> > On Wed, Nov 9, 2022 at 7:27 AM Arnaldo Carvalho de Melo <acme@xxxxxxxxxx> wrote:
> > >
> > > Em Mon, Nov 07, 2022 at 11:35:04PM -0800, Ian Rogers escreveu:
> > > > The previous build would add -Itools/lib and get dependencies for
> > > > libtraceevent, libsubcmd, libsymbol, libapi and libbpf meaning that
> > > > overriding these libraries would change the link time dependency but
> > > > the headers would erroneously come from tools/lib. Fix the build to
> > > > install headers and then depend on these. To reduce exposing internal
> > > > headers/APIs some clean up is performed. tools/lib/symbol has a
> > > > Makefile added, while tools/lib/api and tools/lib/subcmd have install
> > > > targets added. The pattern used for the dependencies in Makefile.perf
> > > > is modelled on libbpf.
> > >
> > > It builds with O=, I tried it one by one, but there are problems with
> > > make from a detached tarball, that is how I do the container builds, see
> > > below, I'm trying to figure this out...
> >
> > Sorry, I didn't know to test this. The added Build and Makefile in
> > libsymbol are missing, so I'd guess that's the major issue.
> >
> > Thanks,
> > Ian
>
> The following fixes this for me:

But then there are some other problems related to making sure those
libraries are built and installed _before_ perf proper starts building
when it expects to find the headers for those libraries.

So perhaps we need to first run the 'install' target for those libraries
before build perf proper?

I'm pushing what I have to perf/tools-libs-includes branch.

- Arnaldo

> --- a/tools/perf/MANIFEST
> +++ b/tools/perf/MANIFEST
> @@ -13,8 +13,7 @@ tools/lib/ctype.c
> tools/lib/hweight.c
> tools/lib/rbtree.c
> tools/lib/string.c
> -tools/lib/symbol/kallsyms.c
> -tools/lib/symbol/kallsyms.h
> +tools/lib/symbol
> tools/lib/find_bit.c
> tools/lib/bitmap.c
> tools/lib/list_sort.c
>
> Thanks,
> Ian
>
> > > ⬢[acme@toolbox perf]$ make perf-tar-src-pkg
> > > TAR
> > > PERF_VERSION = 6.1.rc3.g7e5d8b7a1fbd
> > > ⬢[acme@toolbox perf]$ mv perf-6.1.0-rc3.tar /tmp
> > > ⬢[acme@toolbox perf]$ cd /tmp
> > > ⬢[acme@toolbox tmp]$ tar xf perf-6.1.0-rc3.tar
> > > ⬢[acme@toolbox tmp]$ cd perf-6.1.0-rc3/
> > > ⬢[acme@toolbox perf-6.1.0-rc3]$ make -C tools/perf
> > > make: Entering directory '/tmp/perf-6.1.0-rc3/tools/perf'
> > > BUILD: Doing 'make -j32' parallel build
> > > HOSTCC fixdep.o
> > > HOSTLD fixdep-in.o
> > > LINK fixdep
> > >
> > > Auto-detecting system features:
> > > ... dwarf: [ on ]
> > > ... dwarf_getlocations: [ on ]
> > > ... glibc: [ on ]
> > > ... libbfd: [ on ]
> > > ... libbfd-buildid: [ on ]
> > > ... libcap: [ on ]
> > > ... libelf: [ on ]
> > > ... libnuma: [ on ]
> > > ... numa_num_possible_cpus: [ on ]
> > > ... libperl: [ on ]
> > > ... libpython: [ on ]
> > > ... libcrypto: [ on ]
> > > ... libunwind: [ on ]
> > > ... libdw-dwarf-unwind: [ on ]
> > > ... zlib: [ on ]
> > > ... lzma: [ on ]
> > > ... get_cpuid: [ on ]
> > > ... bpf: [ on ]
> > > ... libaio: [ on ]
> > > ... libzstd: [ on ]
> > >
> > > GEN common-cmds.h
> > > PERF_VERSION = 6.1.rc3.g7e5d8b7a1fbd
> > > CC perf-read-vdso32
> > > GEN perf-archive
> > > GEN perf-iostat
> > > CC dlfilters/dlfilter-test-api-v0.o
> > > CC dlfilters/dlfilter-show-cycles.o
> > > CC jvmti/libjvmti.o
> > > make[3]: *** No rule to make target '/tmp/perf-6.1.0-rc3/tools/perf/libsymbol/libsymbol.a'. Stop.
> > > make[2]: *** [Makefile.perf:907: /tmp/perf-6.1.0-rc3/tools/perf/libsymbol/libsymbol.a] Error 2
> > > make[2]: *** Waiting for unfinished jobs....
> > > CC jvmti/jvmti_agent.o
> > > CC jvmti/libstring.o
> > > CC jvmti/libctype.o
> > > GEN pmu-events/pmu-events.c
> > > INSTALL headers
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libsubcmd/exec-cmd.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libsubcmd/help.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libsubcmd/pager.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libsubcmd/parse-options.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libsubcmd/run-command.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libsubcmd/sigchain.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libsubcmd/subcmd-config.o
> > > INSTALL headers
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent/event-parse.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_jbd2.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_hrtimer.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_kmem.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_kvm.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_mac80211.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_function.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_sched_switch.o
> > > INSTALL headers
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libperf/core.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libapi/cpu.o
> > > MKDIR /tmp/perf-6.1.0-rc3/tools/perf/libapi/fd/
> > > MKDIR /tmp/perf-6.1.0-rc3/tools/perf/libapi/fs/
> > > MKDIR /tmp/perf-6.1.0-rc3/tools/perf/libapi/fs/
> > > MKDIR /tmp/perf-6.1.0-rc3/tools/perf/libapi/fs/
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libapi/fd/array.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libapi/fs/fs.o
> > > GEN /tmp/perf-6.1.0-rc3/tools/perf/libbpf/bpf_helper_defs.h
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libapi/fs/tracing_path.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libapi/fs/cgroup.o
> > > INSTALL /tmp/perf-6.1.0-rc3/tools/perf/libbpf/include/bpf/bpf.h
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libapi/debug.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_futex.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_xen.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_scsi.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_cfg80211.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_tlb.o
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_jbd2-in.o
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_hrtimer-in.o
> > > INSTALL /tmp/perf-6.1.0-rc3/tools/perf/libbpf/include/bpf/libbpf.h
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_kmem-in.o
> > > INSTALL headers
> > > INSTALL /tmp/perf-6.1.0-rc3/tools/perf/libbpf/include/bpf/btf.h
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_mac80211-in.o
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_sched_switch-in.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libperf/cpumap.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libperf/threadmap.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libapi/str_error_r.o
> > > INSTALL /tmp/perf-6.1.0-rc3/tools/perf/libbpf/include/bpf/libbpf_common.h
> > > LINK /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_jbd2.so
> > > LINK /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_hrtimer.so
> > > INSTALL /tmp/perf-6.1.0-rc3/tools/perf/libbpf/include/bpf/libbpf_legacy.h
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_function-in.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libperf/evsel.o
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_kvm-in.o
> > > LINK /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_kmem.so
> > > LINK /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_mac80211.so
> > > LINK /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_sched_switch.so
> > > INSTALL /tmp/perf-6.1.0-rc3/tools/perf/libbpf/include/bpf/bpf_helpers.h
> > > INSTALL /tmp/perf-6.1.0-rc3/tools/perf/libbpf/include/bpf/bpf_tracing.h
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent/event-plugin.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libperf/evlist.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libperf/mmap.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent/trace-seq.o
> > > INSTALL /tmp/perf-6.1.0-rc3/tools/perf/libbpf/include/bpf/bpf_endian.h
> > > INSTALL /tmp/perf-6.1.0-rc3/tools/perf/libbpf/include/bpf/bpf_core_read.h
> > > if [ ! -d ''/tmp/perf-6.1.0-rc3/tools/perf/libapi'/include/api/fs' ]; then install -d -m 755 ''/tmp/perf-6.1.0-rc3/tools/perf/libapi'/include/api/fs'; fi; install fs/tracing_path.h -m 644 ''/tmp/perf-6.1.0-rc3/tools/perf/libapi'/include/api/fs';
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libapi/fd/libapi-in.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libperf/zalloc.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent/parse-filter.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent/parse-utils.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent/kbuffer-parse.o
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_xen-in.o
> > > INSTALL /tmp/perf-6.1.0-rc3/tools/perf/libbpf/include/bpf/skel_internal.h
> > > INSTALL /tmp/perf-6.1.0-rc3/tools/perf/libbpf/include/bpf/libbpf_version.h
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent/tep_strerror.o
> > > INSTALL /tmp/perf-6.1.0-rc3/tools/perf/libbpf/include/bpf/usdt.bpf.h
> > > INSTALL /tmp/perf-6.1.0-rc3/tools/perf/libbpf/include/bpf/bpf_helper_defs.h
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent/event-parse-api.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libperf/xyarray.o
> > > LINK /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_function.so
> > > LINK /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_xen.so
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_futex-in.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libperf/lib.o
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_scsi-in.o
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_cfg80211-in.o
> > > MKDIR /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/
> > > MKDIR /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/
> > > MKDIR /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/
> > > MKDIR /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/
> > > LINK /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_kvm.so
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/libbpf.o
> > > LINK /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_futex.so
> > > LINK /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_scsi.so
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/bpf.o
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libapi/fs/libapi-in.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/nlattr.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/btf.o
> > > MKDIR /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_tlb-in.o
> > > LINK /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_cfg80211.so
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/str_error.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/netlink.o
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libapi/libapi-in.o
> > > LD jvmti/jvmti-in.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/libbpf_errno.o
> > > LINK /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/plugin_tlb.so
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/bpf_prog_linfo.o
> > > AR /tmp/perf-6.1.0-rc3/tools/perf/libapi/libapi.a
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/libbpf_probes.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/hashmap.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/btf_dump.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/ringbuf.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/strset.o
> > > GEN /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent_plugins/libtraceevent-dynamic-list
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/linker.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/gen_loader.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/relo_core.o
> > > CC /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/usdt.o
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libsubcmd/libsubcmd-in.o
> > > AR /tmp/perf-6.1.0-rc3/tools/perf/libsubcmd/libsubcmd.a
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libperf/libperf-in.o
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent/libtraceevent-in.o
> > > AR /tmp/perf-6.1.0-rc3/tools/perf/libperf/libperf.a
> > > LINK /tmp/perf-6.1.0-rc3/tools/perf/libtraceevent/libtraceevent.a
> > > CC pmu-events/pmu-events.o
> > > LD pmu-events/pmu-events-in.o
> > > LD /tmp/perf-6.1.0-rc3/tools/perf/libbpf/staticobjs/libbpf-in.o
> > > LINK /tmp/perf-6.1.0-rc3/tools/perf/libbpf/libbpf.a
> > > make[1]: *** [Makefile.perf:240: sub-make] Error 2
> > > make: *** [Makefile:70: all] Error 2
> > > make: Leaving directory '/tmp/perf-6.1.0-rc3/tools/perf'
> > > ⬢[acme@toolbox perf-6.1.0-rc3]$
> > >
> > >
> > >
> > > > The problem and solution were motivated by this issue and discussion:
> > > > https://lore.kernel.org/lkml/CAEf4BzbbOHQZUAe6iWaehKCPQAf3VC=hq657buqe2_yRKxaK-A@xxxxxxxxxxxxxx/
> > > >
> > > > Ian Rogers (14):
> > > > tools lib api: Add install target
> > > > tools lib subcmd: Add install target
> > > > perf build: Install libsubcmd locally when building
> > > > perf build: Install libapi locally when building
> > > > perf build: Install libperf locally when building
> > > > perf build: Install libtraceevent locally when building
> > > > tools lib api: Add missing install headers
> > > > tools lib perf: Add missing install headers
> > > > tool lib symbol: Add Makefile/Build
> > > > perf build: Install libsymbol locally when building
> > > > perf expr: Tidy hashmap dependency
> > > > perf thread_map: Reduce exposure of libperf internal API
> > > > perf cpumap: Tidy libperf includes
> > > > perf build: Use tools/lib headers from install path
> > > >
> > > > tools/lib/api/Makefile | 52 ++++++
> > > > tools/lib/perf/Makefile | 10 +-
> > > > tools/lib/subcmd/Makefile | 49 ++++++
> > > > tools/lib/symbol/Build | 1 +
> > > > tools/lib/symbol/Makefile | 115 +++++++++++++
> > > > tools/perf/.gitignore | 7 +-
> > > > tools/perf/Makefile.config | 2 -
> > > > tools/perf/Makefile.perf | 152 ++++++++++++------
> > > > tools/perf/builtin-stat.c | 1 +
> > > > tools/perf/builtin-trace.c | 4 +-
> > > > tools/perf/tests/cpumap.c | 2 +-
> > > > tools/perf/tests/expr.c | 1 +
> > > > tools/perf/tests/openat-syscall.c | 1 +
> > > > tools/perf/tests/pmu-events.c | 1 +
> > > > tools/perf/tests/thread-map.c | 1 +
> > > > tools/perf/util/Build | 5 -
> > > > tools/perf/util/auxtrace.h | 2 +-
> > > > tools/perf/util/bpf-loader.c | 4 -
> > > > tools/perf/util/bpf_counter.c | 2 +-
> > > > tools/perf/util/cpumap.c | 1 +
> > > > tools/perf/util/cpumap.h | 2 +-
> > > > tools/perf/util/evsel.c | 5 +-
> > > > tools/perf/util/evsel.h | 2 -
> > > > tools/perf/util/expr.c | 1 +
> > > > tools/perf/util/expr.h | 7 +-
> > > > tools/perf/util/metricgroup.c | 1 +
> > > > tools/perf/util/python.c | 6 +-
> > > > .../scripting-engines/trace-event-python.c | 2 +-
> > > > tools/perf/util/stat-shadow.c | 1 +
> > > > tools/perf/util/stat.c | 4 -
> > > > tools/perf/util/thread_map.c | 1 +
> > > > tools/perf/util/thread_map.h | 2 -
> > > > 32 files changed, 361 insertions(+), 86 deletions(-)
> > > > create mode 100644 tools/lib/symbol/Build
> > > > create mode 100644 tools/lib/symbol/Makefile
> > > >
> > > > --
> > > > 2.38.1.431.g37b22c650d-goog
> > >
> > > --
> > >
> > > - Arnaldo

--

- Arnaldo

Next message: Mark Brown: "Re: [PATCH] regmap-irq: Use the new num_config_regs property in regmap_add_irq_chip_fwnode"
Previous message: syzbot: "Re: [syzbot] memory leak in drm_vma_node_allow"
In reply to: Ian Rogers: "Re: [PATCH v1 00/14] Fix perf tools/lib includes"
Next in thread: Ian Rogers: "Re: [PATCH v1 00/14] Fix perf tools/lib includes"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]