Re: [PATCH v4 0/10] perf: enable compression of record mode trace to save storage space

From: Jiri Olsa
Date: Wed Mar 13 2019 - 10:37:31 EST


On Thu, Feb 28, 2019 at 11:35:24AM +0300, Alexey Budankov wrote:
>
> The patch set implements runtime trace compression (-z option) in
> record mode and trace auto decompression in report and inject modes.
> Streaming Zstandard (Zstd) API (zstd) is used for compression and
> decompression of data that come from kernel mmaped data buffers.
>
> Usage of implemented -z,--compression_level=n option provides ~3-5x
> avg. trace file size reduction on variety of tested workloads what
> saves storage space on larger server systems where trace file size
> can easily reach several tens or even hundreds of GiBs, especially
> when profiling with dwarf-based stacks and tracing of context switches.
> Implemented -f,--mmap-flush option can be used to avoid compressing
> every single byte of data and increase compression ratio at the same
> time lowering tool runtime overhead. Default option value is 1 what
> is equal to the current perf record implementation. The option is
> independent from -z setting and doesn't vary with compression level:
>
> $ tools/perf/perf record -z 1 -e cycles -- matrix.gcc
> $ tools/perf/perf record --aio=1 -z 1 -e cycles -- matrix.gcc
> $ tools/perf/perf record -z 1 -f 1024 -e cycles -- matrix.gcc
> $ tools/perf/perf record --aio=1 -z 1 -f 1024 -e cycles -- matrix.gcc
>
> Runtime compression overhead has been measured for serial and AIO
> trace writing modes when profiling matrix multiplication workload
> with the following results:
>
> -------------------------------------------------------------
> | SERIAL | AIO-1 |
> -----------------------------------------------------------------
> |-z | OVH(x) | ratio(x) size(MiB) | OVH(x) | ratio(x) size(MiB) |
> |----------------------------------------------------------------
> | 0 | 1,00 | 1,000 179,424 | 1,00 | 1,000 187,527 |
> | 1 | 1,04 | 8,427 181,148 | 1,01 | 8,474 188,562 |
> | 2 | 1,07 | 8,055 186,953 | 1,03 | 7,912 191,773 |
> | 3 | 1,04 | 8,283 181,908 | 1,03 | 8,220 191,078 |
> | 5 | 1,09 | 8,101 187,705 | 1,05 | 7,780 190,065 |
> | 8 | 1,05 | 9,217 179,191 | 1,12 | 6,111 193,024 |
> -----------------------------------------------------------------
>
> OVH = (Execution time with -z N) / (Execution time with -z 0)
>
> ratio - compression ratio
> size - number of bytes that was compressed
>
> size ~= trace file x ratio
>
> See complete description of measurement conditions with details below.
>
> Introduced compression functionality can be disabled or configured from
> the command line using NO_LIBZSTD and LIBZSTD_DIR defines:
>
> $ make -C tools/perf NO_LIBZSTD=1 clean all
> $ make -C tools/perf LIBZSTD_DIR=/path/to/zstd/sources/ clean all
>
> If your system has some version of the zstd package preinstalled then
> the build system finds and uses it during the build. Auto detection
> feature status is reported just before compilation starts, as usual.
> If you still prefer to compile with some version of zstd that is not
> preinstalled you have capability to refer the compilation to that
> version using LIBZSTD_DIR define.
>
> See 'perf test' results below for enabled and disabled (NO_LIBZSTD=1)
> feature configurations.
>
> ---
> Alexey Budankov (10):
> feature: implement libzstd check, LIBZSTD_DIR and NO_LIBZSTD defines
> perf record: implement -f,--mmap-flush=<threshold> option
> perf session: define bytes_transferred and bytes_compressed metrics
> perf record: implement COMPRESSED event record and its attributes
> perf mmap: implement dedicated memory buffer for data compression
> perf util: introduce Zstd based streaming compression API
> perf record: implement -z,--compression_level=n option and compression
> perf report: implement record trace decompression
> perf inject: enable COMPRESSED records decompression
> perf tests: implement Zstd comp/decomp integration test

the record messages disappeared totaly:

[jolsa@krava perf]$ ./perf record -z 1 ls
[jolsa@krava perf]$ ./perf record -z 22 ls
[jolsa@krava perf]$ ./perf record ls
[jolsa@krava perf]$


seems like we bail out for some reason with no message:

[jolsa@krava perf]$ ./perf record ls
[jolsa@krava perf]$ ./perf report --stat
WARNING: The perf.data file's data size field is 0 which is unexpected.
Was the 'perf record' command properly terminated?
Error:
failed to process sample

jirka