Re: ERROR: INT DW_ATE_unsigned_1 Error emitting BTF type
From: Yonghong Song
Date: Sat Feb 06 2021 - 03:32:26 EST
On 2/5/21 10:52 PM, Sedat Dilek wrote:
On Sat, Feb 6, 2021 at 7:26 AM Sedat Dilek <sedat.dilek@xxxxxxxxx> wrote:
On Sat, Feb 6, 2021 at 6:53 AM Sedat Dilek <sedat.dilek@xxxxxxxxx> wrote:
On Sat, Feb 6, 2021 at 6:44 AM Sedat Dilek <sedat.dilek@xxxxxxxxx> wrote:
On Sat, Feb 6, 2021 at 4:34 AM Sedat Dilek <sedat.dilek@xxxxxxxxx> wrote:
On Fri, Feb 5, 2021 at 10:54 PM Yonghong Song <yhs@xxxxxx> wrote:
On 2/5/21 12:31 PM, Sedat Dilek wrote:
On Fri, Feb 5, 2021 at 9:03 PM Yonghong Song <yhs@xxxxxx> wrote:
On 2/5/21 11:24 AM, Arnaldo Carvalho de Melo wrote:
Em Fri, Feb 05, 2021 at 11:10:08AM -0800, Yonghong Song escreveu:
On 2/5/21 11:06 AM, Sedat Dilek wrote:
On Fri, Feb 5, 2021 at 7:53 PM Sedat Dilek <sedat.dilek@xxxxxxxxx> wrote:
Grepping through linux.git/tools I guess some BTF tools/libs need to
know what BTF_INT_UNSIGNED is?
BTF_INT_UNSIGNED needs kernel support. Maybe to teach pahole to
ignore this for now until kernel infrastructure is ready.
Yeah, I thought about doing that.
Not sure whether this information will be useful or not
for BTF. This needs to be discussed separately.
Maybe search for the rationale for its introduction in DWARF.
In LLVM, we have:
uint8_t BTFEncoding;
switch (Encoding) {
case dwarf::DW_ATE_boolean:
BTFEncoding = BTF::INT_BOOL;
break;
case dwarf::DW_ATE_signed:
case dwarf::DW_ATE_signed_char:
BTFEncoding = BTF::INT_SIGNED;
break;
case dwarf::DW_ATE_unsigned:
case dwarf::DW_ATE_unsigned_char:
BTFEncoding = 0;
break;
I think DW_ATE_unsigned can be ignored in pahole since
the default encoding = 0. A simple comment is enough.
Yonghong Son, do you have a patch/diff for me?
Looking at error message from log:
LLVM_OBJCOPY=/opt/binutils/bin/objcopy /opt/pahole/bin/pahole -J
.tmp_vmlinux.btf
[115] INT DW_ATE_unsigned_1 Error emitting BTF type
Encountered error while encoding BTF.
Not exactly what is the root cause. Maybe bt->bit_size is not
encoded correctly. Could you put vmlinux (in the above it is
.tmp_vmlinux.btf) somewhere, I or somebody else can investigate
and provide a proper fix.
[ TO: Masahiro ]
Thanks for taking care Yonghong - hope this is your first name, if not
I am sorry.
In case of mixing my first and last name you will make me female -
Dilek is a Turkish female first name :-).
So, in some cultures you need to be careful.
Anyway... back to business and facts.
Out of frustration I killed my last build via `make distclean`.
The whole day I tested diverse combination of GCC-10 and LLVM-12
together with BTF Kconfigs, selfmade pahole, etc.
I will do ne run with some little changes:
#1: Pass LLVM_IAS=1 to make (means use Clang's Integrated ASsembler -
as per Nick this leads to the same error - should be unrelated)
#2: I did: DEBUG_INFO_COMPRESSED y -> n
#2 I did in case you need vmlinux and I have to upload - I will
compress the resulting vmlinux with ZSTD.
You need vmlinux or .tmp_vmlinux.btf file?
Nick was not allowed from his company to download from a Dropbox link.
So, as an alternative I can offer GoogleDrive...
...or bomb into your INBOX :-).
Now, why I CCed Masahiro:
In case of ERRORs when running `scripts/link-vmlinux.sh` above files
will be removed.
Last, I found a hack to bypass this - means to keep these files (I
need to check old emails).
Masahiro, you see a possibility to have a way to keep these files in
case of ERRORs without doing hackery?
From a previous post in this thread:
+ info BTF .btf.vmlinux.bin.o
+ [ != silent_ ]
+ printf %-7s %s\n BTF .btf.vmlinux.bin.o
BTF .btf.vmlinux.bin.o
+ LLVM_OBJCOPY=llvm-objcopy /opt/pahole/bin/pahole -J .tmp_vmlinux.btf
[2] INT long unsigned int Error emitting BTF type
Encountered error while encoding BTF.
+ llvm-objcopy --only-section=.BTF --set-section-flags
.BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o
...
+ info BTFIDS vmlinux
+ [ != silent_ ]
+ printf %-7s %s\n BTFIDS vmlinux
BTFIDS vmlinux
+ ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
FAILED: load BTF from vmlinux: Invalid argument
+ on_exit
+ [ 255 -ne 0 ]
+ cleanup
+ rm -f .btf.vmlinux.bin.o
+ rm -f .tmp_System.map
+ rm -f .tmp_vmlinux.btf .tmp_vmlinux.kallsyms1
.tmp_vmlinux.kallsyms1.S .tmp_vmlinux.kallsyms1.o
.tmp_vmlinux.kallsyms2 .tmp_vmlinux.kallsyms2.S .tmp_vmlinux.kallsyms
2.o
+ rm -f System.map
+ rm -f vmlinux
+ rm -f vmlinux.o
make[3]: *** [Makefile:1166: vmlinux] Error 255
^^^ Look here.
With this diff:
$ git diff scripts/link-vmlinux.sh
diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index eef40fa9485d..40f1b6aae553 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -330,7 +330,7 @@ vmlinux_link vmlinux "${kallsymso}" ${btf_vmlinux_bin_o}
# fill in BTF IDs
if [ -n "${CONFIG_DEBUG_INFO_BTF}" -a -n "${CONFIG_BPF}" ]; then
info BTFIDS vmlinux
- ${RESOLVE_BTFIDS} vmlinux
+ ##${RESOLVE_BTFIDS} vmlinux
fi
if [ -n "${CONFIG_BUILDTIME_TABLE_SORT}" ]; then
This files are kept - not removed:
$ LC_ALL=C ll .*btf* vmlinux vmlinux.o
-rwxr-xr-x 1 dileks dileks 31M Feb 6 06:37 .btf.vmlinux.bin.o
-rwxr-xr-x 1 dileks dileks 348M Feb 6 06:37 .tmp_vmlinux.btf
-rwxr-xr-x 1 dileks dileks 348M Feb 6 06:37 vmlinux
-rw-r--r-- 1 dileks dileks 344M Feb 6 06:37 vmlinux.o
Pleas let me know where to upload - Dropbox or GoogleDrive or
elsewhere and give me a link.
WOW, ZSTD is great :-).
$ zstd -19 -T0 -v vmlinux
*** zstd command line interface 64-bits v1.4.8, by Yann Collet ***
Note: 2 physical core(s) detected
vmlinux : 22.71% (364466016 => 82784801 bytes, vmlinux.zst)
$ du -m vmlinux*
348 vmlinux
79 vmlinux.zst
Dropbox link:
https://www.dropbox.com/sh/kvyh8ps7na0r1h5/AABfyNfDZ2bESse_bo4h05fFa?dl=0
I hope this is public available.
Inspecting vmlinux with llvm-dwarf:
$ /opt/llvm-toolchain/bin/llvm-dwarfdump vmlinux | grep DW_AT_name |
grep DW_ATE_ | sort | uniq
DW_AT_name ("DW_ATE_signed_1")
DW_AT_name ("DW_ATE_signed_16")
DW_AT_name ("DW_ATE_signed_32")
DW_AT_name ("DW_ATE_signed_64")
DW_AT_name ("DW_ATE_signed_8")
DW_AT_name ("DW_ATE_unsigned_1")
DW_AT_name ("DW_ATE_unsigned_128")
DW_AT_name ("DW_ATE_unsigned_16")
DW_AT_name ("DW_ATE_unsigned_24")
DW_AT_name ("DW_ATE_unsigned_32")
DW_AT_name ("DW_ATE_unsigned_40")
DW_AT_name ("DW_ATE_unsigned_64")
DW_AT_name ("DW_ATE_unsigned_8")
- Sedat -
Thanks for the above dropbot link, I am able to reproduce the issue.
I tried to use latest llvm + Nick's patch + latest pahole + dwarf5
config option to compile kernel with LLVM=1 LLVM_IAS=1, somehow
I did not hit the issue. It complained like
MODPOST vmlinux.symvers
WARNING: modpost: vmlinux.o(.text+0x6ce73): Section mismatch in
reference from the function __nodes
_weight() to the variable .init.data:numa_nodes_parsed
The function __nodes_weight() references
the variable __initdata numa_nodes_parsed.
This is often because __nodes_weight lacks a __initdata
annotation or the annotation of numa_nodes_parsed is wrong.
but otherwise compilation is fine.
With the above vmlinux, the issue appears to be handling
DW_ATE_signed_1, DW_ATE_unsigned_{1,24,40}.
The following patch should fix the issue:
-bash-4.4$ git diff
diff --git a/dwarf_loader.c b/dwarf_loader.c
index b73d786..0341b5e 100644
--- a/dwarf_loader.c
+++ b/dwarf_loader.c
@@ -467,8 +467,16 @@ static struct base_type *base_type__new(Dwarf_Die
*die, struct cu *cu)
if (bt != NULL) {
tag__init(&bt->tag, cu, die);
- bt->name = strings__add(strings, attr_string(die,
DW_AT_name));
- bt->bit_size = attr_numeric(die, DW_AT_byte_size) * 8;
+ const char *name = attr_string(die, DW_AT_name);
+ bt->name = strings__add(strings, name);
+ /* DW_ATE_unsigned_1 has DW_AT_byte_size == 0.
+ * specially process it.
+ */
+ if (strcmp(name, "DW_ATE_unsigned_1") == 0)
+ bt->bit_size = 1;
+ else
+ bt->bit_size = attr_numeric(die,
DW_AT_byte_size) * 8;
+
uint64_t encoding = attr_numeric(die, DW_AT_encoding);
bt->is_bool = encoding == DW_ATE_boolean;
bt->is_signed = encoding == DW_ATE_signed;
diff --git a/libbtf.c b/libbtf.c
index 9f76283..b5aa077 100644
--- a/libbtf.c
+++ b/libbtf.c
@@ -367,13 +367,32 @@ static void btf_log_func_param(const struct
btf_elf *btfe,
}
}
+/* btf requires power-of-2 bytes, yet dwarf may have something like
+ * DW_ATE_unsigned_24 which encodes as 24 bits (3 bytes).
+ */
+static int bits_to_int_bytes(uint16_t bit_size)
+{
+ if (bit_size <= 8)
+ return 1;
+ if (bit_size <= 16)
+ return 2;
+ if (bit_size <= 32)
+ return 4;
+ if (bit_size <= 64)
+ return 8;
+ if (bit_size <= 128)
+ return 16;
+ /* BTF supports upto 16byte int (__int128). */
+ return -1;
+}
+
int32_t btf_elf__add_base_type(struct btf_elf *btfe, const struct
base_type *bt,
const char *name)
{
struct btf *btf = btfe->btf;
const struct btf_type *t;
uint8_t encoding = 0;
- int32_t id;
+ int32_t id, nbytes;
if (bt->is_signed) {
encoding = BTF_INT_SIGNED;
@@ -384,7 +403,13 @@ int32_t btf_elf__add_base_type(struct btf_elf
*btfe, const struct base_type *b
t,
return -1;
}
- id = btf__add_int(btf, name, BITS_ROUNDUP_BYTES(bt->bit_size),
encoding);
+ nbytes = bits_to_int_bytes(bt->bit_size);
+ if (nbytes < 0) {
+ fprintf(stderr, "not supported bit_size %hu\n",
bt->bit_size);
+ return -1;
+ }
+
+ id = btf__add_int(btf, name, nbytes, encoding);
if (id < 0) {
btf_elf__log_err(btfe, BTF_KIND_INT, name, true, "Error
emitting BTF type");
} else {
-bash-4.4$
Please help do a test. I can submit a formal patch tomorrow.