binfmt_misc & different PE binaries

From: Pali Rohár
Date: Thu Jul 06 2023 - 07:56:00 EST


Hello,

I would like to ask how to properly register binfmt_misc for different
PE binaries, so kernel could execute the correct loader for them.

I mean, how to register support for Win32 (console/gui) PE binaries and
also for CLR PE binaries (dotnet). Win32 needs to be executed under wine
and CLR ideally under dotnet core (or mono).

I have read kernel documentation files admin-guide/binfmt-misc.rst
and admin-guide/mono.rst. But seems that they are in conflicts as both
wants to registers its own handler for the same magic:

echo ':DOSWin:M::MZ::/usr/local/bin/wine:' > register

echo ':CLR:M::MZ::/usr/bin/mono:' > /proc/sys/fs/binfmt_misc/register

Not mentioning the fact that they register DOS MZ handler, which matches
not only all PE binaries (including EFI, libraries, other processors),
but also all kind of other NE/LE/LX binaries and different DOS extenders.

>From documentation it looks like that even registering PE binaries is
impossible by binfmt_misc as PE is detected by checking that indirect
reference from 0x3C is PE\0\0. And distinguish between Win32 and CLR
needs to parse PE COM descriptor directory.

Or it is possible to write binfmt_misc pattern match based on indirect
offset?