Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Wed, Jan 23, 2019 at 02:37:47PM -0800, Daniel Colascione wrote:
> On Wed, Jan 23, 2019 at 1:29 PM Karim Yaghmour
> <karim.yaghmour@xxxxxxxxxxx> wrote:
[...]
> > Personally I advocated a more aggressive approach with Joel in private:
> > just put the darn headers straight into the kernel image, it's the
> > *only* artifact we're sure will follow the Android device whatever
> > happens to it (like built-in ftrace).
> 
> I was thinking along similar lines. Ordinarily, we make loadable
> kernel modules. What we kind of want here is a non-loadable kernel
> module --- or a non-loadable section in the kernel image proper. I'm
> not familiar with early-stage kernel loader operation: I know it's
> possible to crease discardable sections in the kernel image, but can
> we create sections that are never slurped into memory in the first
> place? If not, maybe loading and immediately discarding the header
> section is good enough.

I am happy to see if I can shrink it down further. Especially using xz and
stripping all comments period. I am optimistic this can be brought down
further to a point where it would make sense to everyone to build it into the
kernel. Lets see.

Last time I stripped comments, it went down by ~40%. What I haven't tried is
doing this *with* xz compression. I am also open to brainstorming what else
can be stripped.

OTOH the reason I didn't focus much on size is: modules are pretty much
universal and I'm confident of wide spread use of this feature for Android-based
products and not needing of "chasing headers" if we modularize it, since
Android's project treble has modularized things and modules are now default
enabled. Putting headers into a module lets us enjoy the ride there.

I am quite hosed for the next week or so to work on this, but I should be
able to get back to it after.

cheers,

 - Joel


> 
> > To that end, I even had some crazy
> > ideas on how to compress the headers even further than with std
> > compression algorithms -- here's a snippet from an email I sent Joel
> > some time back detailing such a hack:
> > > Since C headers have fairly constrained semantics and since the types of semantics generally used to name structs, etc. in the Linux kernel are well established, we can likely devise a very customized compression algorithm for the purpose.
> 
> Would such a thing really do better than LZMA? LZMA already has very
> clever techniques for eliminating long-range redundancies in
> compressible text, including redundancies at the sub-byte level. I can
> certainly understand the benefit of stripping comments, since removing
> comments really does decrease the total amount of information the
> compressor has to preserve, but I'm not sure how much the encoding
> scheme you propose below would help, since it reminds me of the
> encoding scheme that LZMA would discover automatically.
> 
> > Whether such craziness makes sense or is adopted or not isn't mine to
> > chart, but I certainly can't see eBPF reaching the same mass deployment
> > ftrace has within the Android ecosystem until there's a way to use it
> > without having to chase kernel headers independently of kernel images.
> > There are "too many clicks" involved and someone somewhere will drop the
> > ball if it's not glued to the kernel in some way shape or form. Any
> > solution that solves this is one I'd love to hear about.
> 
> I agree. There definitely needs to be a "just collect a damn trace"
> button that works on any device, and for this button to work and
> incorporate eBPF, the system needs to be able to describe itself.