Re: Linux Testing Microconference at LPC

[Guillaume Tucker]

Hi Steve,

On 18/04/2019 14:26, Steven Rostedt wrote:
> On Thu, 18 Apr 2019 11:22:47 +0100
> "Gustavo Padovan" <gustavo.padovan@xxxxxxxxxxxxx> wrote:
> 
>> Guillaume would like to talk about the his work on kernelCI on automated bisection, functional testing and modular pipelines.
> 
> Can you be more specific about what Guillaume wants to talk about for
> those topics? I hope it's not just a "let everyone know what Guillaume
> has done" talk. Plumbers is about discussions of on going and future
> work. If these are all work-in-progress and Guillaume is looking for
> input from other stakeholder developers, then that is exactly what
> Plumbers is about. But if this is just to show developers what was done
> and how to use the finished work, then save that for something like
> Open Source Summit.

Sure, this is very much about discussing how to grow KernelCI in
a more community-driven way.  It should also help defining the
role of KernelCI in the wider kernel community, alongside other
test infrastructures.

As Gustavo mentioned, there is the bisection tool which has
already started to bear fruit.  It has a few known limitations
that I've already started to address and several aspects that
would be worth discussing for future development.  In particular,
it quickly gets a lot more complex when bisecting long test
suites compared to the simple boot tests we're doing now.

Then there is the topic of adding more test suites to cover more
areas of the kernel.  Ultimately it would be good to have a way
to enable anyone to submit a new test suite to KernelCI and not
just the small team of developers working on it now, to scale
proportionally with the size of the task.  It's also worth
discussing a strategy as to how to expand testing (which areas to
cover first, how to make best use of the available test capacity
etc.).

One more thing I have in mind is an idea I started to explain in
a document[1] whereby some components of the current KernelCI
system could have alternative instances.  For example, if an
organisation is doing some upstream kernel testing and wants to
join KernelCI to contribute the results, rather than requiring
all the tests to be run in LAVA or all the kernels to be built
with Jenkins on kernelci.org, there could be alternative
automated build systems for some binaries to be produced and
alternative test systems to test them.  Having a common test
results format also helps, but it's only the last step in the
pipeline so there is more to it if we want to explore the full
potential of collaborative testing.  So that seems like a good
subject for discussion too as it's mostly still all up in the
air.

Best wishes,
Guillaume

[1] https://groups.io/g/kernelci/topic/kernelci_modular_pipeline/29692355