Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Chuck Zmudzinski]

On 10/31/2023 8:08 AM, Marek Szyprowski wrote:
> Hi,
> 
> On 31.10.2023 00:03, Mario Marietto wrote:
>> We are a team of linux enthusiasts who are trying to boot Xen on a 
>> Samsung XE303C12 Chromebook aka "snow" following the suggestions in 
>> the slide show presentation here: 
>> https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm 
>> This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it 
>> is a Samsung armv7 chip with virtualization extensions. In particular, 
>> we have it working fairly well both on the bare metal with a recent 
>> 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS kernel with 
>> KVM, the older LTS kernel version is used to test KVM because support 
>> for KVM on arm v7 was removed from Linux around kernel version 5.7. So 
>> we know we have the hypervisor mode enabled because we were able to 
>> use it with KVM. For Xen, we are using the latest Debian build of Xen 
>> 4.17 for the Debian armhf architecture: (XEN) Xen version 4.17.2-pre 
>> (Debian 4.17.1+2-gb773c48e36-1) 
>> (pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx) (arm-linux-gnueabihf-gcc 
>> (Debian 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023 The 
>> Linux kernel is a custom build that adds the Xen config kernel options 
>> (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the same 
>> Chromebook model on the bare metal. I can provide the config options 
>> of the kernel that was used if that is helpful. Our method of booting 
>> is to have u-boot boot the Xen hypervisor and load the device tree 
>> after adding the dom0 to the otherwise unaltered device tree from the 
>> Linux kernel using u-boot fdt commands to add a /chosen node, as 
>> described on the Xen wiki and in the pages linked from there. We have 
>> also tried adding and loading an initrd.img using the device tree 
>> /chosen node but that made no difference in our tests. We actually 
>> have the Linux LTS kernel version 6.1.59 working as dom0 with Xen 
>> using the same version of u-boot that we used for KVM, but with a big 
>> problem. The problem we see is that when booting the 6.1.59 kernel 
>> version as dom0 with Xen, the screen is totally dark and the only way 
>> to access the system is remotely through ssh. Logs indicate most 
>> everything else is working, such as the wifi card so we can access it 
>> remotely via ssh and a USB optical mouse lights up when connected so 
>> USB is also working. Obviously, the disk is also working. The 
>> Chromebook is configured to boot from the device's SD card slot by 
>> turning on Chrome OS developer mode options to enable booting from the 
>> SD card slot. The mystery is that when booting the exact same 6.1.59 
>> kernel on the bare metal instead of booting it as dom0 on Xen, it 
>> boots up with full access to the screen and we can interact with the 
>> system using the X.org windows system. But booting as dom0 with Xen, 
>> the screen is totally dark and the only access we have to the system 
>> is through the network via ssh. Also, when booting the 5.4.257 kernel 
>> with KVM in hypervisor mode, the screen works and we can interact with 
>> the system through the X.org windows system. Exploring the log file,we 
>> have seen the errors below :
>>
>> Without Xen (or in bare metal):
>>
>> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for 
>> DMA mapping operations
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops 
>> 0xc0d96354)
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14450000.mixer (ops 
>> 0xc0d97554)
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 
>> 145b0000.dp-controller (ops 0xc0d97278)
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14530000.hdmi (ops 
>> 0xc0d97bd0)
>> ...
>> devuan-bunsen kernel: Console: switching to colour frame buffer device 
>> 170x48
>> devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb 
>> frame buffer device
>> devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for 
>> exynos-drm on minor 0
>>
>> In this case,the kernel is able to use the exynos-drm kernel to start 
>> the fb0 device. But with Xen we get this error with exynos-drm:
>>
>> devuan-bunsen kernel: [drm] Exynos DRM: using 14400000.fimd device for 
>> DMA mapping operations
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 14400000.fimd (ops 
>> 0xc0d96354)
>> devuan-bunsen kernel: exynos-mixer 14450000.mixer: 
>> [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks 
>> support for IOMMU
>> devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind 
>> 14450000.mixer (ops 0xc0d97554): -22
>> devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
>> devuan-bunsen kernel: exynos-dp: probe of 145b0000.dp-controller 
>> failed with error -22
>>
>> I'm trying to find for a solution and I've googled a little bit and I 
>> found this web site : 
>> https://lore.kernel.org/linux-arm-kernel/20220208171823.226211-8-krzysztof.kozlowski@xxxxxxxxxxxxx/ 
>> with your email address and I tried to ask for some help for fixing 
>> the bug. Any ideas why booting the same Linux kernel that results in a 
>> working X.org display on the bare metal instead as dom0 on Xen would 
>> cause the display to remain dark, but most other basic functions would 
>> work, such as network, disk, and USB ? thanks.
> 
> 
> Thanks for the detailed description! Good to hear that those boards are 
> still being used for various projects. I also have Snow Chromebook and 
> use it for daily tests of linux-next branch.

Adding Julien Grall and Stefano Stabellini

Hi Marek,

Thanks for responding to Mario's question. I also have been doing these
experiments with a Chromebook Snow, and I am the one who reported this
problem on the xen-users ML here:

https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00021.html

You might find that thread interesting, especially here with some additional
log messages from the exynos_drm driver (exynos_drm_dma.c, I believe):

https://lists.xenproject.org/archives/html/xen-users/2023-10/msg00032.html

This issue is also discussed some on the xen-devel ML here:

https://lists.xenproject.org/archives/html/xen-devel/2023-11/msg00003.html

> 
> Frankly speaking I have no idea what might happen wrong. There have been 
> some changes recently in the Exynos IOMMU driver related to 
> initialization, maybe your changes related to Xen enabling changed 
> somehow the order of device initialization during boot. I assume that 
> the device-tree you use for the bare metal run and Xen enabled run 
> doesn't differ in the areas describing the hardware blocks.
> 
> Please check if cherry-picking the commit 
> https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f 
> to your v6.1.59 based kernel helps anyhow.

I tried adding that fix of the exynos IOMMU initialization from
Linux > 6.2 on top of the 6.1.59 kernel I used for the original report,
but that made no difference on Xen - it still failed with the mixer lacks
support for IOMMU message and the screen is totally dark.

> 
> If not, then as a temporary workaround please disable 
> CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config 
> and check what will happen (You will lose the HDMI output, but maybe 
> this won't a big issue).

This change causes the GPU to work fairly well AFAICS. Removing the mixer
and HDMI allowed the GPU to initialize, and the display manager started
normally and enabled logging into an ordinary X11 session. Based on the log
messages I was seeing, this was an obvious thing to try. Thanks for
suggesting it.

But I have a question:

How are the mixer and HDMI devices related to the main drm device? The problem
in the exynos_drm_dma driver was that on Xen, the main drm device wanted to
use IOMMU version of dma_ops, but the mixer (and probably also the HDMI if
it wouldn't have exited first) wanted to use the Xen swiotlb version of dma_ops,
but on bare metal all three devices want to use the IOMMU version of dma_ops.

The problem obviously has something to do with the fact that Xen does not
expose the same IOMMU capability to Linux as is available when running on
the bare metal.

Cheers,

> 
> 
> Best regards