Re: LTP test related to virtio releasing and reassigning resource leads to guest hung

From: Stefan Hajnoczi
Date: Tue Aug 15 2023 - 11:20:03 EST


On Thu, 10 Aug 2023 at 22:27, longguang.yue <kvmluck@xxxxxxx> wrote:
>
>
> 一)
> Can you post the guest kernel messages (dmesg)? If the guest is hanging
> then it may be easiest to configure a serial console so the kernel
> messages are sent to the host where you can see them.
>
> Does the hang occur during the LTP code you linked or afterwards when
> the PCI device is bound to a virtio driver?
>
>
> > I used conosle, the hang occurred afterwards. dmesg shows that tpci test is finished without error.
> LTP test case: https://github.com/linux-test-project/ltp/blob/522d7fba4afc84e07b252aa4cd91b241e81d6613/testcases/kernel/device-drivers/pci/tpci_kernel/ltp_tpci.c#L428
> kernel 5.10, qemu 6.2
>
> different guest-configuration tests show different results. guest did not crash if hung-task-panic=0, in my case i enable hung-task-panic in order to trace.
>
> test case 1:
> xml machine pc,virtio disk, virtio net —— guest's io hung, network broke down, though console is avilable but io operation hung.
>
> #ps -aux| grep D
> root 7 0.0 0.0 0 0 ? D 14:37 0:00 [kworker/u16:0+flush-253:0]
> root 483 0.0 0.0 0 0 ? D 14:37 0:00 [jbd2/vda3-8]
>
> test case 2:
> xml machine q35,virtio/q35,scsi ——disk did not hung but network broke down. ping errors though everything looks ok and no crash and no kernel error
>
>
>
> 二)
> I didn't see your original email so I missed the panic. I'd still like
> to see the earlier kernel messages before the panic in order to
> understand how the PCI device is bound.
>
> Is the vda device with hung I/O the same device that was accessed by
> the LTP test earlier? I guess the LTP test runs against the device and
> then the virtio driver binds to the device again afterwards?
>
> > the test is
> ```
> // iterate all devices
> ……
> for (i = 0; i < 7; ++i) { // iterate current device's resources
> if (r->flags & IORESOURCE_MEM &&
> r->flags & IORESOURCE_PREFETCH) {
> pci_release_resource(dev, i);
> ret = pci_assign_resource(dev, i);
> prk_info("assign resource to '%d', ret '%d'", i, ret);
> rc |= (ret < 0 && ret != -EBUSY) ? TFAIL : TPASS;
> }
> }
> ```
> test does not do virtio device unbind and bind.
> I only notice mem resource changed. see 'test-case 12'
>
> ———————————
> [ 88.905705] ltp_tpci: test-case 12
> [ 88.905706] ltp_tpci: assign resources
> [ 88.905706] ltp_tpci: assign resource #0
> [ 88.905707] ltp_tpci: name = 0000:00:07.0, flags = 262401, start 0xc080, end 0xc0ff
> [ 88.905707] ltp_tpci: assign resource #1
> [ 88.905708] ltp_tpci: name = 0000:00:07.0, flags = 262656, start 0xfebd4000, end 0xfebd4fff
> [ 88.905709] ltp_tpci: assign resource #2
> [ 88.905709] ltp_tpci: name = 0000:00:07.0, flags = 0, start 0x0, end 0x0
> [ 88.905710] ltp_tpci: assign resource #3
> [ 88.905710] ltp_tpci: name = 0000:00:07.0, flags = 0, start 0x0, end 0x0
> [ 88.905711] ltp_tpci: assign resource #4
> [ 88.905711] ltp_tpci: name = 0000:00:07.0, flags = 1319436, start 0xfe00c000, end 0xfe00ffff
> [ 88.905713] virtio-pci 0000:00:07.0: BAR 4: releasing [mem 0xfe00c000-0xfe00ffff 64bit pref]
> [ 88.905715] virtio-pci 0000:00:07.0: BAR 4: assigned [mem 0x24000c000-0x24000ffff 64bit pref]
> [ 88.906693] ltp_tpci: assign resource to '4', ret '0'
> [ 88.906694] ltp_tpci: assign resource #5
> [ 88.906694] ltp_tpci: name = (null), flags = 0, start 0x0, end 0x0
> [ 88.906695] ltp_tpci: assign resource #6
> [ 88.906695] ltp_tpci: name = 0000:00:07.0, flags = 0, start 0x0, end 0x0
>
> [ 88.906800] ltp_tpci: test-case 13

I don't know. Maybe the test case is leaving the device is a state
that conflicts with the virtio drivers that are bound after testing
finishes.

One approach is to trace the PCI BAR accesses after the test runs and
compare against a trace when the tpci driver hasn't been loaded. That
way you might be able to find out what is different.

Stefan