INFO: rcu detected stall in net_rx_action

From: Wei Chen
Date: Mon Oct 10 2022 - 11:38:43 EST


Dear Linux Developer,

Recently when using our tool to fuzz kernel, the following crash was triggered:

HEAD commit: 64570fbc14f8 Linux 5.15-rc5
git tree: upstream
compiler: clang 12.0.0
console output:
https://drive.google.com/file/d/1BOhVEmi3RPIxx-F0LMLsgflaj0r0MyKv/view?usp=sharing
kernel config: https://drive.google.com/file/d/1lNwvovjLNrcuyFGrg05IoSmgO5jaKBBJ/view?usp=sharing

Unfortunately, I don't have any reproducer for this crash yet.

rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 0-...!: (88 ticks this GP) idle=4c5/1/0x4000000000000000
softirq=42739/42739 fqs=1
(t=15633 jiffies g=62957 q=125)
rcu: rcu_preempt kthread starved for 15193 jiffies! g62957 f0x0
RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now
expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt state:R running task stack:27696 pid: 14
ppid: 2 flags:0x00004000
Call Trace:
__schedule+0xc1a/0x11e0
schedule+0x14b/0x210
schedule_timeout+0x1b4/0x310
rcu_gp_fqs_loop+0x1fd/0x770
rcu_gp_kthread+0xa5/0x340
kthread+0x419/0x510
ret_from_fork+0x1f/0x30
rcu: Stack dump where RCU GP kthread last ran:
NMI backtrace for cpu 0
CPU: 0 PID: 13 Comm: ksoftirqd/0 Not tainted 5.15.0-rc5+ #14
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.13.0-1ubuntu1.1 04/01/2014
Call Trace:
<IRQ>
dump_stack_lvl+0x1d8/0x2c4
nmi_cpu_backtrace+0x452/0x480
nmi_trigger_cpumask_backtrace+0x1a3/0x330
rcu_check_gp_kthread_starvation+0x1f9/0x270
rcu_sched_clock_irq+0x1de4/0x2bc0
update_process_times+0x1ab/0x220
tick_sched_timer+0x2a0/0x440
__hrtimer_run_queues+0x51a/0xae0
hrtimer_interrupt+0x3c9/0x1130
__sysvec_apic_timer_interrupt+0xf9/0x280
sysvec_apic_timer_interrupt+0x8c/0xb0
</IRQ>
asm_sysvec_apic_timer_interrupt+0x12/0x20
RIP: 0010:e1000_clean+0x15ad/0x40b0
Code: c5 c8 04 00 00 4c 89 eb 48 c1 eb 03 42 80 3c 23 00 74 08 4c 89
ef e8 a2 2c 65 fc 49 8b 45 00 b9 9d 00 00 00 89 88 d0 00 00 00 <42> 80
3c 23 00 74 08 4c 89 ef e8 84 2c 65 fc 49 8b 45 00 8b 40 08
RSP: 0018:ffffc90000707840 EFLAGS: 00000246
RAX: ffffc900065c0000 RBX: 1ffff1100371f229 RCX: 000000000000009d
RDX: 0000000000000100 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffffc90000707ac8 R08: ffffffff856f35c6 R09: ffffed100371f2a7
R10: ffffed100371f2a7 R11: 0000000000000000 R12: dffffc0000000000
R13: ffff88801b8f9148 R14: 0000000000004e20 R15: 1ffff920000e0f2c
__napi_poll+0xbd/0x550
net_rx_action+0x67b/0xfc0
__do_softirq+0x372/0x783
run_ksoftirqd+0xa2/0x100
smpboot_thread_fn+0x570/0xa20
kthread+0x419/0x510
ret_from_fork+0x1f/0x30
NMI backtrace for cpu 0
CPU: 0 PID: 13 Comm: ksoftirqd/0 Not tainted 5.15.0-rc5+ #14
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.13.0-1ubuntu1.1 04/01/2014
Call Trace:
<IRQ>
dump_stack_lvl+0x1d8/0x2c4
nmi_cpu_backtrace+0x452/0x480
nmi_trigger_cpumask_backtrace+0x1a3/0x330
rcu_dump_cpu_stacks+0x22d/0x390
rcu_sched_clock_irq+0x1de9/0x2bc0
update_process_times+0x1ab/0x220
tick_sched_timer+0x2a0/0x440
__hrtimer_run_queues+0x51a/0xae0
hrtimer_interrupt+0x3c9/0x1130
__sysvec_apic_timer_interrupt+0xf9/0x280
sysvec_apic_timer_interrupt+0x8c/0xb0
</IRQ>
asm_sysvec_apic_timer_interrupt+0x12/0x20
RIP: 0010:e1000_clean+0x15ad/0x40b0
Code: c5 c8 04 00 00 4c 89 eb 48 c1 eb 03 42 80 3c 23 00 74 08 4c 89
ef e8 a2 2c 65 fc 49 8b 45 00 b9 9d 00 00 00 89 88 d0 00 00 00 <42> 80
3c 23 00 74 08 4c 89 ef e8 84 2c 65 fc 49 8b 45 00 8b 40 08
RSP: 0018:ffffc90000707840 EFLAGS: 00000246
RAX: ffffc900065c0000 RBX: 1ffff1100371f229 RCX: 000000000000009d
RDX: 0000000000000100 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffffc90000707ac8 R08: ffffffff856f35c6 R09: ffffed100371f2a7
R10: ffffed100371f2a7 R11: 0000000000000000 R12: dffffc0000000000
R13: ffff88801b8f9148 R14: 0000000000004e20 R15: 1ffff920000e0f2c
__napi_poll+0xbd/0x550
net_rx_action+0x67b/0xfc0
__do_softirq+0x372/0x783
run_ksoftirqd+0xa2/0x100
smpboot_thread_fn+0x570/0xa20
kthread+0x419/0x510
ret_from_fork+0x1f/0x30
----------------
Code disassembly (best guess), 1 bytes skipped:
0: c8 04 00 00 enterq $0x4,$0x0
4: 4c 89 eb mov %r13,%rbx
7: 48 c1 eb 03 shr $0x3,%rbx
b: 42 80 3c 23 00 cmpb $0x0,(%rbx,%r12,1)
10: 74 08 je 0x1a
12: 4c 89 ef mov %r13,%rdi
15: e8 a2 2c 65 fc callq 0xfc652cbc
1a: 49 8b 45 00 mov 0x0(%r13),%rax
1e: b9 9d 00 00 00 mov $0x9d,%ecx
23: 89 88 d0 00 00 00 mov %ecx,0xd0(%rax)
* 29: 42 80 3c 23 00 cmpb $0x0,(%rbx,%r12,1) <-- trapping instruction
2e: 74 08 je 0x38
30: 4c 89 ef mov %r13,%rdi
33: e8 84 2c 65 fc callq 0xfc652cbc
38: 49 8b 45 00 mov 0x0(%r13),%rax
3c: 8b 40 08 mov 0x8(%rax),%eax

Best,
Wei