Question about btrfs no space left error and forced readonly

From: ChenXiaoSong
Date: Wed Jan 17 2024 - 02:54:04 EST

Greetings,

We had a btrfs no space left error and forced readonly, the stack trace is [1]. This happened on our kernel release version based on lts 4.19.

Is there already a fix patch for this issue?

Or can anybody please advise on how to debug this further?

Thanks,
ChenXiaoSong.

[1]
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328533] ------------[ cut here ]------------
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328594] WARNING: CPU: 42 PID: 460094 at fs/btrfs/extent-tree.c:6805 __btrfs_free_extent.isra.71+0x65c/0xbd0 [btrfs]
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328595] Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache fuse xt_nat veth nf_conntrack_netlink xt_conntrack br_netfilter bridge sch_htb xt_addrtype xt_set ipt_MASQUERADE xt_mark ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipportip ip_set_hash_ipport dummy xt_comment iptable_nat nf_nat_ipv4 nf_nat iptable_filter nls_utf8 isofs bonding 8021q garp mrp stp llc ip_set nfnetlink ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 sunrpc amd64_edac_mod edac_mce_amd btrfs xor zstd_decompress zstd_compress raid6_pq libcrc32c kvm_amd ccp kvm irqbypass ipmi_ssif ast ttm drm_kms_helper syscopyarea sysfillrect igb sysimgblt i2c_algo_bit fb_sys_fops pcspkr txgbe sg dca drm i2c_piix4 k10temp ipmi_si ipmi_devintf ipmi_msghandler binfmt_misc
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328634] ip_tables ext4 mbcache jbd2 sd_mod crc32c_intel ahci libahci megaraid_sas libata dm_mirror dm_region_hash dm_log dm_mod
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328647] CPU: 42 PID: 460094 Comm: httpWorkerThrea Kdump: loaded Tainted: G W 4.19.90-17.ky10.x86_64 #1
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328648] Hardware name: Sugon H620-G30/65N32-US, BIOS 0SSSX245 06/07/2020
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328656] RIP: 0010:__btrfs_free_extent.isra.71+0x65c/0xbd0 [btrfs]
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328659] Code: 44 89 e9 ba c8 1a 00 00 48 c7 c6 a0 aa 06 c1 e8 fc ba 09 00 e9 fe fa ff ff 0f 0b 44 89 ee 48 c7 c7 68 7c 07 c1 e8 14 e9 2d e5 <0f> 0b e9 ce fa ff ff 41 83 f8 29 0f 86 d6 01 00 00 48 8b 7c 24 08
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328660] RSP: 0018:ffffae7f9f6e78c8 EFLAGS: 00010286
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328661] RAX: 0000000000000000 RBX: 000000000000a0c9 RCX: 0000000000000006
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328662] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff90a4ffa96850
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328663] RBP: 00000470931db000 R08: 000000000210773e R09: 0000000000000004
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328664] R10: 00000000ffffffe4 R11: 0000000000000001 R12: ffff909b302e2a80
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328664] R13: 00000000ffffffe4 R14: 0000000000000000 R15: 000000000003fc62
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328666] FS: 00007f4aedae2700(0000) GS:ffff90a4ffa80000(0000) knlGS:0000000000000000
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328667] CR2: 00007efce4073ff0 CR3: 0000006413d3c000 CR4: 00000000003406e0
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328668] Call Trace:
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328682] __btrfs_run_delayed_refs+0x4f8/0x1150 [btrfs]
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328707] btrfs_run_delayed_refs+0xe7/0x1b0 [btrfs]
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328717] btrfs_truncate_inode_items+0xa56/0xeb0 [btrfs]
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328728] btrfs_evict_inode+0x496/0x4f0 [btrfs]
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328733] evict+0xd2/0x1a0
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328736] __dentry_kill+0xdd/0x180
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328738] dentry_kill+0x4d/0x260
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328740] dput+0x183/0x200
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328743] __fput+0x118/0x1f0
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328747] task_work_run+0x8a/0xb0
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328750] do_exit+0x2ec/0xbf0
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328753] do_group_exit+0x3a/0xa0
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328755] get_signal+0x13f/0x7a0
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328758] do_signal+0x36/0x610
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328768] exit_to_usermode_loop+0x71/0xe0
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328770] do_syscall_64+0x1a3/0x1d0
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328773] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328779] RIP: 0033:0x7f4b12fae10c
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328785] Code: Bad RIP value.
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328785] RSP: 002b:00007f4aedae1550 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328786] RAX: fffffffffffffe00 RBX: 00007f4b0d1914e0 RCX: 00007f4b12fae10c
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328787] RDX: 0000000000000000 RSI: 0000000000000080 RDI: 00007f4b0d19150c
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328787] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000794039ff8
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328788] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f4b0d1914b8
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328789] R13: 0000000000008223 R14: 0000000000000000 R15: 00007f4b0d19150c
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328790] ---[ end trace 74a7f2461a0f9473 ]---
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.328961] BTRFS: error (device dm-9) in __btrfs_free_extent:6805: errno=-28 No space left
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.439057] BTRFS info (device dm-9): forced readonly
Dec 8 15:25:17 pjdhcpaasnap8pn kernel: [13439425.439067] BTRFS: error (device dm-9) in btrfs_run_delayed_refs:2935: errno=-28 No space left