Re: WARNING: lock held when returning to user space in fuse_lock_inode

From: Miklos Szeredi
Date: Tue Jul 17 2018 - 08:46:13 EST


On Tue, Jul 17, 2018 at 1:36 PM, Dmitry Vyukov <dvyukov@xxxxxxxxxx> wrote:
> On Tue, Jul 17, 2018 at 1:14 PM, Miklos Szeredi <miklos@xxxxxxxxxx> wrote:
>> On Thu, Jul 12, 2018 at 5:49 PM, syzbot
>> <syzbot+3f7b29af1baa9d0a55be@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
>>> Hello,
>>>
>>> syzbot found the following crash on:
>>>
>>> HEAD commit: c25c74b7476e Merge tag 'trace-v4.18-rc3-2' of git://git.ke..
>>> git tree: upstream
>>> console output: https://syzkaller.appspot.com/x/log.txt?x=177bcec2400000
>>> kernel config: https://syzkaller.appspot.com/x/.config?x=25856fac4e580aa7
>>> dashboard link: https://syzkaller.appspot.com/bug?extid=3f7b29af1baa9d0a55be
>>> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
>>> syzkaller repro:https://syzkaller.appspot.com/x/repro.syz?x=13aa7678400000
>>> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=17492678400000
>>>
>>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>>> Reported-by: syzbot+3f7b29af1baa9d0a55be@xxxxxxxxxxxxxxxxxxxxxxxxx
>>>
>>> random: sshd: uninitialized urandom read (32 bytes read)
>>> random: sshd: uninitialized urandom read (32 bytes read)
>>> random: sshd: uninitialized urandom read (32 bytes read)
>>>
>>> ================================================
>>> WARNING: lock held when returning to user space!
>>> 4.18.0-rc4+ #143 Not tainted
>>> ------------------------------------------------
>>> syz-executor012/4539 is leaving the kernel with locks still held!
>>> 1 lock held by syz-executor012/4539:
>>> #0: (____ptrval____) (&fi->mutex){+.+.}, at: fuse_lock_inode+0xaf/0xe0
>>> fs/fuse/inode.c:363
>>
>> False positive.
>>
>> fi->mutex is definitely not held by the acquiring task when returning
>> to userspace. Maybe syzkaller is confused by the fact that there are
>> several interdependent tasks involved with fuse: the one calling into
>> fuse by doing something (looking up ./file0/file0) and the one that
>> reads the fuse device (returning with the LOOKUP request for "file0").
>> The second one will return with that lock held, but it's not the one
>> that acquired it, so there's no bug at all here.
>
> Hi Miklos,
>
> syzkaller is unrelated here. That's what kernel self-detects and
> prints. So either way there is something to fix in kernel here: either
> fuse or lockdep.
>
> +Alistair did some analysis offline, hope you don't mind if I repost
> your description:
> ===
> Just from reading the code, I think I can see how this happens. Fuse
> is wrapping its inode mutex with a check for "parallel_dirops", which
> is set up in process_init_reply(). The FUSE_PARALLEL_DIROPS appears to
> always be set, in fuse_send_init(), but its initial state is to be
> disabled. So if the mutex gets taken, and it'll never be unlocked if
> the initial command is flushed by fuse_readdir()'s use of
> fuse_lock_inode().
> ===

Ah, indeed. Fix attached.

Thanks,
Miklos
From: Miklos Szeredi <mszeredi@xxxxxxxxxx>
Subject: fuse: fix inital parallel dirops

If parallel dirops are enabled in FUSE_INIT reply, then first operation may
leave fi->mutex held.

Reported-by: syzbot+3f7b29af1baa9d0a55be@xxxxxxxxxxxxxxxxxxxxxxxxx
Signed-off-by: Miklos Szeredi <mszeredi@xxxxxxxxxx>
---
fs/fuse/dir.c | 10 ++++++----
fs/fuse/fuse_i.h | 4 ++--
fs/fuse/inode.c | 14 ++++++++++----
3 files changed, 18 insertions(+), 10 deletions(-)

--- a/fs/fuse/dir.c
+++ b/fs/fuse/dir.c
@@ -355,11 +355,12 @@ static struct dentry *fuse_lookup(struct
struct inode *inode;
struct dentry *newent;
bool outarg_valid = true;
+ bool locked;

- fuse_lock_inode(dir);
+ locked = fuse_lock_inode(dir);
err = fuse_lookup_name(dir->i_sb, get_node_id(dir), &entry->d_name,
&outarg, &inode);
- fuse_unlock_inode(dir);
+ fuse_unlock_inode(dir, locked);
if (err == -ENOENT) {
outarg_valid = false;
err = 0;
@@ -1340,6 +1341,7 @@ static int fuse_readdir(struct file *fil
struct fuse_conn *fc = get_fuse_conn(inode);
struct fuse_req *req;
u64 attr_version = 0;
+ bool locked;

if (is_bad_inode(inode))
return -EIO;
@@ -1367,9 +1369,9 @@ static int fuse_readdir(struct file *fil
fuse_read_fill(req, file, ctx->pos, PAGE_SIZE,
FUSE_READDIR);
}
- fuse_lock_inode(inode);
+ locked = fuse_lock_inode(inode);
fuse_request_send(fc, req);
- fuse_unlock_inode(inode);
+ fuse_unlock_inode(inode, locked);
nbytes = req->out.args[0].size;
err = req->out.h.error;
fuse_put_request(fc, req);
--- a/fs/fuse/fuse_i.h
+++ b/fs/fuse/fuse_i.h
@@ -974,8 +974,8 @@ int fuse_do_setattr(struct dentry *dentr

void fuse_set_initialized(struct fuse_conn *fc);

-void fuse_unlock_inode(struct inode *inode);
-void fuse_lock_inode(struct inode *inode);
+void fuse_unlock_inode(struct inode *inode, bool locked);
+bool fuse_lock_inode(struct inode *inode);

int fuse_setxattr(struct inode *inode, const char *name, const void *value,
size_t size, int flags);
--- a/fs/fuse/inode.c
+++ b/fs/fuse/inode.c
@@ -357,15 +357,21 @@ int fuse_reverse_inval_inode(struct supe
return 0;
}

-void fuse_lock_inode(struct inode *inode)
+bool fuse_lock_inode(struct inode *inode)
{
- if (!get_fuse_conn(inode)->parallel_dirops)
+ bool locked = false;
+
+ if (!get_fuse_conn(inode)->parallel_dirops) {
mutex_lock(&get_fuse_inode(inode)->mutex);
+ locked = true;
+ }
+
+ return locked;
}

-void fuse_unlock_inode(struct inode *inode)
+void fuse_unlock_inode(struct inode *inode, bool locked)
{
- if (!get_fuse_conn(inode)->parallel_dirops)
+ if (locked)
mutex_unlock(&get_fuse_inode(inode)->mutex);
}