[PATCH v2] vfs: fix bug when opening a file with O_DIRECT on a file system that does not support it will leave an empty file

[Qinghua Jin]

Colin Ian King reported the following

1. create a minix file system and mount it
2. open a file on the file system with O_RDWR | O_CREAT | O_TRUNC | O_DIRECT
3. open fails with -EINVAL but leaves an empty file behind.  All other open() failures don't leave the
failed open files behind.

The reason is because when checking the O_DIRECT in do_dentry_open, the inode has created, and later err
processing can't remove the inode.

The patch will remove the file in last step of open in do_open().

Signed-off-by: Qinghua Jin <qhjin.dev@xxxxxxxxx>
Reported-by:  Colin Ian King <colin.king@xxxxxxxxxxxxx>
---
 fs/namei.c | 6 ++++++
 fs/open.c  | 6 ------
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/fs/namei.c b/fs/namei.c
index 1f9d2187c765..081feb804154 100644
--- a/fs/namei.c
+++ b/fs/namei.c
@@ -3426,6 +3426,12 @@ static int do_open(struct nameidata *nd,
 		error = vfs_open(&nd->path, file);
 	if (!error)
 		error = ima_file_check(file, op->acc_mode);
+	if (!error && (file->f_flags & O_DIRECT)) {
+		if (!file->f_mapping->a_ops || !file->f_mapping->a_ops->direct_IO) {
+			do_unlinkat(AT_FDCWD, getname_kernel(nd->name->name));
+			return -EINVAL;
+		}
+	}
 	if (!error && do_truncate)
 		error = handle_truncate(mnt_userns, file);
 	if (unlikely(error > 0)) {
diff --git a/fs/open.c b/fs/open.c
index f732fb94600c..2829c3613c0f 100644
--- a/fs/open.c
+++ b/fs/open.c
@@ -838,12 +838,6 @@ static int do_dentry_open(struct file *f,
 
 	file_ra_state_init(&f->f_ra, f->f_mapping->host->i_mapping);
 
-	/* NB: we're sure to have correct a_ops only after f_op->open */
-	if (f->f_flags & O_DIRECT) {
-		if (!f->f_mapping->a_ops || !f->f_mapping->a_ops->direct_IO)
-			return -EINVAL;
-	}
-
 	/*
 	 * XXX: Huge page cache doesn't support writing yet. Drop all page
 	 * cache for this file before processing writes.
-- 
2.30.2