[PATCH] vfs: fix bug when opening a file with O_DIRECT on a file system that does not support it will leave an empty file

[Qinghua Jin]

Colin Ian King reported the following

1. create a minix file system and mount it
2. open a file on the file system with O_RDWR | O_CREAT | O_TRUNC | O_DIRECT
3. open fails with -EINVAL but leaves an empty file behind.  All other open() failures don't leave the
failed open files behind.

The reason is because when checking the O_DIRECT in do_dentry_open, the inode has created, and later err
processing can't remove the inode:

        /* NB: we're sure to have correct a_ops only after f_op->open */
        if (f->f_flags & O_DIRECT) {
                if (!f->f_mapping->a_ops || !f->f_mapping->a_ops->direct_IO)
                        return -EINVAL;
        }

The patch will check the O_DIRECT before creating the inode in lookup_open function.

Signed-off-by: Qinghua Jin <qhjin_dev@xxxxxxx>
Reported-by:  Colin Ian King <colin.king@xxxxxxxxxxxxx>
---
 fs/namei.c | 7 +++++++
 fs/open.c  | 6 ------
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/fs/namei.c b/fs/namei.c
index 1f9d2187c765..24c6bcba702d 100644
--- a/fs/namei.c
+++ b/fs/namei.c
@@ -3277,6 +3277,13 @@ static struct dentry *lookup_open(struct nameidata *nd, struct file *file,
 			goto out_dput;
 		}
 
+		if (open_flag & O_DIRECT) {
+			if (!dir_inode->i_mapping || !dir_inode->i_mapping->a_ops ||
+					!dir_inode->i_mapping->a_ops->direct_IO) {
+				error = -EINVAL;
+				goto out_dput;
+			}
+		}
 		error = dir_inode->i_op->create(mnt_userns, dir_inode, dentry,
 						mode, open_flag & O_EXCL);
 		if (error)
diff --git a/fs/open.c b/fs/open.c
index f732fb94600c..2829c3613c0f 100644
--- a/fs/open.c
+++ b/fs/open.c
@@ -838,12 +838,6 @@ static int do_dentry_open(struct file *f,
 
 	file_ra_state_init(&f->f_ra, f->f_mapping->host->i_mapping);
 
-	/* NB: we're sure to have correct a_ops only after f_op->open */
-	if (f->f_flags & O_DIRECT) {
-		if (!f->f_mapping->a_ops || !f->f_mapping->a_ops->direct_IO)
-			return -EINVAL;
-	}
-
 	/*
 	 * XXX: Huge page cache doesn't support writing yet. Drop all page
 	 * cache for this file before processing writes.
-- 
2.30.2