[PATCH] do_open(): Fix O_DIRECTORY | O_CREAT behavior

[Pedro Falcato]

On Linux, open(O_DIRECTORY | O_CREAT) has historically meant "open
directory or create a regular file". This has remained mostly true,
except open(O_DIR | O_CREAT) has started returning an error *while
creating the file*. Restore the old behavior.

Signed-off-by: Pedro Falcato <pedro.falcato@xxxxxxxxx>
---
I did not explicitly add a Fixes: tag because I was unable to bisect this locally,
but it seems to me that this was introduced in the path walking refactoring done in early 2020.
Al, if you have a rough idea of what may have added this bug, feel free to add a Fixes.

This should also probably get CC'd to stable, but I'll leave this to your criteria.
 fs/namei.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/fs/namei.c b/fs/namei.c
index edfedfbccae..7b26db2f0f8 100644
--- a/fs/namei.c
+++ b/fs/namei.c
@@ -3540,8 +3540,18 @@ static int do_open(struct nameidata *nd,
 		if (unlikely(error))
 			return error;
 	}
-	if ((nd->flags & LOOKUP_DIRECTORY) && !d_can_lookup(nd->path.dentry))
-		return -ENOTDIR;
+
+	if ((open_flag & (O_DIRECTORY | O_CREAT)) != (O_DIRECTORY | O_CREAT) ||
+	    !(file->f_mode & FMODE_CREATED)) {
+		/* O_DIRECTORY | O_CREAT has the strange property of being the
+		 * only open(O_DIRECTORY) lookup that can create and return a
+		 * regular file *if we indeed did create*. Because of this,
+		 * only return -ENOTDIR if we're not O_DIR | O_CREAT or if we
+		 * did not create a file.
+		 */
+		if ((nd->flags & LOOKUP_DIRECTORY) && !d_can_lookup(nd->path.dentry))
+			return -ENOTDIR;
+	}
 
 	do_truncate = false;
 	acc_mode = op->acc_mode;
-- 
2.39.2