[PATCH] tty-fix-mismatch-in-lookup-han

[Alan Cox]

The lookup methods return NULL or a pointer. The caller expects an ERR_PTR.
It seems to make sense to just use NULL and return -ENODEV here as it keeps
the code simple and clean and the way it always used to work (effectively)
before the restructure

Problem noticed by Jiri Slaby.

Signed-off-by: Alan Cox <alan@xxxxxxxxxxxxxxxxxxx>
---
 drivers/char/tty_io.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/char/tty_io.c b/drivers/char/tty_io.c
index bc84e12..0cc0c60 100644
--- a/drivers/char/tty_io.c
+++ b/drivers/char/tty_io.c
@@ -1817,9 +1817,9 @@ got_driver:
 		/* check whether we're reopening an existing tty */
 		tty = tty_driver_lookup_tty(driver, inode, index);
 
-		if (IS_ERR(tty)) {
+		if (tty == NULL) {
 			mutex_unlock(&tty_mutex);
-			return PTR_ERR(tty);
+			return -ENODEV;
 		}
 	}
 
-- 

It seems that fixing IS_ERR() check uncovered some underlying problem
since we are later checking for tty == NULL in __tty_open().

The following patch fixes the issue for me (Alan, feel free to merge
it back into your tty-fix-mismatch-in-lookup-han patch or replace by
a more complete __tty_open() reorganization):

From: Bartlomiej Zolnierkiewicz <bzolnier@xxxxxxxxx>
Subject: [PATCH] tty: fix __tty_open()

On tty_driver_lookup_tty() failure __tty_open() should continue
instead of returning an error, fix it.

Also update tty_driver_lookup_tty() documentation while at it.

Signed-off-by: Bartlomiej Zolnierkiewicz <bzolnier@xxxxxxxxx>
---
 drivers/char/tty_io.c |   10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)

Index: b/drivers/char/tty_io.c
===================================================================
--- a/drivers/char/tty_io.c
+++ b/drivers/char/tty_io.c
@@ -1205,7 +1205,7 @@ static void tty_line_name(struct tty_dri
  *	@driver: the driver for the tty
  *	@idx:	 the minor number
  *
- *	Return the tty, if found or ERR_PTR() otherwise.
+ *	Return the tty, if found or NULL otherwise.
  *
  *	Locking: tty_mutex must be held. If tty is found, the mutex must
  *	be held until the 'fast-open' is also done. Will change once we
@@ -1813,16 +1813,10 @@ retry_open:
 		return -ENODEV;
 	}
 got_driver:
-	if (!tty) {
+	if (!tty)
 		/* check whether we're reopening an existing tty */
 		tty = tty_driver_lookup_tty(driver, inode, index);
 
-		if (tty == NULL) {
-			mutex_unlock(&tty_mutex);
-			return -ENODEV;
-		}
-	}
-
 	if (tty) {
 		retval = tty_reopen(tty);
 		if (retval)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/