Re: [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queueresources at blk_release_queue())

From: Steffen Maier
Date: Thu Oct 13 2011 - 09:09:27 EST

Next message: Ravishankar Karkala Mallikarjunayya: "[PATCH 6/6] Staging: comedi: fix printk issue in ni_pcimio.c"
Previous message: Peter De Schrijver: "Re: [PATCH v9 0/3] Add support for tegra2 based ventana board"
Next in thread: James Bottomley: "Re: [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queueresources at blk_release_queue())"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

This fix also went into 3.0.5 via
http://git.kernel.org/?p=linux/kernel/git/stable/stable-queue.git;a=blob;f=releases/3.0.5/block-free-queue-resources-at-blk_release_queue.patch
(originated at http://marc.info/?l=linux-scsi&m=131669751909474&w=2 and http://marc.info/?l=linux-scsi&m=131669414205696&w=2)

However, it seems we still have a use-after-free bug, now causing the following oops with kernel 3.0.6 when removing paths to storage while generating load on device-mapper multipath disks:

Unable to handle kernel pointer dereference at virtual kernel address 6b6b6b6b6b6b6000
Oops: 0038 [#1] PREEMPT SMP DEBUG_PAGEALLOC
Modules linked in: iptable_filter ip_tables x_tables dm_round_robin sunrpc qeth_l3 binfmt_misc dm_multipath scsi_dh dm_mod ipv6 qeth ccwgroup [last unloaded: scsi_wait_scan]
CPU: 1 Not tainted 3.0.6-50.x.20111006-s390xdefault #1
Process blast.LzS_64_SL (pid: 26613, task: 0000000063e2a398, ksp: 0000000064de3560)
Krnl PSW : 0704100180000000 000000000048a038 (scsi_print_command+0x44/0xf8)
R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:0 CC:1 PM:0 EA:3
Krnl GPRS: 000000000000006b 6b6b6b6b6b6b6b6b 000000006717f800 000000000094f2e0
000000000061242e 000000000062bd88 0000000066fb90d8 0000000065391ad7
000000006717f800 000000006717f800 000000006716a090 000000006717f800
0000000000000004 0000000000672f88 0000000064de3838 0000000064de3808
Krnl Code: 000000000048a026: f0b80004ebbf srp 4(12,%r0),3007(%r14),8
000000000048a02c: f0a0000407f4 srp 4(11,%r0),2036,0
000000000048a032: e31020800004 lg %r1,128(%r2)
>000000000048a038: e31010b00004 lg %r1,176(%r1)
000000000048a03e: b9020011 ltgr %r1,%r1
000000000048a042: a7840032 brc 8,48a0a6
000000000048a046: e33020000004 lg %r3,0(%r2)
000000000048a04c: c04000182f4c larl %r4,78fee4
Call Trace:
([<000000006717f800>] 0x6717f800)
[<0000000000487f28>] scsi_log_send+0xf0/0x130
[<000000000048824c>] scsi_dispatch_cmd+0xc8/0x4bc
[<0000000000490694>] scsi_request_fn+0x3b8/0x480
[<00000000003fe4c8>] __elv_add_request+0x174/0x35c
[<0000000000402fd6>] blk_insert_cloned_request+0x72/0xcc
[<000003e001e134f2>] dm_dispatch_request+0x5a/0x94 [dm_mod]
[<000003e001e15f36>] dm_request_fn+0x216/0x398 [dm_mod]
[<0000000000405590>] queue_unplugged+0x60/0x1a8
[<0000000000407470>] blk_flush_plug_list+0x218/0x29c
[<00000000006136ec>] io_schedule+0x94/0x100
[<000000000020e6d6>] sleep_on_page_killable+0x22/0x60
[<00000000006140a8>] __wait_on_bit_lock+0xc0/0x124
[<000000000020e5f8>] __lock_page_killable+0x78/0x84
[<0000000000210690>] generic_file_aio_read+0x56c/0x800
[<000000000027a0a0>] do_sync_read+0xcc/0x11c
[<000000000027a974>] vfs_read+0xb0/0x194
[<000000000027aab0>] SyS_read+0x58/0xb4
[<00000000006185f6>] sysc_noemu+0x16/0x1c
[<000003fffd473bd4>] 0x3fffd473bd4
INFO: lockdep is turned off.
Last Breaking-Event-Address:
[<000000000048a020>] scsi_print_command+0x2c/0xf8

During a different run, we also got a similar oops which was not logged but only its panic_on_oops:

Kernel panic - not syncing: Fatal exception: panic_on_oops
CPU: 1 Tainted: G D 3.0.6-50.x.20111006-s390xdefault #1
Process blast.LzS_64_SL (pid: 6551, task: 00000000a2942398, ksp: 00000000a11834e0)
00000000a1183630 00000000a11835b0 0000000000000002 0000000000000000
00000000a1183650 00000000a11835c8 00000000a11835c8 000000000061242e
0000000000000000 0000000000000000 00000000a3176ed7 0000000000100ed2
000000000000000d 000000000000000c 00000000a1183618 0000000000000000
0000000000000000 0000000000100a18 00000000a11835b0 00000000a11835f0
Call Trace:
([<0000000000100926>] show_trace+0xee/0x144)
[<0000000000612258>] panic+0xb0/0x234
[<0000000000100ed2>] die+0x166/0x168
[<000000000011eede>] do_no_context+0xba/0xf8
[<000000000061ad28>] do_asce_exception+0x130/0x18c
[<0000000000618918>] pgm_exit+0x0/0x4
[<000000000048a0d4>] scsi_print_command+0xe0/0xf8
([<000000000048a0e6>] scsi_print_command+0xf2/0xf8)
[<0000000000487f28>] scsi_log_send+0xf0/0x130
[<000000000048824c>] scsi_dispatch_cmd+0xc8/0x4bc
[<0000000000490694>] scsi_request_fn+0x3b8/0x480
[<00000000003fe4c8>] __elv_add_request+0x174/0x35c
[<0000000000402fd6>] blk_insert_cloned_request+0x72/0xcc
[<000003e001e134f2>] dm_dispatch_request+0x5a/0x94 [dm_mod]
[<000003e001e15f36>] dm_request_fn+0x216/0x398 [dm_mod]
[<0000000000405590>] queue_unplugged+0x60/0x1a8
[<0000000000407470>] blk_flush_plug_list+0x218/0x29c
[<000000000040751e>] blk_finish_plug+0x2a/0x58
[<000000000021034e>] generic_file_aio_read+0x22a/0x800
[<000000000027a0a0>] do_sync_read+0xcc/0x11c
[<000000000027a974>] vfs_read+0xb0/0x194
[<000000000027aab0>] SyS_read+0x58/0xb4
[<00000000006185f6>] sysc_noemu+0x16/0x1c
[<000003fffd395bd4>] 0x3fffd395bd4
INFO: lockdep is turned off.

The other CPU was doing scsi_print_command successfully to the console (sclp_console_write) when this panic occurred.

Earlier, we tried Alan's patch
http://marc.info/?l=linux-scsi&m=130963676807728&w=2
on 3.0.4, but in contrast to Cascardo's experience
http://marc.info/?l=linux-scsi&m=131482024119345&w=2
it did not work for us.

Initially, we encountered use-after-free bugs in
scsi_print_command / scsi_dispatch_cmd
http://marc.info/?l=linux-scsi&m=130824013229933&w=2
__elv_add_request
http://marc.info/?l=linux-scsi&m=130858383702949&w=2

Also related is the following thread:
http://marc.info/?l=linux-scsi&m=130977887419062&w=2

Unfortunately, I don't understand what exactly is going wrong here.
Can you shed some light on this?

On 09/28/2011 04:14 PM, Jens Axboe wrote:

Alright, lets call this fully tested and fixed then. Linus, I committed
it, please pull:

git://git.kernel.dk/linux-block.git for-linus

Hannes Reinecke (1):
block: Free queue resources at blk_release_queue()

block/blk-core.c | 13 ++++++-------
block/blk-sysfs.c | 5 +++++
2 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/block/blk-core.c b/block/blk-core.c
index b2ed78a..d34433a 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -348,9 +348,10 @@ void blk_put_queue(struct request_queue *q)
EXPORT_SYMBOL(blk_put_queue);

/*
- * Note: If a driver supplied the queue lock, it should not zap that lock
- * unexpectedly as some queue cleanup components like elevator_exit() and
- * blk_throtl_exit() need queue lock.
+ * Note: If a driver supplied the queue lock, it is disconnected
+ * by this function. The actual state of the lock doesn't matter
+ * here as the request_queue isn't accessible after this point
+ * (QUEUE_FLAG_DEAD is set) and no other requests will be queued.
*/
void blk_cleanup_queue(struct request_queue *q)
{
@@ -367,10 +368,8 @@ void blk_cleanup_queue(struct request_queue *q)
queue_flag_set_unlocked(QUEUE_FLAG_DEAD, q);
mutex_unlock(&q->sysfs_lock);

- if (q->elevator)
- elevator_exit(q->elevator);
-
- blk_throtl_exit(q);
+ if (q->queue_lock !=&q->__queue_lock)
+ q->queue_lock =&q->__queue_lock;

blk_put_queue(q);
}
diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index e681805..60fda88 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -479,6 +479,11 @@ static void blk_release_queue(struct kobject *kobj)

blk_sync_queue(q);

+ if (q->elevator)
+ elevator_exit(q->elevator);
+
+ blk_throtl_exit(q);
+
if (rl->rq_pool)
mempool_destroy(rl->rq_pool);

--
Steffen Maier

Linux on System z Development

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Martin Jetter
GeschÃftsfÃhrung: Dirk Wittkopp
Sitz der Gesellschaft: BÃblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Ravishankar Karkala Mallikarjunayya: "[PATCH 6/6] Staging: comedi: fix printk issue in ni_pcimio.c"
Previous message: Peter De Schrijver: "Re: [PATCH v9 0/3] Add support for tegra2 based ventana board"
Next in thread: James Bottomley: "Re: [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queueresources at blk_release_queue())"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]