Re: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle

[Bart Van Assche]

On 01/17/18 18:41, Ming Lei wrote:
> BLK_STS_RESOURCE can be returned from driver when any resource
> is running out of. And the resource may not be related with tags,
> such as kmalloc(GFP_ATOMIC), when queue is idle under this kind of
> BLK_STS_RESOURCE, restart can't work any more, then IO hang may
> be caused.
>
> Most of drivers may call kmalloc(GFP_ATOMIC) in IO path, and almost
> all returns BLK_STS_RESOURCE under this situation. But for dm-mpath,
> it may be triggered a bit easier since the request pool of underlying
> queue may be consumed up much easier. But in reality, it is still not
> easy to trigger it. I run all kinds of test on dm-mpath/scsi-debug
> with all kinds of scsi_debug parameters, can't trigger this issue
> at all. But finally it is triggered in Bart's SRP test, which seems
> made by genius, :-)
>
> [ ... ]
>
>   static void blk_mq_timeout_work(struct work_struct *work)
>   {
>   	struct request_queue *q =
> @@ -966,8 +1045,10 @@ static void blk_mq_timeout_work(struct work_struct *work)
>   		 */
>   		queue_for_each_hw_ctx(q, hctx, i) {
>   			/* the hctx may be unmapped, so check it here */
> -			if (blk_mq_hw_queue_mapped(hctx))
> +			if (blk_mq_hw_queue_mapped(hctx)) {
>   				blk_mq_tag_idle(hctx);
> +				blk_mq_fixup_restart(hctx);
> +			}
>   		}
>   	}
>   	blk_queue_exit(q);

Hello Ming,

My comments about the above are as follows:
- It can take up to q->rq_timeout jiffies after a .queue_rq()
  implementation returned BLK_STS_RESOURCE before blk_mq_timeout_work()
  gets called. However, it can happen that only a few milliseconds after
  .queue_rq() returned BLK_STS_RESOURCE that the condition that caused
  it to return BLK_STS_RESOURCE gets cleared. So the above approach can
  result in long delays during which it will seem like the queue got
  stuck. Additionally, I think that the block driver should decide how
  long it takes before a queue is rerun and not the block layer core.
- The lockup that I reported only occurs with the dm driver but not any
  other block driver. So why to modify the block layer core since this
  can be fixed by modifying the dm driver?
- A much simpler fix and a fix that is known to work exists, namely
  inserting a blk_mq_delay_run_hw_queue() call in the dm driver.

Bart.