Re: get/put_cpu() usage in block/blk-mq.c

From: Mike Galbraith
Date: Tue May 09 2017 - 11:41:10 EST


On Tue, 2017-05-09 at 09:15 -0600, Jens Axboe wrote:
> On 05/09/2017 09:13 AM, Jens Axboe wrote:
> > On 05/09/2017 09:04 AM, Mike Galbraith wrote:
> > > On Tue, 2017-05-09 at 08:53 -0600, Jens Axboe wrote:
> > > > On 05/09/2017 12:07 AM, Mike Galbraith wrote:
> > > > > Hi Jens,
> > > > >
> > > > > I was about to fix up this splat..
> > > > >
> > > > > [ 445.022141] loop: module loaded
> > > > > [ 445.078116] BUG: using smp_processor_id() in preemptible
> > > > > [00000000] code: loop0/3801
> > > > > [ 445.085873] caller is debug_smp_processor_id+0x17/0x20
> > > > > [ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: G
> > > > > E 4.12.0-default #40
> > > > > [ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]
> > > > > -/69Y5698 , BIOS -[D6E150AUS-1.10]- 12/15/2010
> > > > > [ 445.108564] Call Trace:
> > > > > [ 445.111016] dump_stack+0x65/0x89
> > > > > [ 445.114330] check_preemption_disabled+0xde/0xf0
> > > > > [ 445.118945] debug_smp_processor_id+0x17/0x20
> > > > > [ 445.123300] blk_stat_add+0xb0/0x130
> > > > > [ 445.126876] __blk_mq_complete_request+0xb5/0x150
> > > > > [ 445.131575] blk_mq_complete_request+0x16/0x20
> > > > > [ 445.136020] loop_queue_work+0x5f/0xaa0 [loop]
> > > > > [ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40
> > > > > [ 445.144816] ? finish_task_switch+0x85/0x270
> > > > > [ 445.149085] ? __schedule+0x291/0x8c0
> > > > > [ 445.152747] kthread_worker_fn+0xc2/0x1d0
> > > > > [ 445.156754] kthread+0x114/0x150
> > > > > [ 445.159983] ? __kthread_init_worker+0xb0/0xb0
> > > > > [ 445.164424] ? kthread_park+0x60/0x60
> > > > > [ 445.168085] ret_from_fork+0x2c/0x40
> > > >
> > > > Is it from this_cpu_ptr() in blk_stat_add()?
> > >
> > > Yeah.
> >
> > So why is this complaining, doesn't rcu_read_lock() disable
> > preemption?
>
> Ah, I guess it doesn't if PREEMPT_RCU is set. How about the below?

Should do it. I was about to run LTP (where it turned up) again
anyway, I'll add this. No news is good news, what you should hear.

> diff --git a/block/blk-stat.c b/block/blk-stat.c
> index 6c2f40940439..022f5925a427 100644
> --- a/block/blk-stat.c
> +++ b/block/blk-stat.c
> @@ -96,13 +96,16 @@ void blk_stat_add(struct request *rq)
>
> rcu_read_lock();
> list_for_each_entry_rcu(cb, &q->stats->callbacks, list) {
> - if (blk_stat_is_active(cb)) {
> - bucket = cb->bucket_fn(rq);
> - if (bucket < 0)
> - continue;
> - stat = &this_cpu_ptr(cb->cpu_stat)[bucket];
> - __blk_stat_add(stat, value);
> - }
> + if (!blk_stat_is_active(cb))
> + continue;
> +
> + bucket = cb->bucket_fn(rq);
> + if (bucket < 0)
> + continue;
> +
> + stat = &get_cpu_var(cb->cpu_stat)[bucket];
> + __blk_stat_add(stat, value);
> + put_cpu_var(cb->cpu_stat);
> }
> rcu_read_unlock();
> }
>