Re: [tip: timers/core] hrtimer: Annotate lockless access to timer->state

[Paul E. McKenney]

On Thu, Nov 07, 2019 at 07:48:50AM -0800, Eric Dumazet wrote:
> On Thu, Nov 7, 2019 at 12:53 AM Paul E. McKenney <paulmck@xxxxxxxxxx> wrote:
> >
> > On Wed, Nov 06, 2019 at 02:59:36PM -0800, Eric Dumazet wrote:
> > > On Wed, Nov 6, 2019 at 2:53 PM Eric Dumazet <edumazet@xxxxxxxxxx> wrote:
> > > >
> > > > On Wed, Nov 6, 2019 at 2:24 PM tip-bot2 for Eric Dumazet
> > > > <tip-bot2@xxxxxxxxxxxxx> wrote:
> > > > >
> > > > > The following commit has been merged into the timers/core branch of tip:
> > > > >
> > > > > Commit-ID:     56144737e67329c9aaed15f942d46a6302e2e3d8
> > > > > Gitweb:        https://git.kernel.org/tip/56144737e67329c9aaed15f942d46a6302e2e3d8
> > > > > Author:        Eric Dumazet <edumazet@xxxxxxxxxx>
> > > > > AuthorDate:    Wed, 06 Nov 2019 09:48:04 -08:00
> > > > > Committer:     Thomas Gleixner <tglx@xxxxxxxxxxxxx>
> > > > > CommitterDate: Wed, 06 Nov 2019 23:18:31 +01:00
> > > > >
> > > > > hrtimer: Annotate lockless access to timer->state
> > > > >
> > > >
> > > > I guess we also need to fix timer_pending(), since timer->entry.pprev
> > > > could change while we read it.
> > >
> > > It is interesting seeing hlist_add_head() has a WRITE_ONCE(h->first, n);,
> > > but no WRITE_ONCE() for the pprev change.
> > >
> > > The WRITE_ONCE() was added in commit 1c97be677f72b3c338312aecd36d8fff20322f32
> > > ("list: Use WRITE_ONCE() when adding to lists and hlists")
> >
> > The theory is that while the ->next pointer is concurrently accessed by
> > RCU readers, the ->pprev pointer is accessed only by updaters, who need
> > to supply sufficient synchronization.
> >
> > But what is this theory missing in practice?
> 
> Here is some context : I am helping triaging about 400 KCSAN data-race
> splats in syzbot moderation queue.
> 
> Take a look at the timer related one in [1]
> 
> If we want to avoid potential load/store-tearing, minimall patch would be :
> 
> diff --git a/include/linux/list.h b/include/linux/list.h
> index 85c92555e31f85f019354e54d6efb8e79c2aee17..9139803b851cc37bb759c8d7c12ee7e36c61f009
> 100644
> --- a/include/linux/list.h
> +++ b/include/linux/list.h
> @@ -761,7 +761,7 @@ static inline void __hlist_del(struct hlist_node *n)
> 
>         WRITE_ONCE(*pprev, next);
>         if (next)
> -               next->pprev = pprev;
> +               WRITE_ONCE(next->pprev, pprev);
>  }
> 
>  static inline void hlist_del(struct hlist_node *n)
> diff --git a/include/linux/timer.h b/include/linux/timer.h
> index 1e6650ed066d5d28251b0bd385fc37ef94c96532..c7c8dd89f2797389ca96473e60c7297fd38d8259
> 100644
> --- a/include/linux/timer.h
> +++ b/include/linux/timer.h
> @@ -164,7 +164,7 @@ static inline void destroy_timer_on_stack(struct
> timer_list *timer) { }
>   */
>  static inline int timer_pending(const struct timer_list * timer)
>  {
> -       return timer->entry.pprev != NULL;
> +       return READ_ONCE(timer->entry.pprev) != NULL;
>  }
> 
>  extern void add_timer_on(struct timer_list *timer, int cpu);
> 
> But really many other WRITE_ONCE() would be needed in include/linux/list.h
> 
> [1]
> 
> BUG: KCSAN: data-race in del_timer / detach_if_pending
> 
> write to 0xffff88808697d870 of 8 bytes by task 10 on cpu 0:
>  __hlist_del include/linux/list.h:764 [inline]
>  detach_timer kernel/time/timer.c:815 [inline]
>  detach_if_pending+0xcd/0x2d0 kernel/time/timer.c:832
>  try_to_del_timer_sync+0x60/0xb0 kernel/time/timer.c:1226
>  del_timer_sync+0x6b/0xa0 kernel/time/timer.c:1365
>  schedule_timeout+0x2d2/0x6e0 kernel/time/timer.c:1896
>  rcu_gp_fqs_loop+0x37c/0x580 kernel/rcu/tree.c:1639
>  rcu_gp_kthread+0x143/0x230 kernel/rcu/tree.c:1799
>  kthread+0x1d4/0x200 drivers/block/aoe/aoecmd.c:1253
>  ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:352
> 
> read to 0xffff88808697d870 of 8 bytes by task 12060 on cpu 1:
>  del_timer+0x3b/0xb0 kernel/time/timer.c:1198
>  sk_stop_timer+0x25/0x60 net/core/sock.c:2845
>  inet_csk_clear_xmit_timers+0x69/0xa0 net/ipv4/inet_connection_sock.c:523
>  tcp_clear_xmit_timers include/net/tcp.h:606 [inline]
>  tcp_v4_destroy_sock+0xa3/0x3f0 net/ipv4/tcp_ipv4.c:2096
>  inet_csk_destroy_sock+0xf4/0x250 net/ipv4/inet_connection_sock.c:836
>  tcp_close+0x6f3/0x970 net/ipv4/tcp.c:2497
>  inet_release+0x86/0x100 net/ipv4/af_inet.c:427
>  __sock_release+0x85/0x160 net/socket.c:590
>  sock_close+0x24/0x30 net/socket.c:1268
>  __fput+0x1e1/0x520 fs/file_table.c:280
>  ____fput+0x1f/0x30 fs/file_table.c:313
>  task_work_run+0xf6/0x130 kernel/task_work.c:113
>  tracehook_notify_resume include/linux/tracehook.h:188 [inline]
>  exit_to_usermode_loop+0x2b4/0x2c0 arch/x86/entry/common.c:163
> 
> Reported by Kernel Concurrency Sanitizer on:
> CPU: 1 PID: 12060 Comm: syz-executor.5 Not tainted 5.4.0-rc3+ #0
> Hardware name: Google Google Compute Engine/Google Compute Engine,
> BIOS Google 01/01/2011
> ==================================================================

OK, so this is due to timer_pending() lockless access to ->entry.pprev
to determine whether or not the timer is on the list.  New one on me!

Given that use case, I don't have an objection to your patch to list.h.

Except...

Would it make sense to add a READ_ONCE() to hlist_unhashed()
and to then make timer_pending() invoke hlist_unhashed()?  That
would better confine the needed uses of READ_ONCE().

							Thanx, Paul