[patch 2/3] scheduler: add full memory barriers upon task switch at runqueue lock/unlock

From: Mathieu Desnoyers
Date: Sun Jan 31 2010 - 16:12:52 EST


Depends on:
"Create spin lock/spin unlock with distinct memory barrier"

A full memory barrier is wanted before and after runqueue data structure
modifications so these can be read safely by sys_membarrier without holding the
rq lock.

Adds no overhead on x86, because LOCK-prefixed atomic operations of the spin
lock/unlock already imply a full memory barrier. Combines the spin lock
acquire/release barriers with the full memory barrier to diminish the
performance impact on other architectures. (per-architecture spinlock-mb.h
should be gradually implemented to replace the generic version)

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxx>
CC: KOSAKI Motohiro <kosaki.motohiro@xxxxxxxxxxxxxx>
CC: Steven Rostedt <rostedt@xxxxxxxxxxx>
CC: "Paul E. McKenney" <paulmck@xxxxxxxxxxxxxxxxxx>
CC: Nicholas Miell <nmiell@xxxxxxxxxxx>
CC: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
CC: mingo@xxxxxxx
CC: laijs@xxxxxxxxxxxxxx
CC: dipankar@xxxxxxxxxx
CC: akpm@xxxxxxxxxxxxxxxxxxxx
CC: josh@xxxxxxxxxxxxxxxx
CC: dvhltc@xxxxxxxxxx
CC: niv@xxxxxxxxxx
CC: tglx@xxxxxxxxxxxxx
CC: peterz@xxxxxxxxxxxxx
CC: Valdis.Kletnieks@xxxxxx
CC: dhowells@xxxxxxxxxx
---
kernel/sched.c | 24 ++++++++++++++++++++----
1 file changed, 20 insertions(+), 4 deletions(-)

Index: linux-2.6-lttng/kernel/sched.c
===================================================================
--- linux-2.6-lttng.orig/kernel/sched.c 2010-01-31 14:59:42.000000000 -0500
+++ linux-2.6-lttng/kernel/sched.c 2010-01-31 15:09:51.000000000 -0500
@@ -893,7 +893,12 @@ static inline void finish_lock_switch(st
*/
spin_acquire(&rq->lock.dep_map, 0, 0, _THIS_IP_);

- raw_spin_unlock_irq(&rq->lock);
+ /*
+ * Order mm_cpumask and rq->curr updates before following memory
+ * accesses. Required by sys_membarrier().
+ */
+ smp_mb__before_spin_unlock();
+ raw_spin_unlock_irq__no_release(&rq->lock);
}

#else /* __ARCH_WANT_UNLOCKED_CTXSW */
@@ -916,10 +921,15 @@ static inline void prepare_lock_switch(s
*/
next->oncpu = 1;
#endif
+ /*
+ * Order mm_cpumask and rq->curr updates before following memory
+ * accesses. Required by sys_membarrier().
+ */
+ smp_mb__before_spin_unlock();
#ifdef __ARCH_WANT_INTERRUPTS_ON_CTXSW
- raw_spin_unlock_irq(&rq->lock);
+ raw_spin_unlock_irq__no_release(&rq->lock);
#else
- raw_spin_unlock(&rq->lock);
+ raw_spin_unlock__no_release(&rq->lock);
#endif
}

@@ -5490,7 +5500,13 @@ need_resched_nonpreemptible:
if (sched_feat(HRTICK))
hrtick_clear(rq);

- raw_spin_lock_irq(&rq->lock);
+ raw_spin_lock_irq__no_acquire(&rq->lock);
+ /*
+ * Order memory accesses before mm_cpumask and rq->curr updates.
+ * Required by sys_membarrier() when prev != next. We only learn about
+ * next later, so we issue this mb() unconditionally.
+ */
+ smp_mb__after_spin_lock();
update_rq_clock(rq);
clear_tsk_need_resched(prev);


--
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/