Re: [PATCH] locking/rwsem: don't let a read lock steal from a queued writer

From: Lorenzo Stoakes (ARM)

Date: Thu Nov 19 2026 - 09:14:57 EST


I know the cycle is grim on workload (equally so for us), but ping on this, as
we're relying on this behaviour for a series [0].

I could re-implement as an optional mode and respin [0] if necessary, but Peter
objected [1] so hoisted the rwsem change to here :)

Thanks!

[0]: https://lore.kernel.org/linux-mm/20261110-khugepaged-latency-fix-v3-0-6eea743cf9a3@xxxxxxxxxx/
[1]: https://lore.kernel.org/all/20261103115159.GK1813469@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/

On Thu, Nov 05, 2026 at 08:24:27PM +0000, Lorenzo Stoakes (ARM) wrote:
> Since commit 1a728dff855a ("locking/rwsem: Enable reader optimistic lock
> stealing"), a racing read lock can steal an rwsem from under a queued
> writer whose handoff timeout hasn't expired yet.
>
> This is most problematic when an rwsem read lock is released then
> reacquired in order to relieve lock contention, as instead it will simply
> have the lock handed right back to it ahead of the intended beneficiary.
>
> Naturally, this behaviour rather defeats the object of doing this kind of
> thing, so this is less than ideal.
>
> Fix this by updating the optimistic read lock stealing code in
> rwsem_down_read_slowpath() to explicitly check whether the first queued
> lock is a waiting writer, and to not allow the theft if so.
>
> When a steal is disallowed, keep the existing sem->wait_lock held when
> jumping to the queue code to avoid useless lock acquisition/release.
>
> In order to perform this check, sem->wait_lock is acquired whenever there
> are queued waiters in the steal code, rather than the previous behaviour of
> only doing so when the reader count is 1.
>
> Since failing to steal a read lock unconditionally results in acquisition
> of the sem->wait_lock, this doesn't seem likely to be a significant
> overhead.
>
> Also update the comment describing optimistic read lock acquisition
> conditions to include this newly introduced behaviour.
>
> Care is taken to ensure correct ordering of lock acquisition, lock waking
> and marking the lock reader-owned.
>
> Signed-off-by: Lorenzo Stoakes (ARM) <ljs@xxxxxxxxxx>
> ---
> kernel/locking/rwsem.c | 30 ++++++++++++++++++------------
> 1 file changed, 18 insertions(+), 12 deletions(-)
>
> diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c
> index b9c180ac1eee..cdb3dd6260bf 100644
> --- a/kernel/locking/rwsem.c
> +++ b/kernel/locking/rwsem.c
> @@ -59,7 +59,8 @@
> * A fast path reader optimistic lock stealing is supported when the rwsem
> * is previously owned by a writer and the following conditions are met:
> * - rwsem is not currently writer owned
> - * - the handoff isn't set.
> + * - the handoff isn't set
> + * - a writer isn't at the head of the wait queue.
> */
> #define RWSEM_READER_OWNED (1UL << 0)
> #define RWSEM_NONSPINNABLE (1UL << 1)
> @@ -1034,31 +1035,36 @@ rwsem_down_read_slowpath(struct rw_semaphore *sem, long count, unsigned int stat
> * Reader optimistic lock stealing.
> */
> if (!(count & (RWSEM_WRITER_LOCKED | RWSEM_FLAG_HANDOFF))) {
> - rwsem_set_reader_owned(sem);
> - lockevent_inc(rwsem_rlock_steal);
> -
> - /*
> - * Wake up other readers in the wait queue if it is
> - * the first reader.
> - */
> - if ((rcnt == 1) && (count & RWSEM_FLAG_WAITERS)) {
> + if (count & RWSEM_FLAG_WAITERS) {
> raw_spin_lock_irq(&sem->wait_lock);
> - if (sem->first_waiter)
> + first = sem->first_waiter;
> + /* A writer at the head of queue takes precedence. */
> + if (first && first->type == RWSEM_WAITING_FOR_WRITE)
> + goto queue_locked;
> + rwsem_set_reader_owned(sem);
> + /* If first reader, wake other readers in queue. */
> + if (first && rcnt == 1)
> rwsem_mark_wake(sem, RWSEM_WAKE_READ_OWNED,
> &wake_q);
> raw_spin_unlock_irq(&sem->wait_lock);
> - wake_up_q(&wake_q);
> + } else {
> + rwsem_set_reader_owned(sem);
> }
> +
> + lockevent_inc(rwsem_rlock_steal);
> + if (!wake_q_empty(&wake_q))
> + wake_up_q(&wake_q);
> return sem;
> }
>
> queue:
> + raw_spin_lock_irq(&sem->wait_lock);
> +queue_locked:
> waiter.task = current;
> waiter.type = RWSEM_WAITING_FOR_READ;
> waiter.timeout = jiffies + RWSEM_WAIT_TIMEOUT;
> waiter.handoff_set = false;
>
> - raw_spin_lock_irq(&sem->wait_lock);
> first = sem->first_waiter;
> if (!first) {
> /*
>
> ---
> base-commit: 05441d697f7f9a927264f4058cbbc6a03dfcac9d
> change-id: 20261105-rwsem-dont-steal-from-queued-writer-04b11d9d406b
>
> Cheers,
> --
> Lorenzo Stoakes (ARM) <ljs@xxxxxxxxxx>
>

--
Cheers, Lorenzo