Commit f2f76a2c authored by Will Deacon's avatar Will Deacon Committed by Greg Kroah-Hartman

locking/qspinlock: Kill cmpxchg() loop when claiming lock from head of queue

commit c61da58d upstream.

When a queued locker reaches the head of the queue, it claims the lock
by setting _Q_LOCKED_VAL in the lockword. If there isn't contention, it
must also clear the tail as part of this operation so that subsequent
lockers can avoid taking the slowpath altogether.

Currently this is expressed as a cmpxchg() loop that practically only
runs up to two iterations. This is confusing to the reader and unhelpful
to the compiler. Rewrite the cmpxchg() loop without the loop, so that a
failed cmpxchg() implies that there is contention and we just need to
write to _Q_LOCKED_VAL without considering the rest of the lockword.
Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
Acked-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: default avatarWaiman Long <longman@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: boqun.feng@gmail.com
Cc: linux-arm-kernel@lists.infradead.org
Cc: paulmck@linux.vnet.ibm.com
Link: http://lkml.kernel.org/r/1524738868-31318-7-git-send-email-will.deacon@arm.comSigned-off-by: default avatarIngo Molnar <mingo@kernel.org>
Signed-off-by: default avatarSebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: default avatarSasha Levin <sashal@kernel.org>
parent 075703d7
...@@ -465,24 +465,21 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) ...@@ -465,24 +465,21 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
* and nobody is pending, clear the tail code and grab the lock. * and nobody is pending, clear the tail code and grab the lock.
* Otherwise, we only need to grab the lock. * Otherwise, we only need to grab the lock.
*/ */
for (;;) {
/* In the PV case we might already have _Q_LOCKED_VAL set */ /* In the PV case we might already have _Q_LOCKED_VAL set */
if ((val & _Q_TAIL_MASK) != tail || (val & _Q_PENDING_MASK)) { if ((val & _Q_TAIL_MASK) == tail) {
set_locked(lock);
break;
}
/* /*
* The smp_cond_load_acquire() call above has provided the * The smp_cond_load_acquire() call above has provided the
* necessary acquire semantics required for locking. At most * necessary acquire semantics required for locking.
* two iterations of this loop may be ran.
*/ */
old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL); old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL);
if (old == val) if (old == val)
goto release; /* No contention */ goto release; /* No contention */
val = old;
} }
/* Either somebody is queued behind us or _Q_PENDING_VAL is set */
set_locked(lock);
/* /*
* contended path; wait for next if not observed yet, release. * contended path; wait for next if not observed yet, release.
*/ */
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment