• Waiman Long's avatar
    locking/lockdep: Fix debug_locks off performance problem · 81301a15
    Waiman Long authored
    [ Upstream commit 9506a7425b094d2f1d9c877ed5a78f416669269b ]
    
    It was found that when debug_locks was turned off because of a problem
    found by the lockdep code, the system performance could drop quite
    significantly when the lock_stat code was also configured into the
    kernel. For instance, parallel kernel build time on a 4-socket x86-64
    server nearly doubled.
    
    Further analysis into the cause of the slowdown traced back to the
    frequent call to debug_locks_off() from the __lock_acquired() function
    probably due to some inconsistent lockdep states with debug_locks
    off. The debug_locks_off() function did an unconditional atomic xchg
    to write a 0 value into debug_locks which had already been set to 0.
    This led to severe cacheline contention in the cacheline that held
    debug_locks.  As debug_locks is being referenced in quite a few different
    places in the kernel, this greatly slow down the system performance.
    
    To prevent that trashing of debug_locks cacheline, lock_acquired()
    and lock_contended() now checks the state of debug_locks before
    proceeding. The debug_locks_off() function is also modified to check
    debug_locks before calling __debug_locks_off().
    Signed-off-by: 's avatarWaiman Long <longman@redhat.com>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: Will Deacon <will.deacon@arm.com>
    Link: http://lkml.kernel.org/r/1539913518-15598-1-git-send-email-longman@redhat.comSigned-off-by: 's avatarIngo Molnar <mingo@kernel.org>
    Signed-off-by: 's avatarSasha Levin <sashal@kernel.org>
    Signed-off-by: 's avatarGreg Kroah-Hartman <gregkh@linuxfoundation.org>
    81301a15
Name
Last commit
Last update
..
842 Loading commit data...
fonts Loading commit data...
lz4 Loading commit data...
lzo Loading commit data...
mpi Loading commit data...
raid6 Loading commit data...
reed_solomon Loading commit data...
xz Loading commit data...
zlib_deflate Loading commit data...
zlib_inflate Loading commit data...
.gitignore Loading commit data...
Kconfig Loading commit data...
Kconfig.debug Loading commit data...
Kconfig.kasan Loading commit data...
Kconfig.kgdb Loading commit data...
Kconfig.kmemcheck Loading commit data...
Kconfig.ubsan Loading commit data...
Makefile Loading commit data...
argv_split.c Loading commit data...
asn1_decoder.c Loading commit data...
assoc_array.c Loading commit data...
atomic64.c Loading commit data...
atomic64_test.c Loading commit data...
audit.c Loading commit data...
bcd.c Loading commit data...
bch.c Loading commit data...
bitmap.c Loading commit data...
bitrev.c Loading commit data...
bsearch.c Loading commit data...
btree.c Loading commit data...
bug.c Loading commit data...
build_OID_registry Loading commit data...
bust_spinlocks.c Loading commit data...
chacha20.c Loading commit data...
check_signature.c Loading commit data...
checksum.c Loading commit data...
clz_ctz.c Loading commit data...
clz_tab.c Loading commit data...
cmdline.c Loading commit data...
compat_audit.c Loading commit data...
cordic.c Loading commit data...
cpu-notifier-error-inject.c Loading commit data...
cpu_rmap.c Loading commit data...
cpumask.c Loading commit data...
crc-ccitt.c Loading commit data...
crc-itu-t.c Loading commit data...
crc-t10dif.c Loading commit data...
crc16.c Loading commit data...
crc32.c Loading commit data...
crc32defs.h Loading commit data...
crc7.c Loading commit data...
crc8.c Loading commit data...
ctype.c Loading commit data...
debug_info.c Loading commit data...
debug_locks.c Loading commit data...
debugobjects.c Loading commit data...
dec_and_lock.c Loading commit data...
decompress.c Loading commit data...
decompress_bunzip2.c Loading commit data...
decompress_inflate.c Loading commit data...
decompress_unlz4.c Loading commit data...
decompress_unlzma.c Loading commit data...
decompress_unlzo.c Loading commit data...
decompress_unxz.c Loading commit data...
devres.c Loading commit data...
digsig.c Loading commit data...
div64.c Loading commit data...
dma-debug.c Loading commit data...
dma-noop.c Loading commit data...
dump_stack.c Loading commit data...
dynamic_debug.c Loading commit data...
dynamic_queue_limits.c Loading commit data...
earlycpio.c Loading commit data...
extable.c Loading commit data...
fault-inject.c Loading commit data...
fdt.c Loading commit data...
fdt_empty_tree.c Loading commit data...
fdt_ro.c Loading commit data...
fdt_rw.c Loading commit data...
fdt_strerror.c Loading commit data...
fdt_sw.c Loading commit data...
fdt_wip.c Loading commit data...
find_bit.c Loading commit data...
flex_array.c Loading commit data...
flex_proportions.c Loading commit data...
gcd.c Loading commit data...
gen_crc32table.c Loading commit data...
genalloc.c Loading commit data...
glob.c Loading commit data...
halfmd4.c Loading commit data...
hexdump.c Loading commit data...
hweight.c Loading commit data...
idr.c Loading commit data...
inflate.c Loading commit data...
int_sqrt.c Loading commit data...
interval_tree.c Loading commit data...
interval_tree_test.c Loading commit data...
iomap.c Loading commit data...
iomap_copy.c Loading commit data...
iommu-common.c Loading commit data...
iommu-helper.c Loading commit data...
ioremap.c Loading commit data...
iov_iter.c Loading commit data...
irq_poll.c Loading commit data...
irq_regs.c Loading commit data...
is_single_threaded.c Loading commit data...
jedec_ddr_data.c Loading commit data...
kasprintf.c Loading commit data...
kfifo.c Loading commit data...
klist.c Loading commit data...
kobject.c Loading commit data...
kobject_uevent.c Loading commit data...
kstrtox.c Loading commit data...
kstrtox.h Loading commit data...
lcm.c Loading commit data...
libcrc32c.c Loading commit data...
list_debug.c Loading commit data...
list_sort.c Loading commit data...
llist.c Loading commit data...
locking-selftest-hardirq.h Loading commit data...
locking-selftest-mutex.h Loading commit data...
locking-selftest-rlock-hardirq.h Loading commit data...
locking-selftest-rlock-softirq.h Loading commit data...
locking-selftest-rlock.h Loading commit data...
locking-selftest-rsem.h Loading commit data...
locking-selftest-softirq.h Loading commit data...
locking-selftest-spin-hardirq.h Loading commit data...
locking-selftest-spin-softirq.h Loading commit data...
locking-selftest-spin.h Loading commit data...
locking-selftest-wlock-hardirq.h Loading commit data...
locking-selftest-wlock-softirq.h Loading commit data...
locking-selftest-wlock.h Loading commit data...
locking-selftest-wsem.h Loading commit data...
locking-selftest.c Loading commit data...
lockref.c Loading commit data...
lru_cache.c Loading commit data...
md5.c Loading commit data...
memory-notifier-error-inject.c Loading commit data...
memweight.c Loading commit data...
net_utils.c Loading commit data...
netdev-notifier-error-inject.c Loading commit data...
nlattr.c Loading commit data...
nmi_backtrace.c Loading commit data...
nodemask.c Loading commit data...
notifier-error-inject.c Loading commit data...
notifier-error-inject.h Loading commit data...
of-reconfig-notifier-error-inject.c Loading commit data...
oid_registry.c Loading commit data...
once.c Loading commit data...
parser.c Loading commit data...
pci_iomap.c Loading commit data...
percpu-refcount.c Loading commit data...
percpu_counter.c Loading commit data...
percpu_ida.c Loading commit data...
percpu_test.c Loading commit data...
plist.c Loading commit data...
pm-notifier-error-inject.c Loading commit data...
radix-tree.c Loading commit data...
random32.c Loading commit data...
ratelimit.c Loading commit data...
rational.c Loading commit data...
rbtree.c Loading commit data...
rbtree_test.c Loading commit data...
reciprocal_div.c Loading commit data...
rhashtable.c Loading commit data...
sbitmap.c Loading commit data...
scatterlist.c Loading commit data...
seq_buf.c Loading commit data...
sg_pool.c Loading commit data...
sg_split.c Loading commit data...
sha1.c Loading commit data...
show_mem.c Loading commit data...
smp_processor_id.c Loading commit data...
sort.c Loading commit data...
stackdepot.c Loading commit data...
stmp_device.c Loading commit data...
string.c Loading commit data...
string_helpers.c Loading commit data...
strncpy_from_user.c Loading commit data...
strnlen_user.c Loading commit data...
swiotlb.c Loading commit data...
syscall.c Loading commit data...
test-kstrtox.c Loading commit data...
test-string_helpers.c Loading commit data...
test_bitmap.c Loading commit data...
test_bpf.c Loading commit data...
test_firmware.c Loading commit data...
test_hash.c Loading commit data...
test_hexdump.c Loading commit data...
test_kasan.c Loading commit data...
test_module.c Loading commit data...
test_printf.c Loading commit data...
test_rhashtable.c Loading commit data...
test_static_key_base.c Loading commit data...
test_static_keys.c Loading commit data...
test_user_copy.c Loading commit data...
test_uuid.c Loading commit data...
textsearch.c Loading commit data...
timerqueue.c Loading commit data...
ts_bm.c Loading commit data...
ts_fsm.c Loading commit data...
ts_kmp.c Loading commit data...
ubsan.c Loading commit data...
ubsan.h Loading commit data...
ucs2_string.c Loading commit data...
uuid.c Loading commit data...
vsprintf.c Loading commit data...
win_minmax.c Loading commit data...