rjmcmahon
/
umber-kernel
1
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity
umber-kernel/kernel
History
Colin Lord 359ca3038b tracing: Fix false sharing in hwlat get_sample() 
[ Upstream commit f743435f988cb0cf1f521035aee857851b25e06d ]

The get_sample() function in the hwlat tracer assumes the caller holds
hwlat_data.lock, but this is not actually happening. The result is
unprotected data access to hwlat_data, and in per-cpu mode can result in
false sharing which may show up as false positive latency events.

The specific case of false sharing observed was primarily between
hwlat_data.sample_width and hwlat_data.count. These are separated by
just 8B and are therefore likely to share a cache line. When one thread
modifies count, the cache line is in a modified state so when other
threads read sample_width in the main latency detection loop, they fetch
the modified cache line. On some systems, the fetch itself may be slow
enough to count as a latency event, which could set up a self
reinforcing cycle of latency events as each event increments count which
then causes more latency events, continuing the cycle.

The other result of the unprotected data access is that hwlat_data.count
can end up with duplicate or missed values, which was observed on some
systems in testing.

Convert hwlat_data.count to atomic64_t so it can be safely modified
without locking, and prevent false sharing by pulling sample_width into
a local variable.

One system this was tested on was a dual socket server with 32 CPUs on
each numa node. With settings of 1us threshold, 1000us width, and
2000us window, this change reduced the number of latency events from
500 per second down to approximately 1 event per minute. Some machines
tested did not exhibit measurable latency from the false sharing.

Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Link: https://patch.msgid.link/20260210074810.6328-1-clord@mykolab.com
Signed-off-by: Colin Lord <clord@mykolab.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
 		2026-03-04 07:20:29 -05:00
..
bpf bpf: Properly mark live registers for indirect jumps 2026-03-04 07:19:41 -05:00
cgroup cgroup/cpuset: Don't fail cpuset.cpus change in v2 2026-03-04 07:19:56 -05:00
configs kcfi: Rename CONFIG_CFI_CLANG to CONFIG_CFI 2025-09-24 14:29:14 -07:00
debug kdb: remove redundant check for scancode 0xe0 2025-09-20 21:19:09 +01:00
dma dma/pool: distinguish between missing and exhausted atomic pools 2026-02-06 16:57:36 +01:00
entry hyperv-next for v6.18 2025-10-07 08:40:15 -07:00
events perf/core: Fix slow perf_event_task_exit() with LBR callstacks 2026-03-04 07:19:41 -05:00
futex futex: Optimize per-cpu reference counting 2025-11-06 12:30:54 +01:00
gcov gcov: add support for GCC 15 2025-11-09 21:19:44 -08:00
irq genirq/cpuhotplug: Notify about affinity changes breaking the affinity mask 2026-03-04 07:19:40 -05:00
kcsan Kernel Concurrency Sanitizer (KCSAN) updates for v6.18 2025-10-02 08:31:44 -07:00
livepatch livepatch: Match old_sympos 0 and 1 in klp_find_func() 2026-01-02 12:56:39 +01:00
locking locktorture: Fix memory leak in param_set_cpumask() 2025-12-18 14:02:51 +01:00
module module: add helper function for reading module_buildid() 2026-02-26 14:59:17 -08:00
power PM: hibernate: Fix crash when freeing invalid crypto compressor 2026-01-17 16:35:14 +01:00
printk printk: Avoid irq_work for printk_deferred() on suspend 2026-01-02 12:57:09 +01:00
rcu rcu: Fix rcu_read_unlock() deadloop due to softirq 2026-02-26 14:58:57 -08:00
sched sched/debug: Fix updating of ppos on server write ops 2026-03-04 07:19:43 -05:00
time hrtimer: Fix trace oddity 2026-02-26 14:59:03 -08:00
trace tracing: Fix false sharing in hwlat get_sample() 2026-03-04 07:20:29 -05:00
unwind unwind: Finish up unwind when a task exits 2025-07-31 10:20:11 -04:00
.gitignore kheaders: rebuild kheaders_data.tar.xz when a file is modified within a minute 2025-06-24 20:30:37 +09:00
Kconfig.freezer
…
Kconfig.hz kernel: Fix "select" wording on HZ_250 description 2025-02-21 09:20:30 +01:00
Kconfig.kexec kho: warn and fail on metadata or preserved memory in scratch area 2025-11-09 21:19:41 -08:00
Kconfig.locks
…
Kconfig.preempt softirq: Allow to drop the softirq-BKL lock on PREEMPT_RT 2025-09-17 16:25:41 +02:00
Makefile kho: warn and fail on metadata or preserved memory in scratch area 2025-11-09 21:19:41 -08:00
acct.c kernel/acct.c: saner struct file treatment 2025-09-27 20:13:56 -04:00
async.c
…
audit.c audit: fix skb leak when audit rate limit is exceeded 2025-09-10 19:55:00 -04:00
audit.h audit: create audit_stamp structure 2025-08-30 10:15:28 -04:00
audit_fsnotify.c VFS/audit: introduce kern_path_parent() for audit 2025-09-23 12:37:35 +02:00
audit_tree.c mount-related stuff for this cycle 2025-10-03 10:19:44 -07:00
audit_watch.c VFS/audit: introduce kern_path_parent() for audit 2025-09-23 12:37:35 +02:00
auditfilter.c audit/stable-6.18 PR 20250926 2025-09-30 08:22:16 -07:00
auditsc.c audit: add record for multiple object contexts 2025-08-30 10:15:30 -04:00
backtracetest.c
…
bounds.c
…
capability.c capability: Remove unused has_capability 2025-03-07 22:03:09 -06:00
cfi.c cfi: Move BPF CFI types and helpers to generic code 2025-07-31 18:23:53 -07:00
compat.c
…
configs.c
…
context_tracking.c context_tracking: Make RCU watch ct_kernel_exit_state() warning 2025-03-04 18:44:29 -08:00
cpu.c cpu: Make atomic hotplug callbacks run with interrupts disabled on UP 2025-12-18 14:03:41 +01:00
cpu_pm.c
…
crash_core.c crash: fix crashkernel resource shrink 2025-11-15 10:52:01 -08:00
crash_core_test.c crash: add KUnit tests for crash_exclude_mem_range 2025-09-13 17:32:55 -07:00
crash_dump_dm_crypt.c crash_dump: retrieve dm crypt keys in kdump kernel 2025-05-21 10:48:21 -07:00
crash_reserve.c crash: let architecture decide crash memory export to iomem_resource 2026-01-02 12:57:11 +01:00
cred.c copy_process: pass clone_flags as u64 across calltree 2025-09-01 15:31:34 +02:00
delayacct.c delayacct: remove redundant code and adjust indentation 2025-05-27 19:40:33 -07:00
dma.c
…
elfcorehdr.c
…
exec_domain.c
…
exit.c task_stack.h: clean-up stack_not_used() implementation 2025-09-21 14:22:00 -07:00
exit.h
…
extable.c
…
fail_function.c
…
fork.c Patch series in this pull request: 2025-10-02 18:44:54 -07:00
freezer.c mm/oom_kill: thaw the entire OOM victim process 2025-09-21 14:22:35 -07:00
gen_kheaders.sh kheaders: make it possible to override TAR 2025-08-06 10:23:36 +09:00
groups.c
…
hung_task.c hung_task: dump blocker task if it is not hung 2025-09-13 17:32:43 -07:00
iomem.c mm/memremap: Pass down MEMREMAP_* flags to arch_memremap_wb() 2025-02-21 15:05:38 +01:00
irq_work.c
…
jump_label.c jump_label: Use RCU in all users of __module_text_address(). 2025-03-10 11:54:46 +01:00
kallsyms.c kallsyms/ftrace: set module buildid in ftrace_mod_address_lookup() 2026-02-26 14:59:17 -08:00
kallsyms_internal.h
…
kallsyms_selftest.c kallsyms: use kmalloc_array() instead of kmalloc() 2025-09-28 11:36:14 -07:00
kallsyms_selftest.h
…
kcmp.c kcmp: improve performance adding an unlikely hint to task comparisons 2025-02-21 10:25:33 +01:00
kcov.c kcov: use write memory barrier after memcpy() in kcov_move_area() 2025-09-13 17:32:44 -07:00
kexec.c kexec: enable CMA based contiguous allocation 2025-08-02 12:01:38 -07:00
kexec_core.c kernel/kexec: fix IMA when allocation happens in CMA area 2026-01-08 10:17:10 +01:00
kexec_elf.c kexec: initialize ELF lowest address to ULONG_MAX 2025-03-16 22:30:47 -07:00
kexec_file.c x86/kexec: carry forward the boot DTB on kexec 2025-09-13 17:32:43 -07:00
kexec_handover.c kho: warn and exit when unpreserved page wasn't preserved 2025-11-09 21:19:47 -08:00
kexec_handover_debug.c kho: warn and fail on metadata or preserved memory in scratch area 2025-11-09 21:19:41 -08:00
kexec_handover_internal.h kho: warn and fail on metadata or preserved memory in scratch area 2025-11-09 21:19:41 -08:00
kexec_internal.h kexec: enable CMA based contiguous allocation 2025-08-02 12:01:38 -07:00
kheaders.c
…
kprobes.c kprobes: Add missing kerneldoc for __get_insn_slot 2025-07-15 18:45:34 +09:00
kstack_erase.c stackleak: Rename stackleak_track_stack to __sanitizer_cov_stack_depth 2025-07-21 21:40:39 -07:00
ksyms_common.c
…
ksysfs.c
…
kthread.c ipvs: Fix estimator kthreads preferred affinity 2025-08-13 08:34:33 +02:00
latencytop.c
…
module_signature.c
…
notifier.c
…
nscommon.c ns: drop assert 2025-09-25 09:23:54 +02:00
nsproxy.c namespace-6.18-rc1 2025-09-29 11:20:29 -07:00
nstree.c ns: move ns type into struct ns_common 2025-09-25 09:23:54 +02:00
padata.c padata: WQ_PERCPU added to alloc_workqueue users 2025-09-13 12:11:06 +08:00
panic.c panic: only warn about deprecated panic_print on write access 2026-01-30 10:32:22 +01:00
params.c params: Replace deprecated strcpy() with strscpy() and memcpy() 2025-08-16 21:47:25 +02:00
pid.c namespace-6.18-rc1 2025-09-29 11:20:29 -07:00
pid_namespace.c namespace-6.18-rc1 2025-09-29 11:20:29 -07:00
pid_sysctl.h
…
profile.c
…
ptrace.c ptrace: introduce PTRACE_SET_SYSCALL_INFO request 2025-05-11 17:48:15 -07:00
range.c
…
reboot.c - The 7 patch series "powerpc/crash: use generic crashkernel 2025-04-01 10:06:52 -07:00
regset.c
…
relay.c relayfs: support a counter tracking if data is too big to write 2025-07-09 22:57:52 -07:00
resource.c Reinstate "resource: avoid unnecessary lookups in find_next_iomem_res()" 2025-12-18 14:03:25 +01:00
resource_kunit.c
…
rseq.c rseq: Protect event mask against membarrier IPI 2025-09-13 19:51:59 +02:00
scftorture.c
…
scs.c scs: fix a wrong parameter in __scs_magic 2026-01-02 12:57:19 +01:00
seccomp.c Performance events updates for v6.18: 2025-09-30 11:11:21 -07:00
signal.c signal: Fix memory leak for PIDFD_SELF* sentinels 2025-08-19 13:51:28 +02:00
smp.c smp: Fix up and expand the smp_call_function_many() kerneldoc 2025-09-18 22:21:28 +02:00
smpboot.c sched/smp: Use the SMP version of idle_thread_set_boot_cpu() 2025-06-13 08:47:20 +02:00
smpboot.h
…
softirq.c softirq: Allow to drop the softirq-BKL lock on PREEMPT_RT 2025-09-17 16:25:41 +02:00
stacktrace.c
…
static_call.c
…
static_call_inline.c Modules changes for 6.15-rc1 2025-03-30 15:44:36 -07:00
stop_machine.c sched/core: Fix migrate_swap() vs. hotplug 2025-07-01 15:02:03 +02:00
sys.c Patch series in this pull request: 2025-10-02 18:44:54 -07:00
sys_ni.c uprobes/x86: Add uprobe syscall to speed up uprobe 2025-08-21 20:09:20 +02:00
sysctl-test.c sysctl: move u8 register test to lib/test_sysctl.c 2025-04-14 14:13:41 +02:00
sysctl.c sysctl: rename kern_table -> sysctl_subsys_table 2025-07-23 11:56:02 +02:00
task_work.c task_work: Fix NMI race condition 2025-12-18 14:02:42 +01:00
taskstats.c
…
torture.c torture: Delay CPU-hotplug operations until boot completes 2025-08-14 15:26:30 -07:00
tracepoint.c tracepoint: Print the function symbol when tracepoint_debug is set 2025-03-21 15:30:10 -04:00
tsacct.c pid: change bacct_add_tsk() to use task_ppid_nr_ns() 2025-08-19 13:38:20 +02:00
ucount.c ucount: check for CAP_SYS_RESOURCE using ns_capable_noaudit() 2026-02-26 14:59:20 -08:00
uid16.c
…
uid16.h
…
umh.c
…
up.c
…
user-return-notifier.c
…
user.c ns: move ns type into struct ns_common 2025-09-25 09:23:54 +02:00
user_namespace.c ns: move ns type into struct ns_common 2025-09-25 09:23:54 +02:00
utsname.c namespace-6.18-rc1 2025-09-29 11:20:29 -07:00
utsname_sysctl.c
…
vhost_task.c vhost: Take a reference on the task in struct vhost_task. 2025-09-21 17:44:20 -04:00
vmcore_info.c crash: export PAGE_UNACCEPTED_MAPCOUNT_VALUE to vmcoreinfo 2025-05-11 17:54:04 -07:00
watch_queue.c vfs-6.15-rc1.pipe 2025-03-24 09:52:37 -07:00
watchdog.c watchdog: skip checks when panic is in progress 2025-09-13 17:32:53 -07:00
watchdog_buddy.c watchdog: fix opencoded cpumask_next_wrap() in watchdog_next_cpu() 2025-07-31 11:28:03 -04:00
watchdog_perf.c watchdog: skip checks when panic is in progress 2025-09-13 17:32:53 -07:00
workqueue.c workqueue: Process rescuer work items one-by-one using a cursor 2026-02-26 14:59:10 -08:00
workqueue_internal.h
…