<div dir="ltr"><div>Acked-by: Luke Nowakowski-Krijger <<a href="mailto:luke.nowakowskikrijger@canonical.com">luke.nowakowskikrijger@canonical.com</a>></div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 1, 2023 at 11:17 PM Chengen Du <<a href="mailto:chengen.du@canonical.com">chengen.du@canonical.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">BugLink: <a href="https://bugs.launchpad.net/bugs/2008971" rel="noreferrer" target="_blank">https://bugs.launchpad.net/bugs/2008971</a> SRU Justification: [Impact] The perf tool use scandir() to iterate threads and sort in alphabetical order when synthesizing PERF_RECORD_ events. If the process ID is 9999 and it has one thread (tid = 10000), the thread will be processed before the process. It results in PERF_RECORD_FORK events that come before PERF_RECORD_MMAP2 events. The callstack will have missing symbols for threads where `PERF_RECORD_FORK` events are processed before the `PERF_RECORD_MMAP2` event for the corresponding process. [Fix] Do not use alphasort when calling scandir() 363afa3aef24f5e08df6a539f5dc3aae4cddcc1a (perf synthetic-events: Don't sort the task scan result from /proc) [Test Plan] <test.lua> function update_last_pid() local file <close> = io.open("/proc/sys/kernel/ns_last_pid", "w") file:write(9997) end update_last_pid() os.execute("~/reproducer") <reproducer.cpp> #include <iostream> #include <thread> #include <unistd.h> constexpr int kThreadNum = 10; void thread_job() { sleep(30); } int main(void) { std::thread threads[kThreadNum]; std::cout << "Parent process with pid " << getpid() << std::endl; for (int i = 0; i < kThreadNum; ++i) { threads[i] = std::thread(thread_job); } for (int i = 0; i < kThreadNum; ++i) { threads[i].join(); } std::cout << "All threads have finished" << std::endl; return 0; } The flow is to set /proc/sys/kernel/ns_last_pid first, which represents the last pid allocated in the current pid namespace. The script (test.lua) sets ns_last_pid to 9997 and executes the reproducer (reproducer.cpp). After the reproducer creates ten threads, we execute the perf command as follows: perf record -F 49 -e cpu-clock -a -g sleep 20. Here is the result of command: perf report -f --tasks --mmaps -D | egrep -i 'perf_record_fork|perf_record_mmap' | grep 9999 Before applying the patch, the output of the perf command was as follows: 0 0 0x34910 [0x40]: PERF_RECORD_FORK(9999:10000):(9999:9999) 0 0 0x34990 [0x40]: PERF_RECORD_FORK(9999:10001):(9999:9999) 0 0 0x34a10 [0x40]: PERF_RECORD_FORK(9999:10002):(9999:9999) 0 0 0x34a90 [0x40]: PERF_RECORD_FORK(9999:10003):(9999:9999) 0 0 0x34b10 [0x40]: PERF_RECORD_FORK(9999:10004):(9999:9999) 0 0 0x34b90 [0x40]: PERF_RECORD_FORK(9999:10005):(9999:9999) 0 0 0x34c10 [0x40]: PERF_RECORD_FORK(9999:10006):(9999:9999) 0 0 0x34c90 [0x40]: PERF_RECORD_FORK(9999:10007):(9999:9999) 0 0 0x34d10 [0x40]: PERF_RECORD_FORK(9999:10008):(9999:9999) 0 0 0x34d90 [0x40]: PERF_RECORD_FORK(9999:10009):(9999:9999) 0 0 0x34e10 [0x40]: PERF_RECORD_FORK(9999:9999):(9998:9998) 0 0 0x34e90 [0x80]: PERF_RECORD_MMAP2 9999/9999: [0x555de0159000(0x1000) @ 0x1000 fd:00 2097758 0]: r-xp /root/reproducer 0 0 0x34f10 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd17f8000(0x7c000) @ 0xe000 fd:00 1055098 0]: r-xp /usr/lib/x86_64-linux-gnu/libm.so.6 0 0 0x34fa0 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd18f9000(0x195000) @ 0x28000 fd:00 1054988 0]: r-xp /usr/lib/x86_64-linux-gnu/libc.so.6 0 0 0x35030 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd1afc000(0x17000) @ 0x3000 fd:00 1054745 0]: r-xp /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 0 0 0x350c0 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd1bb3000(0x110000) @ 0x9a000 fd:00 1068050 0]: r-xp /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30 0 0 0x35158 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd1d4e000(0x2a000) @ 0x2000 fd:00 1054949 0]: r-xp /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 0 0 0x351f0 [0x70]: PERF_RECORD_MMAP2 9999/9999: [0x7ffc983bb000(0x2000) @ 0 00:00 0 0]: r-xp [vdso] 0 0 0x35260 [0x78]: PERF_RECORD_MMAP2 9999/9999: [0xffffffffff600000(0x1000) @ 0 00:00 0 0]: --xp [vsyscall] After applying the patch, the output of the perf command is as follows: 0 0 0x30c28 [0x40]: PERF_RECORD_FORK(9999:9999):(9998:9998) 0 0 0x30ca8 [0x80]: PERF_RECORD_MMAP2 9999/9999: [0x5642c7635000(0x1000) @ 0x1000 fd:00 2097758 0]: r-xp /root/reproducer 0 0 0x30d28 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada03d000(0x7c000) @ 0xe000 fd:00 1055098 0]: r-xp /usr/lib/x86_64-linux-gnu/libm.so.6 0 0 0x30db8 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada13e000(0x195000) @ 0x28000 fd:00 1054988 0]: r-xp /usr/lib/x86_64-linux-gnu/libc.so.6 0 0 0x30e48 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada341000(0x17000) @ 0x3000 fd:00 1054745 0]: r-xp /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 0 0 0x30ed8 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada3f8000(0x110000) @ 0x9a000 fd:00 1068050 0]: r-xp /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30 0 0 0x30f70 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada593000(0x2a000) @ 0x2000 fd:00 1054949 0]: r-xp /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 0 0 0x31008 [0x70]: PERF_RECORD_MMAP2 9999/9999: [0x7ffcb2116000(0x2000) @ 0 00:00 0 0]: r-xp [vdso] 0 0 0x31078 [0x78]: PERF_RECORD_MMAP2 9999/9999: [0xffffffffff600000(0x1000) @ 0 00:00 0 0]: --xp [vsyscall] 0 0 0x310f0 [0x40]: PERF_RECORD_FORK(9999:10000):(9999:9999) 0 0 0x31170 [0x40]: PERF_RECORD_FORK(9999:10001):(9999:9999) 0 0 0x311f0 [0x40]: PERF_RECORD_FORK(9999:10002):(9999:9999) 0 0 0x31270 [0x40]: PERF_RECORD_FORK(9999:10003):(9999:9999) 0 0 0x312f0 [0x40]: PERF_RECORD_FORK(9999:10004):(9999:9999) 0 0 0x31370 [0x40]: PERF_RECORD_FORK(9999:10005):(9999:9999) 0 0 0x313f0 [0x40]: PERF_RECORD_FORK(9999:10006):(9999:9999) 0 0 0x31470 [0x40]: PERF_RECORD_FORK(9999:10007):(9999:9999) 0 0 0x314f0 [0x40]: PERF_RECORD_FORK(9999:10008):(9999:9999) 0 0 0x31570 [0x40]: PERF_RECORD_FORK(9999:10009):(9999:9999) [Where problems could occur] The fix will apply upstream commits, so the regression can be considered as low. Namhyung Kim (1): perf synthetic-events: Don't sort the task scan result from /proc tools/perf/util/synthetic-events.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- 2.34.1 -- kernel-team mailing list <a href="mailto:kernel-team@lists.ubuntu.com" target="_blank">kernel-team@lists.ubuntu.com</a> <a href="https://lists.ubuntu.com/mailman/listinfo/kernel-team" rel="noreferrer" target="_blank">https://lists.ubuntu.com/mailman/listinfo/kernel-team</a> </blockquote></div></div>