[SRU][Jammy][PATCH 0/1] Do not sort the task scan result from /proc when synthesizing perf events

Thu Mar 2 07:16:15 UTC 2023

BugLink: https://bugs.launchpad.net/bugs/2008971

SRU Justification:

[Impact]
The perf tool use scandir() to iterate threads and sort in alphabetical order when synthesizing PERF_RECORD_ events.
If the process ID is 9999 and it has one thread (tid = 10000), the thread will be processed before the process.
It results in PERF_RECORD_FORK events that come before PERF_RECORD_MMAP2 events.
The callstack will have missing symbols for threads where `PERF_RECORD_FORK` events are processed before the `PERF_RECORD_MMAP2` event for the corresponding process.

[Fix]
Do not use alphasort when calling scandir()

363afa3aef24f5e08df6a539f5dc3aae4cddcc1a (perf synthetic-events: Don't sort the task scan result from /proc)

[Test Plan]
<test.lua>
function update_last_pid()
  local file <close> = io.open("/proc/sys/kernel/ns_last_pid", "w")
  file:write(9997)
end

update_last_pid()
os.execute("~/reproducer")

<reproducer.cpp>
#include <iostream>
#include <thread>
#include <unistd.h>

constexpr int kThreadNum = 10;

void thread_job() { sleep(30); }

int main(void) {
  std::thread threads[kThreadNum];

  std::cout << "Parent process with pid " << getpid() << std::endl;

  for (int i = 0; i < kThreadNum; ++i) {
    threads[i] = std::thread(thread_job);
  }

  for (int i = 0; i < kThreadNum; ++i) {
    threads[i].join();
  }

  std::cout << "All threads have finished" << std::endl;
  return 0;
}

The flow is to set /proc/sys/kernel/ns_last_pid first, which represents the last pid allocated in the current pid namespace.
The script (test.lua) sets ns_last_pid to 9997 and executes the reproducer (reproducer.cpp).
After the reproducer creates ten threads, we execute the perf command as follows: perf record -F 49 -e cpu-clock -a -g sleep 20.
Here is the result of command: perf report -f --tasks --mmaps -D | egrep -i 'perf_record_fork|perf_record_mmap' | grep 9999

Before applying the patch, the output of the perf command was as follows:
0 0 0x34910 [0x40]: PERF_RECORD_FORK(9999:10000):(9999:9999)
0 0 0x34990 [0x40]: PERF_RECORD_FORK(9999:10001):(9999:9999)
0 0 0x34a10 [0x40]: PERF_RECORD_FORK(9999:10002):(9999:9999)
0 0 0x34a90 [0x40]: PERF_RECORD_FORK(9999:10003):(9999:9999)
0 0 0x34b10 [0x40]: PERF_RECORD_FORK(9999:10004):(9999:9999)
0 0 0x34b90 [0x40]: PERF_RECORD_FORK(9999:10005):(9999:9999)
0 0 0x34c10 [0x40]: PERF_RECORD_FORK(9999:10006):(9999:9999)
0 0 0x34c90 [0x40]: PERF_RECORD_FORK(9999:10007):(9999:9999)
0 0 0x34d10 [0x40]: PERF_RECORD_FORK(9999:10008):(9999:9999)
0 0 0x34d90 [0x40]: PERF_RECORD_FORK(9999:10009):(9999:9999)
0 0 0x34e10 [0x40]: PERF_RECORD_FORK(9999:9999):(9998:9998)
0 0 0x34e90 [0x80]: PERF_RECORD_MMAP2 9999/9999: [0x555de0159000(0x1000) @ 0x1000 fd:00 2097758 0]: r-xp /root/reproducer
0 0 0x34f10 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd17f8000(0x7c000) @ 0xe000 fd:00 1055098 0]: r-xp /usr/lib/x86_64-linux-gnu/libm.so.6
0 0 0x34fa0 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd18f9000(0x195000) @ 0x28000 fd:00 1054988 0]: r-xp /usr/lib/x86_64-linux-gnu/libc.so.6
0 0 0x35030 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd1afc000(0x17000) @ 0x3000 fd:00 1054745 0]: r-xp /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
0 0 0x350c0 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd1bb3000(0x110000) @ 0x9a000 fd:00 1068050 0]: r-xp /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
0 0 0x35158 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd1d4e000(0x2a000) @ 0x2000 fd:00 1054949 0]: r-xp /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
0 0 0x351f0 [0x70]: PERF_RECORD_MMAP2 9999/9999: [0x7ffc983bb000(0x2000) @ 0 00:00 0 0]: r-xp [vdso]
0 0 0x35260 [0x78]: PERF_RECORD_MMAP2 9999/9999: [0xffffffffff600000(0x1000) @ 0 00:00 0 0]: --xp [vsyscall]

After applying the patch, the output of the perf command is as follows:
0 0 0x30c28 [0x40]: PERF_RECORD_FORK(9999:9999):(9998:9998)
0 0 0x30ca8 [0x80]: PERF_RECORD_MMAP2 9999/9999: [0x5642c7635000(0x1000) @ 0x1000 fd:00 2097758 0]: r-xp /root/reproducer
0 0 0x30d28 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada03d000(0x7c000) @ 0xe000 fd:00 1055098 0]: r-xp /usr/lib/x86_64-linux-gnu/libm.so.6
0 0 0x30db8 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada13e000(0x195000) @ 0x28000 fd:00 1054988 0]: r-xp /usr/lib/x86_64-linux-gnu/libc.so.6
0 0 0x30e48 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada341000(0x17000) @ 0x3000 fd:00 1054745 0]: r-xp /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
0 0 0x30ed8 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada3f8000(0x110000) @ 0x9a000 fd:00 1068050 0]: r-xp /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
0 0 0x30f70 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada593000(0x2a000) @ 0x2000 fd:00 1054949 0]: r-xp /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
0 0 0x31008 [0x70]: PERF_RECORD_MMAP2 9999/9999: [0x7ffcb2116000(0x2000) @ 0 00:00 0 0]: r-xp [vdso]
0 0 0x31078 [0x78]: PERF_RECORD_MMAP2 9999/9999: [0xffffffffff600000(0x1000) @ 0 00:00 0 0]: --xp [vsyscall]
0 0 0x310f0 [0x40]: PERF_RECORD_FORK(9999:10000):(9999:9999)
0 0 0x31170 [0x40]: PERF_RECORD_FORK(9999:10001):(9999:9999)
0 0 0x311f0 [0x40]: PERF_RECORD_FORK(9999:10002):(9999:9999)
0 0 0x31270 [0x40]: PERF_RECORD_FORK(9999:10003):(9999:9999)
0 0 0x312f0 [0x40]: PERF_RECORD_FORK(9999:10004):(9999:9999)
0 0 0x31370 [0x40]: PERF_RECORD_FORK(9999:10005):(9999:9999)
0 0 0x313f0 [0x40]: PERF_RECORD_FORK(9999:10006):(9999:9999)
0 0 0x31470 [0x40]: PERF_RECORD_FORK(9999:10007):(9999:9999)
0 0 0x314f0 [0x40]: PERF_RECORD_FORK(9999:10008):(9999:9999)
0 0 0x31570 [0x40]: PERF_RECORD_FORK(9999:10009):(9999:9999)

[Where problems could occur]
The fix will apply upstream commits, so the regression can be considered as low.

Namhyung Kim (1):
  perf synthetic-events: Don't sort the task scan result from /proc

 tools/perf/util/synthetic-events.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

-- 
2.34.1