APPLIED: [SRU][Jammy][PATCH 0/1] Do not sort the task scan result from /proc when synthesizing perf events

Mon Mar 13 10:54:34 UTC 2023

On 02.03.23 08:16, Chengen Du wrote:
> BugLink: https://bugs.launchpad.net/bugs/2008971
> 
> SRU Justification:
> 
> [Impact]
> The perf tool use scandir() to iterate threads and sort in alphabetical order when synthesizing PERF_RECORD_ events.
> If the process ID is 9999 and it has one thread (tid = 10000), the thread will be processed before the process.
> It results in PERF_RECORD_FORK events that come before PERF_RECORD_MMAP2 events.
> The callstack will have missing symbols for threads where `PERF_RECORD_FORK` events are processed before the `PERF_RECORD_MMAP2` event for the corresponding process.
> 
> [Fix]
> Do not use alphasort when calling scandir()
> 
> 363afa3aef24f5e08df6a539f5dc3aae4cddcc1a (perf synthetic-events: Don't sort the task scan result from /proc)
> 
> [Test Plan]
> <test.lua>
> function update_last_pid()
>    local file <close> = io.open("/proc/sys/kernel/ns_last_pid", "w")
>    file:write(9997)
> end
> 
> update_last_pid()
> os.execute("~/reproducer")
> 
> <reproducer.cpp>
> #include <iostream>
> #include <thread>
> #include <unistd.h>
> 
> constexpr int kThreadNum = 10;
> 
> void thread_job() { sleep(30); }
> 
> int main(void) {
>    std::thread threads[kThreadNum];
> 
>    std::cout << "Parent process with pid " << getpid() << std::endl;
> 
>    for (int i = 0; i < kThreadNum; ++i) {
>      threads[i] = std::thread(thread_job);
>    }
> 
>    for (int i = 0; i < kThreadNum; ++i) {
>      threads[i].join();
>    }
> 
>    std::cout << "All threads have finished" << std::endl;
>    return 0;
> }
> 
> The flow is to set /proc/sys/kernel/ns_last_pid first, which represents the last pid allocated in the current pid namespace.
> The script (test.lua) sets ns_last_pid to 9997 and executes the reproducer (reproducer.cpp).
> After the reproducer creates ten threads, we execute the perf command as follows: perf record -F 49 -e cpu-clock -a -g sleep 20.
> Here is the result of command: perf report -f --tasks --mmaps -D | egrep -i 'perf_record_fork|perf_record_mmap' | grep 9999
> 
> Before applying the patch, the output of the perf command was as follows:
> 0 0 0x34910 [0x40]: PERF_RECORD_FORK(9999:10000):(9999:9999)
> 0 0 0x34990 [0x40]: PERF_RECORD_FORK(9999:10001):(9999:9999)
> 0 0 0x34a10 [0x40]: PERF_RECORD_FORK(9999:10002):(9999:9999)
> 0 0 0x34a90 [0x40]: PERF_RECORD_FORK(9999:10003):(9999:9999)
> 0 0 0x34b10 [0x40]: PERF_RECORD_FORK(9999:10004):(9999:9999)
> 0 0 0x34b90 [0x40]: PERF_RECORD_FORK(9999:10005):(9999:9999)
> 0 0 0x34c10 [0x40]: PERF_RECORD_FORK(9999:10006):(9999:9999)
> 0 0 0x34c90 [0x40]: PERF_RECORD_FORK(9999:10007):(9999:9999)
> 0 0 0x34d10 [0x40]: PERF_RECORD_FORK(9999:10008):(9999:9999)
> 0 0 0x34d90 [0x40]: PERF_RECORD_FORK(9999:10009):(9999:9999)
> 0 0 0x34e10 [0x40]: PERF_RECORD_FORK(9999:9999):(9998:9998)
> 0 0 0x34e90 [0x80]: PERF_RECORD_MMAP2 9999/9999: [0x555de0159000(0x1000) @ 0x1000 fd:00 2097758 0]: r-xp /root/reproducer
> 0 0 0x34f10 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd17f8000(0x7c000) @ 0xe000 fd:00 1055098 0]: r-xp /usr/lib/x86_64-linux-gnu/libm.so.6
> 0 0 0x34fa0 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd18f9000(0x195000) @ 0x28000 fd:00 1054988 0]: r-xp /usr/lib/x86_64-linux-gnu/libc.so.6
> 0 0 0x35030 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd1afc000(0x17000) @ 0x3000 fd:00 1054745 0]: r-xp /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
> 0 0 0x350c0 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd1bb3000(0x110000) @ 0x9a000 fd:00 1068050 0]: r-xp /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
> 0 0 0x35158 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd1d4e000(0x2a000) @ 0x2000 fd:00 1054949 0]: r-xp /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
> 0 0 0x351f0 [0x70]: PERF_RECORD_MMAP2 9999/9999: [0x7ffc983bb000(0x2000) @ 0 00:00 0 0]: r-xp [vdso]
> 0 0 0x35260 [0x78]: PERF_RECORD_MMAP2 9999/9999: [0xffffffffff600000(0x1000) @ 0 00:00 0 0]: --xp [vsyscall]
> 
> After applying the patch, the output of the perf command is as follows:
> 0 0 0x30c28 [0x40]: PERF_RECORD_FORK(9999:9999):(9998:9998)
> 0 0 0x30ca8 [0x80]: PERF_RECORD_MMAP2 9999/9999: [0x5642c7635000(0x1000) @ 0x1000 fd:00 2097758 0]: r-xp /root/reproducer
> 0 0 0x30d28 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada03d000(0x7c000) @ 0xe000 fd:00 1055098 0]: r-xp /usr/lib/x86_64-linux-gnu/libm.so.6
> 0 0 0x30db8 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada13e000(0x195000) @ 0x28000 fd:00 1054988 0]: r-xp /usr/lib/x86_64-linux-gnu/libc.so.6
> 0 0 0x30e48 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada341000(0x17000) @ 0x3000 fd:00 1054745 0]: r-xp /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
> 0 0 0x30ed8 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada3f8000(0x110000) @ 0x9a000 fd:00 1068050 0]: r-xp /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
> 0 0 0x30f70 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada593000(0x2a000) @ 0x2000 fd:00 1054949 0]: r-xp /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
> 0 0 0x31008 [0x70]: PERF_RECORD_MMAP2 9999/9999: [0x7ffcb2116000(0x2000) @ 0 00:00 0 0]: r-xp [vdso]
> 0 0 0x31078 [0x78]: PERF_RECORD_MMAP2 9999/9999: [0xffffffffff600000(0x1000) @ 0 00:00 0 0]: --xp [vsyscall]
> 0 0 0x310f0 [0x40]: PERF_RECORD_FORK(9999:10000):(9999:9999)
> 0 0 0x31170 [0x40]: PERF_RECORD_FORK(9999:10001):(9999:9999)
> 0 0 0x311f0 [0x40]: PERF_RECORD_FORK(9999:10002):(9999:9999)
> 0 0 0x31270 [0x40]: PERF_RECORD_FORK(9999:10003):(9999:9999)
> 0 0 0x312f0 [0x40]: PERF_RECORD_FORK(9999:10004):(9999:9999)
> 0 0 0x31370 [0x40]: PERF_RECORD_FORK(9999:10005):(9999:9999)
> 0 0 0x313f0 [0x40]: PERF_RECORD_FORK(9999:10006):(9999:9999)
> 0 0 0x31470 [0x40]: PERF_RECORD_FORK(9999:10007):(9999:9999)
> 0 0 0x314f0 [0x40]: PERF_RECORD_FORK(9999:10008):(9999:9999)
> 0 0 0x31570 [0x40]: PERF_RECORD_FORK(9999:10009):(9999:9999)
> 
> [Where problems could occur]
> The fix will apply upstream commits, so the regression can be considered as low.
> 
> Namhyung Kim (1):
>    perf synthetic-events: Don't sort the task scan result from /proc
> 
>   tools/perf/util/synthetic-events.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 

Applied to jammy:linux/master-next. Thanks.

-Stefan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0xE8675DEECBEECEA3.asc
Type: application/pgp-keys
Size: 44613 bytes
Desc: OpenPGP public key
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20230313/62e1addb/attachment-0001.key>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20230313/62e1addb/attachment-0001.sig>