Profiling ftw?

[John Richard Moser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I read this LWN.net article a while back, thought on it, made some
attempts; but got myself stuck and shrugged it off.  Still, right now
I'm thinking on some performance things, so this is worth mentioning.
The stuff here is a brain dump; there's stuff about ptrace and strace
and getting strace and ltrace together and some coding that needs to be
done, but that's all upstream stuff that's not our problem.

http://lwn.net/Articles/192214/

Here, Dave Jones basically goes over a bunch of crap that does a bunch
of crap that's a waste of time; the obvious implication (heard loud and
clear) is that the developers need to fix their broken implementations
so that they don't do thousands of unnecessary operations.  This will
result in the system running smoother and faster.

I am thinking that kernel instrumentation is a bad path; we should look
for a user-space solution to do these kinds of boot/run tests.
Collecting such data should be easily done; and we should be able to
analyze and graph reports, possibly even in real time.

The tools that would be needed would need to do a few things:

 - Trace and track processes separately, keeping track of fork()ed
   processes
 - Trace and track threads separately
 - Report on system calls (strace)
 - Report on library calls (ltrace)
 - Produce summaries at the end of a run
 - Produce cumulative summaries during run on interval
 - Produce interval summaries (i.e. operations for last interval only)
   during run

I am looking at strace and ltrace for this, but there are some
limitations.  First off it's hard to use them together; but also, I
don't think either tracks which thread it's in, and strace had this to
say about fork()ed processes:

 -f
    Trace child processes as they are  created by currently traced
    processes as a result of the fork(2) system call.  The new process
    is attached to as soon as its pid is known (through the return value
    of fork(2) in the parent process). This means that such children may
    run  uncontrolled for a while (especially in the case of a
    vfork(2)), until the parent is scheduled again to complete its
    (v)fork(2) call. If the parent process decides to wait(2) for a
    child that is currently being traced, it is suspended until an
    appropriate child process either terminates or incurs a signal that
    would cause it to terminate (as determined from the child’s current
    signal disposition).

As I understand, strace uses ptrace() so you should be able to use
PTRACE_O_TRACEFORK PTRACE_O_TRACEVFORK PTRACE_O_TRACECLONE options to
fix this behavior and make it perfect.  This is an exercise for the
strace developers.

ltrace traces library calls, which is awesome.  Unfortunately it doesn't
separate syscalls for better analysis.  It's very useful to have these
separate.  It also won't trace calls to dlopen()ed libraries.

It'd be most useful to fix these bugs, and then get strace and ltrace
upstream to work on merging the two into a library with two front-ends.
 Such a library could then be used for another front-end which could do
the reporting/measuring per interval so that we can get a graph of what
i.e. Firefox is doing under different loads.

Following that, in theory there could be a switch that would cause init
to strace everything (except itself) as it comes up, and produce
(emitted to a file) a full report of the boot process, which could be
graphed and analyzed.  This would be like the bootchart timers, except
it would have fine-grained syscall and library call tracking.

The resulting information could be used by developers to grind down the
performance issues in their code.  This would be a sheer brute force
method of getting the system to load faster, which would make a great
supplement to other hacks like linker optimizations and the like.

Does anyone have any questions, comments, or ideas on whether or not
there's a useful path to be taken here?  I'm not sure if I should be
flagging upstream on this, or writing a spec, or pestering #-devel, or
what.  I just want some good on-line quick and dirty profilers.

- --
All content of all messages exchanged herein are left in the
Public Domain, unless otherwise explicitly stated.

    Creative brains are a valuable, limited resource. They shouldn't be
    wasted on re-inventing the wheel when there are so many fascinating
    new problems waiting out there.
                                                 -- Eric Steven Raymond

    We will enslave their women, eat their children and rape their
    cattle!
                  -- Bosc, Evil alien overlord from the fifth dimension
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQIVAwUBROZBFAs1xW0HCTEFAQKvSxAAoc7pe+1MNB0MsQmJ4zjw3gnQusAUD2VU
S4++kyuJQlNVCBUaFJzy9ln134tydvavIvQ/DB7K+rehQYv149Ljo0Gcrni7f4fs
QuLkuUJJtZIvcUfFCYRdEq44nOwt2LbLOAzaLkBlvFmYSft/HFu3/heiE0ka+/ox
ZJOdc2Lw9qARUSmEMC86Qf6/cFPCnnDeFVZ7j6fwJiH9LeutJf/mYxVyUm5WCHiK
RaP5oH07uNOCxl2UQRpHmM9Id/FzuNlstdT8PWvVQMFDZ6oIq2NaeDtFcc8A8FoK
1lQ1Ktu8G45V9LklIUxR9UEkFSi1NC8NZr/IkX8HBmx7OsahaFpGLHR1kGJnJWHK
4KjA+Tdv7q92Gl39ftQSGEs0pWMbgQKZ+liDvblWenqCwQEdlNTcFUi9ITHicdmN
nEic1puyAUWbsiQDuASrMdy+EYvUNGLZ8pRk7pr2EgwzxZ0Oj7vkOdSj+Uaz24cI
i/LPF7BKlgrsDYSG/bKCtdtJS6weD71FXpb17cMb6eZtq4WwuF9TpMX5d+vmDj+/
zz/iIbUTJaYnTvEnwvsvlkZxrZwxoZ2g7re2S/O3wzY2I6tPaPBz9iNgsQ7Ynz5X
P4I+crAXzF8wNHBbbm7emVSv1tJWMkKkNiCnwN35pP38LWIUR9CMFnrZBWsO1Q/+
wFTHotFIwuo=
=B+Fa
-----END PGP SIGNATURE-----