[MERGE] log performance

[John Arbash Meinel]

Aaron Bentley wrote:
> John Arbash Meinel wrote:
>>> Martin commented on this, but it is true that you don't have to build up
>>> the list. If you use:
>>> rev_nos = dict((k, v) for v, k in cut_revs)
>>>
>>> the dict() can accept a generator. I don't know if it saves a lot, but
>>> it seems like it should.
> 
> I would have thought that generator expressions would be slower than
> list comprehensions because of all the function call overhead.  But in
> fact, you're right.  It's faster.

Actually, the overhead is in 'frame setup'. With a generator, you
already have the frame setup. So it is actually quite cheap for
thousands of calls.

Now, with the proposed spec that you can pass more stuff into
generators, it would be interesting to refactor some of our inner loop
functions such that they are a 'generator' where you keep passing in new
arguments.

I'm curious how generators and frames interact with the cpu's physical
stack. Because a python function call isn't really mapped directly onto
a push + jump.
But in general, using a generator over 1000 items is *much* faster than
calling a standard function 1000 times. We might also keep that in mind
with our current code, trying to build things into lists and calling a
generator over them.

> 
>>> I also wanted to comment on the 'which have a 1:1 merge ratio'. I think
>>> this is a good start. The current bzr.dev actually has 6500 revisions,
>>> and only 1800 are on mainline. Which gives us 3.5:1.
> 
> Not too surprising.
> 
>>> In my testing, your changes have a large effect on --short performance.
> 
> Yeah, I use --short too, and I can feel the difference.
> 
>>> As a reference point, we also have --log-format=long, the time didn't
>>> change with or without your patch.
>>> $ time ./bzr log --log-format=long > /dev/null
>>> real    0m4.501s
> 
> Yeah, the only chance of changing --long would be by avoiding 6500
> hasattr calls.

Probably an improvement, but not necessarily a lot.

> 
>>> Anyway, its already merged, but kudos for the improvement. (2.3x faster)
> 
> Thanks.
> 
> Aaron

Yeah, but the biggest thing at this point would be to get our startup
time down. I've worked out several ways to do this. We just have to
decide what evilness we consider acceptable.

Heck, with my 'bzr service' plugin, we could get the startup time down
to low ms. But that probably isn't a general solution.

In testing, time /bin/echo '', takes 7ms on my machine, so that is
probably our minimum process startup overhead time. (python -c '' is
27ms, as is python -c 'import os,sys')

With the service running, 'time client' runs in 10ms, 'time client
--version' is 25ms, and 'time client root' is 50ms.
Plain bzr.dev at this point is 450ms. Definitely a noticeable difference.

I just updated my demandload branch to allow for 'demand_compile'
objects. Basically, just a class which delays compiling until we
actually get a request.
With all my hackery I can get 'bzr rocks' time down to around 100ms. And
'bzr root' time down to 215ms.

By the way, this patch makes the output of --profile-imports a lot
easier to read:

=== modified file 'profile_imports.py'
--- profile_imports.py  2006-06-23 19:09:04 +0000
+++ profile_imports.py  2006-06-23 19:10:01 +0000
@@ -80,6 +80,8 @@

         # indent, cum_time, mod_time, name,
         # scope_name, frame_name, frame_lineno
+        if info[0] == 0:
+            out_file.write('\n')
         out_file.write('%5.1f %5.1f %s %-35s\t@ %s:%d\n'
             % (info[-1]*1000., mod_time*1000., '+'*info[0],
                cur[1][:40], info[1], info[2]))

One problem with demand_compile is that it switches from having one file
which takes 30ms to load, to having 5 6ms compile times spread around
based on the first use. So while it looks like you fixed the big
penalty, you really just spread it around.

(demandloaded 'bzr --no-plugins --no-aliases rocks' is down to 70ms).

John
=:->

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060623/ca03f0f4/attachment.pgp