[MERGE] add performance tweaks.
michael at ellerman.id.au
Mon Jun 12 02:12:15 BST 2006
On 6/11/06, Robert Collins <robertc at robertcollins.net> wrote:
> On Sun, 2006-06-11 at 22:34 +1000, Robert Collins wrote:
> > I'm calling this a set of tweaks rather than major improvements - I see
> > a consistent 300ms improvement with them:
> > test_one_add_kernel_like_tree OK 5056ms/ 9680ms
> > to
> > test_one_add_kernel_like_tree OK 4736ms/11013ms
> > In summary this removes a lot of basename calls, switches the directory
> > walking to use walkdirs, adds slots to inventory objects - without doing
> > the complete workover that Jan is working on (I hope this patch doesn't
> > conflict with his work, but the constructor was starting to show up in
> > my profiles).
> > Key hotspots remaining:
> > * encoding the output (2/18)
> > * fileid generation (1/18)
> > * ignore checking (3/18)
> > * serialization. (8/18)
> > My replacement xml serialiser halves the serialisation cost, but is not
> > compatible with 0.8 [unless the bug 49224 bugfix is applied], and I've
> > not yet profiled the cost of doing entity replacement to make a
> > ascii-safe bytestream.
> > The ignore checking - I think dropping the number of defaults is
> > probably the right thing to do. Checking a modest number (say
> > CVS, .svn, .hg, .darcs, *~) by default, or allowing users to shrink the
> > default, should allow this to be fast where needed - I suspect we will
> > have a high time investment to get significantly better than we have
> > now. (We can get 20% saved by using a non-matching form of the regex by
> > default).
> > Fileid generation remains somewhat high. I plan to play with having a
> > generator and passing in the fileid to make_entry.
> > 10824 0 1.1408 0.2294 bzrlib.inventory:1222(make_entry)
> > +10824 0 0.4944 0.2212 +bzrlib.workingtree:134(gen_file_id)
> > +10240 0 0.3935 0.1611 +bzrlib.inventory:650(__init__)
> > +584 0 0.0235 0.0092 +bzrlib.inventory:540(__init__)
> > Lastly, the encoding : we pay a surprisingly high cost here:
> > 10824 0 2.5402 0.3942 bzrlib.add:66(__call__)
> > +32472 0 1.8984 0.9094 +codecs:174(write)
> > I'm going to reduce the 3 times call multiplier and see if that reduces
> > the visible overhead. If that doesn't, we may want to make quiet the
> > default ;).
Yeah it hurts, and for enormous trees I'm not sure it's really useful.
I have this in my todo list from yonks ago:
- bzr add -q of a kernel tree takes ~30 seconds
- bzr add of a kernel tree takes 1.06m
- perhaps bzr add should just print the top level things that were added
and mention that directories are being recursed into
- then -v can select the current behaviour and -q can print nothing
But never got around to coding it.
More information about the bazaar