[MERGE] add performance tweaks.

[Michael Ellerman]

On 6/11/06, Robert Collins <robertc at robertcollins.net> wrote:
> On Sun, 2006-06-11 at 22:34 +1000, Robert Collins wrote:
> > I'm calling this a set of tweaks rather than major improvements - I see
> > a consistent 300ms improvement with them:
> >
> > test_one_add_kernel_like_tree OK  5056ms/ 9680ms
> >
> > to
> >
> > test_one_add_kernel_like_tree OK  4736ms/11013ms
> >
> > In summary this removes a lot of basename calls, switches the directory
> > walking to use walkdirs, adds slots to inventory objects - without doing
> > the complete workover that Jan is working on (I hope this patch doesn't
> > conflict with his work, but the constructor was starting to show up in
> > my profiles).
> >
> > Key hotspots remaining:
> >  * encoding the output (2/18)
> >  * fileid generation   (1/18)
> >  * ignore checking     (3/18)
> >  * serialization.      (8/18)
> >
> >
> > My replacement xml serialiser halves the serialisation cost, but is not
> > compatible with 0.8 [unless the bug 49224 bugfix is applied], and I've
> > not yet profiled the cost of doing entity replacement to make a
> > ascii-safe bytestream.
> >
> > The ignore checking - I think dropping the number of defaults is
> > probably the right thing to do. Checking a modest number (say
> > CVS, .svn, .hg, .darcs, *~) by default, or allowing users to shrink the
> > default, should allow this to be fast where needed - I suspect we will
> > have a high time investment to get significantly better than we have
> > now. (We can get 20% saved by using a non-matching form of the regex by
> > default).
> >
> > Fileid generation remains somewhat high. I plan to play with having a
> > generator and passing in the fileid to make_entry.
> >  10824 0  1.1408      0.2294   bzrlib.inventory:1222(make_entry)
> > +10824 0  0.4944      0.2212   +bzrlib.workingtree:134(gen_file_id)
> > +10240 0  0.3935      0.1611   +bzrlib.inventory:650(__init__)
> >   +584 0  0.0235      0.0092   +bzrlib.inventory:540(__init__)
> >
> > Lastly, the encoding : we pay a surprisingly high cost here:
> >  10824 0  2.5402      0.3942   bzrlib.add:66(__call__)
> > +32472 0  1.8984      0.9094   +codecs:174(write)
> >
> > I'm going to reduce the 3 times call multiplier and see if that reduces
> > the visible overhead. If that doesn't, we may want to make quiet the
> > default ;).

Yeah it hurts, and for enormous trees I'm not sure it's really useful.

I have this in my todo list from yonks ago:
  - bzr add -q of a kernel tree takes ~30 seconds
  - bzr add of a kernel tree takes 1.06m
  - perhaps bzr add should just print the top level things that were added
    and mention that directories are being recursed into
  - then -v can select the current behaviour and -q can print nothing

But never got around to coding it.

cheers