[RFC] Faster commit - more progress and call for input

[Ian Clatworthy]

For those interested, my latest code for speeding up commit is provided
in the attached bundle. It has a noticeable impact: 50% quicker for
small commits and up to the same for the initial commit.

While not 100% ready for merging, I think this is getting close now to
what I hope to see in 0.17. I fully expect some rigorous
debate/discussion on some of the proposed changes though so this is
marked as 'RFC', not 'MERGE'.

In summary, the changes do the following:

1. buffer the report of commit messages until the end
2. remove calls to dirstate._validate that shouldn't be in production
   code
3. remove locking around some low level dirstate calls, namely
   id2path, kind and is_executable
4. minor code clean-ups in commit.py

In discussions with various people, the first of these appears the most
contentious. I had been blaming progressive writes to ~/.bzr.log for
slowing down the initial commit so much. It now seems that
gnome-terminal is at least as guilty. Here are some times for checking
in a Mozilla tree of 55k files:

Progressive writes to stdout: 25m52s
Progressive writes to stdout > /dev/null: 15m33s
All writes to stdout but at the end: 16m41s
All writes to stdout but at the end > /dev/null: 16m24s

Writing to ~/.bzr.log adds 30 seconds to these times: bad but nothing
compared to gnome-terminal doing its thing. As shown above, the fastest
possible performance is progressive stdout writing being redirected to
/dev/null. Even so, I remain cool on progressive writing. As well as
making the progress bar a lot less friendly, something still feels wrong
to me about reporting things are being 'added' during logically the data
collection phase of commit. I'm putting on my fire-retardant suit in
expectation of the flames expected in saying 'reporting at the end is
the right thing because commit is atomic'. Fire away!!! :-)

Regardless of the outcome of that debate, the more important bit is the
speeding up of small commits. The bundle has some read locks commented
out. The production code to be merged probably needs those comments
removed and explicit tests/asserts that a read lock has been taken. Can
someone please confirm that and point me at the preferred pattern for
that if any?

I'd like to thanks Aaron, John and Robert for some off list emails and
discussions explaining some of the complexity behind commit I didn't
understand. We'll get that information consolidated and available so
more people can understand the design tradeoffs and challenges that
commit needs to make. The good news is that the large improvements
delivered by this patch are just the low hanging fruit. Commit can be
quite a bit faster still, though we won't have time for more advanced
restructuring of commit in 0.17 in all likelihood.

Looking forward to everyone's comments,
Ian C.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: commit-cleanup-and-faster2.patch
Type: text/x-patch
Size: 57397 bytes
Desc: not available
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20070601/e7dd5a3c/attachment-0001.bin