Analysis: Time spent in committing in a checkout

[John Arbash Meinel]

Robert Collins wrote:
> On Fri, 2006-09-08 at 20:35 -0500, John Arbash Meinel wrote:
>> I'm trying out using a checkout of a Launchpad branch. My ping time is
>> approx 100ms, so 'bzr commit' is rather slow. I thought I would
>> investigate what steps are involved, which might shed some light as to
>> why that is. I did a --lsprof of a simple commit which only changes one
>> file. And then tried to break down what was going on.
>>
>>
>> To start with --lsprof says that the total time spent is 24s.
>>
>>
>> From the top-down, we spend all that time in commit(), with this being
>> the basic breakdown:
>>
>>   8.65s bzrlib.repository:243(fetch)
>>   6.77s bzrlib.commit:349(_check_bound_branch)
>>   3.31s bzrlib.repository:2028(commit)
>>   1.80s bzrlib.commit:391(_cleanup)
>>   1.18s bzrlib.commit:444(_gather_parents)
>>   0.69s bzrlib.branch:310(last_revision)
>>
>> There is also:
>>   1.00s bzrlib.decorators:48(write_locked)
>> But unfortunately, our @needs_write_locked stuff makes it difficult to
>> follow certain code paths.
> 
> FWIW, I find it easy enough to follow, because for 4 needs_write_lock
> calls, if theres 4 children, there are 4 outer callers which match with
> the children.

Well, in something like this, there are frequently a lot of outbound
calls from needs_write_lock, including multiple calls to the same
function. (For something like commit, it calls write_lock_ed for both
local and remote, etc).

It isn't terrible to find as long as you track back into the source
code, and you know what functions might require write locks. But it
isn't as easy as just searching for the function in the lsprof output.

In this specific case, there are 22 calls into 'write_locked', spread
out over 23 different children functions. (Some of those are lock_write
and unlock, on workingtree, branch, and repository).

So in a real command, the N-way that goes through 'write_locked' gets
pretty big.

> 
> 
>> I also wonder if we could get Paramiko to allow some of this stuff to be
>> pipelined. Like when taking a lock, a whole lot of that could be a big
>> pipeline that can be stalled at any time, but we don't really need to
>> sync until we read the contents of the final lock directory. I realize
>> opening a file in sftp is a required sync point, since you need the file
>> handle before you can read or write anything.
> 
> It would be nice. In fact optimally it would look something like:
> send mkdir branch-lock-temp
> send create-file branch-lock-info
> send mkdir repo-lock-temp
> send create-file repo-lock-info
> <-waiting->
> either send branch-lock-info-content
>  or send repo-lock-info-content
>  followed by a file-close immediately
> when we send the repo-lock-info file-close send rename of the lockdir
> immediately.
> when we send the branch-lock-info file-close if the repo lock has not
> failed, send rename immediately.
> <-waiting->
> both complete: locked
> branch completes, repo fails: no locks taken at the object level (undo
> the branch lock)
> repo completes, branch fails: repo is locked, branch fail, and will undo
> the repo lock)
> 
> But while this is trivial to write in an event driven fashion, I dont
> see how to write it cleanly in bzr today.
> 
> -Rob
> 

Well, even in an event-driven fashion, things get a little confusing.
Because things are async, I don't know that you can guarantee that you
will get the 'mkdir success' event before you get the 'create-file
success' event.

And in any case, if we go towards a smart server, all of this can be as
simple as a single request to lock the branch. Though I don't know how
rich the XML RPC is intended to be for a smart server branch.

(Actually, for smart server, you probably would work at an even higher
level, and just say that you want to do X with these revisions, and the
server can tell you if that is possible)

John
=:->


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060911/6a7cef44/attachment.pgp