Pulling multiple branches in the same repository

[John A Meinel]

Martin Pool wrote:
> On 08/05/2006, at 5:57 PM, Erik Bågfors wrote:
> 
>> Hi.  I pulled bzr.dev and bzr.0.8 at the same time, into the same
>> repository, and I got the following error message.
>>
...

>> I understand that there is a locking issue here, but I see two
>> problems with this
>> 1) The error message is really bad.  Will confuse a user
> 
> How about something like this:
> 
>   Could not acquire repository lock on /home/bagfors/src/bzr/
>   Currently held by bagfors at zyrgelkwyt process 123, "bzr pull"
> 
> Also we should show something while waiting for the lock.

I do think we need to improve the error message.

> 
> Secondly we should fix the problem causing the UserWarning.  It's useful
> for developers to know about these and fix them up but I'm not sure they
> should be visible to users.
> 

If you wanted, for a released version of bzr, the first thing you could
do is filter out all user warnings. I don't know if bzr is release
worthy enough yet.


>> 2) When adding multiple developers on multiple branches, I think not
>> allowing two of them to work on different branches (at least) at the
>> same time, will give problems.  This might mean that bzr has problems
>> scaling up to a large number of users on the same repository (and
>> branch for that matter)
> 
> Yes, this will need more work.  There's no conceptual reason why we need
> to hold a lock on the repository during insertion of new revisions:
> we're not changing existing information and there is no reason why
> parallel inserts should conflict.  (By contrast we do need to lock
> branches during commits/pulls, because only one attempt should win.)

Other than that 'append' isn't really atomic. Since we have to update
both the .knit and .kndx files, it seems like it would be reasonable to
just lock the individual files your are changing. But we don't do
per-file locks (since we are avoiding os level locking).

I would also say that even if we did per-file stuff, inventory.knit and
revisions.knit are going to be a bottleneck.

> 
> On local storage we should be OK to have multiple writers going into the
> same knit but on pessimistic assumptions about sftp this will not be
> safe.  That may be something to consider for a future format change.
> 
> --Martin
> 

On local stuff, we could again use OS level locks. The other issue,
though, is that if you are pulling from 2 sources, you could easily get
the same revisions 2 times. (If you were pulling bzr.dev and
bzr.dev.0.8, for example).
When both of them open the repository, they won't see revision 'foo'
yet, so they will both download it, and add it.

Can our current code handle repeated revisions in the knit? I don't
really know. Probably it *should*.

All in all, though. It seems much safer to serialize access to the
repository. There are going to be some scaling issues, and we'll need to
address them. I know sqlite handles it by actually appending everything
and using its write ahead log, but not actually grabbing the lock until
the .commit() step.
In our world, that could be something like uploading each revision into
a new directory, and then the final commit() is changing the index file
to point to all of these new files.

I've never worked somewhere that had a really high churn, such that lock
contention was a big issue. If a group was going to be really really
concurrent, they could opt for only using standalone branches. Though if
you have concurrency issues, you probably have a lot of revisions, and a
repository is very nice to manage that load.

In some ways a smart server could help, because it could make sure that
only one memory representation of a given knit was active at a time.
Because only one process is trying to hold the lock.

As such, I don't think we really need to deal with this until it becomes
a big issue. Though I suppose we don't want a design that won't ever
scale either.
I'm guessing one of our biggest issues is trying to maintain a
serverless system. But I think that is going to be a strong point for us
too. "Want to host at Sourceforge, no problem, you just need sftp
access, you don't have to wait for SF to install bzr support." Though
there is always the supermirror. :)

John
=:->

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 249 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060508/37537774/attachment.pgp