lock free dirstate - prototype

[Martin Pool]

2009/9/29 Robert Collins <robert.collins at canonical.com>:
> On Tue, 2009-09-29 at 06:41 +0100, Martin Pool wrote:
>> 2009/9/29 Robert Collins <robertc at robertcollins.net>:
>> > lp:~lifeless/bzr/dirstate2 has working code.
>
> Updated now, --development-rich-root will use this tree format. No 'bzr
> upgrade' support for it yet, so the easiest way to test is' bzr init
> --development-rich-root; bzr pull <bzr.dev>' (and you can do that
> sensibly in a shared repository).
>
> Some open technical things I'd like folk reviewing it to consider:
>  - should we perhaps use a regular lockdir to arbitrate 'changing the
> current pointer'. I don't think it makes things safer, but I may be
> wrong.

Creating and manipulating directories is reportedly pretty slow on
Windows (and maybe other places) so it would be worth avoiding any
unnecessary lockdir-type io.  I'd rather look at taking any useful
features from lockdir into the core format of this.  For instance,
it's probably not safe to rely on any particular behaviour for
rename('newfile', 'file') but you can do a directory-renaming dance.

However, presumably we do need a semantic wt lock - how would that fit in?

>  - how to deal with two writers with the same new content / ABA state
> transitions combining with slow-readers that delete what they think is
> unlinked content, but actually has been relinked. I'm thinking that slow
> readers need to acquire the current file ownership before they unlink
> anything.  Two writers with the same new content is harder; one will win
> renaming current, and one rolling back will delete the others content. I
> think we need some randomness/uniqueness in either the filename or the
> content. (e.g. put a pid in there).

I'm not sure I understand.  I don't think readers should unlink files
- do they really need to.  And won't simultaneously writers be blocked
out by the exclusive semantic lock?

>> Does this write a whole new copy on each update, or incremental files?
>
> Full copy. Theres a bug of stuff we /can/ do in a new tree format; I've
> picked a single conceptual bug which has many ramifications and tried to
> solve that. Some other ones I've not put any coding effort into are:
>  - write less on partial updates
>  - only store the left most parent
>  - persist an id2path map

and, for me, getting away from any need to write on logical read
operations would be great.

>> I don't want to gate development or merge of this work into a
>> development format, but I do think that before we publicize it we must
>> get the formats ui to be better.  The whole issue of understanding
>> what format to use, why things are incompatible, or how to upgrade
>> them is messy.
>
> Its no better or worse than it was for last few releases. I think we
> should improve the formats UI. I'm glad you don't want to make this
> merging this improvement conditional on improving the formats UI, but I
> really don't understand why telling folk about it should be conditional
> on improving the UI *either*.
>
> We have a requirement from Mark to fix the way we manage and announce
> development formats, and I think that taking this new format *out of
> development* must be gated on that; that is different to the format UI
> though - though perhaps that is the actual thing you meant?

Mark, AfC and others have raised the troubles caused by format
transitions.  We should not ask people to go through this again.

I didn't mean "before posting absolutely anything about it" but I do
think we should improve it before putting it on track to be default,
or before asking for widespread testing.

-- 
Martin <http://launchpad.net/~mbp/>