OS locks must DIE

[Martin Pool]

On Wed, Mar 26, 2008 at 1:36 AM, Aaron Bentley <aaron at aaronbentley.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>
>  Alexander Belchenko wrote:
>  > ... So, subject says all I think about OS locks.

https://bugs.launchpad.net/bzr/+bug/206406

So regardless of os locks, if one bzr process is holding a write lock
on the dirstate and also trying to take a readlock, that is a
different separate bug, and one that may be relatively straightforward
to fix.

However, I heartily agree that (our use of) OS locks should die; they
are annoying in more situations than just Windows, including varying
across platforms and filesystems, sometimes not working at all on NFS,
blocking doing workingtree updates over a transport, unnecessary
locking between say commit and diff, etc ..

The reason we use them is because they allow readers to exclude
writers and vice versa, which cannot be done over our general
transport interface or through Lockdirs.  (Aside from anything else,
it would require readers to have physical write access.).  The reason
for this, as Aaron says, is that we wanted to be able to update the
dirstate in place when changes are made, rather than rewriting it
completely.  And the reason for that is that we want operations that
affect only one file not to take time proportional to the whole tree,
as would be needed to write a whole new dirstate.

There is a minor complication in other schemes that on Windows we
cannot rename into place while a file is open for reading, but I think
it's not insurmountable - basically you just report on it as lock
contention and keep trying.  Or it may be possible to move the in-use
file away.

However, I believe we don't actually take advantage of this at
present: we always rewrite the whole thing.  So sadly the os lock is
not helping anything other than giving room to change things without
doing a new format.

So it seems like our options here are:

 * keep the same format (or nearly the same), remove the os lock, on
updates write a whole new file and rename it into place; this fixes
this bug with probably the smallest amount of code and no regressions;
this format will always take O(tree) for any local update
 * journal over time as Aaron says; guard the dirstate with a semantic
lock but write new changes into new files; collapse them into one
large one periodically
 * journal over space, with one file per directory: this may perform
poorly wrt io locality when reading the whole tree

The second of these sounds good, but it also looks like an
optimization on the first, and I would like to try that first.

-- 
Martin <http://launchpad.net/~mbp/>