[MERGE] Add simple Storage object

[Robert Collins]

On Sat, 2008-02-09 at 03:03 -0500, Aaron Bentley wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Robert Collins wrote:
> > I agree there is a difference in clarity here. But - pack and knits
> > don't have the same more for storage any more than weaves and knits;
> 
> Pardon?

I can't remember what 'more' should have been. Either model, or needs or
something.

I've trimmed the rest of the message because I think I can express what
has been niggling at me now. And its all about layers.

> I think it's desirable to preserve the separation of concerns that
> Stores gave us.  Repositories already do plenty, and they have always
> mediated their access to the underlying storage through Stores or
> Versionedfiles.  I think adding new responsibilites for talking to
> indices, requesting raw data from the underlying storage, and creating
> fulltexts from faw data is going to make the cleanliness problem worse,
> not better.

Going back to basics we have:
N disk format probes
Each probe can own a control directory, and once a matching one is found
it becomes a factory object to give us a repository.
This is Format->Repository

The Repository is not necessarily a single object; currently the
factories give us 4 or so main objects, all of which act as factories to
give us many other objects - like file contents, revisions etc.

We have problems with latency. Specifically we have a single threaded
blocking programming model in bzr, and that means that any action we
take that causes any form of IO (even memory IO in a sense) defers all
other actions until it completes. We're working around this by creating
both disk structures and programming structures that allow many related
operations to be expressed at the same time (such as via readv on the
network) reducing the amount of time that we spend idle.

This strongly suggests to me that over time we're going to have less and
less use for VersionedFile, and instead be wanting 'VersionedFiles' or
some such. Aaron and I have talked on IRC about this sort of thing.

One interesting question is whether we want to get an all-in-one
facility, or keep the separation between revision, inventory, files,
signatures. Keeping that separation obviously implies a 4 round trip
minimum for many operations, but OTOH they go have quite different index
and storage needs.

The primary point though is that a structure in python which has lots of
separate objects is very expensive for us because of the IO round trips
that this drives versus single operations on some sort of grouping
object. 'revision_trees' vs 'revision_tree' on Repository is an example
of this.

Now, given that the factory to make a repository can make an arbitrary
structure, we can use different objects to make the job of programming
easier; such as Index vs Pack vs Repository vs Store etc. We can choose
to put methods on Repository or on stores.

I think the current stores _force_ a separation between all the things
we serialise and make it trickier to add new things; the separation of
concerns we have there is also flawed at the moment - we conflate
physical storage with serialisation with model.

I think there are a number of basic things that are different between
Repositories in an interesting way for clients of Repository.
(Interesting vs being arbitrary differences):

What 'optional' features do they support: subtrees, rich-roots,
signed-revisions-0.8-style....

Whats the best way to work with this repository: text-iteration-order,
inventory-iteration-order, ask-for-annotated-always,
ask-for-deltas-always?

Now as we've evolved the bzrlib api we've tried to find precise API's
that have let us ignore these differences as much as possible (such as
the 'look at the lines in the inventory xml 'hack').

Internal to Repository however, there are many more interesting
differences:

How are revisions serialised
How are inventories serialised
All the optional features listed above
How are byte-sequences stored, for (serialised revisions, serialised
inventories, texts, signatures)
How are annotations generated

We've done a lot of work recently to change how byte sequences are
stored - packs, the desire to get a new disk delta format, and going
back further knits, weaves and JBOF storage. But currently the higher
level code - the 'Stores' are layered individually on top of byte
sequence storing api's.

We have a lot of work TODO though on inventory serialisation, revision
serialisation, annotation cache storage.

Without trying to predict what the best outcome will be, I think what we
need to do is to alter the current internal layering, so that Pack
repositories have the least possible friction, and put in thunk layers
in place for all the other repositories.

This is the thing I feel most strongly about here- its that we 'fitted'
knits into the weave code, and then 'packs' into the knit code, and
likewise 'subtrees' largely got fitted into the 'non-subtrees' code.

To rephrase, I think we're not being aggressive enough about pivoting
our API's when a new core feature comes in to make the focus be on that
core feature, with the older stuff getting out of its way.

Now, I may be violently agreeing with Aaron in the following
suggestions, take that as me not getting what he was saying, or
something - or I may have quite a different structure in mind here.

I think the following would work very well for packs, and not be
significantly hard for Knits etc.

I propose an interface UnifiedByteStore. This is responsible for:
storing, indexing and retrieving byte sequences with names that are a
key tuple like ('text', fileid, revisionid), or ('revision', revisionid)
or ('signature', revisionid) or ('inventory', revisionid).
This /is/ RepositoryPackCollection on the Pack repository format, with
perhaps a couple of tweaks.

The public interface will be some variation on:
---
add_bytes(key, value) -> hash
get_bytes(keys) -> iterator of byte_sequences
add_stream(key, object_with_read_close_methods)
get_streams(keys) -> iterator of objects_with_read_close_methods
---
in particular I expect the versionedfile.add_lines keyword parameters
will be desirable for performance, but there is a good chance we can
avoid pushing them down this far. Time will tell.

Packs need write-group management to generate good packs, knits need
file level (or repository wide) locking for data integrity, weaves need
repository wide locking. I think its reasonable to want locking in the
API but it will for most existing implementations mean holding a
reference to a Repository object, rather than being able to be layered
in a single direction.

I also don't think we should require all implementations of these stores
to have locks or write groups - the only bits we need to be able to
substitute arbitrarily is in code that can be given /prepared/ byte
stores for use. That is, most api's on Repository, or on the
revision_store etc, have the opportunity to call private methods on the
byte store they are working with. This makes sense if other things
change from Repository to Repository - and they do. 
We may alternatively choose to have some subclassed and refined
interfaces that add some or all of:
lock_read
lock_write
is_write_locked
is_read_locked
unlock
start_write_group
abort_write_group
commit_write_group
is_in_write_group

Using this we then split out the current stores into two interfaces -
one that takes objects to and from bytes, and one that purely stores
bytes.

The current bzrlib.store code should end with just byte stores.

Graph queries for Pack repositories can be done via private methods on
the unified store, likewise for Knits etc. I suggest this because graph
relationships between keys is not appropriate for a bytestore, but some
stores will have to have indices to perform there core function - so the
disk structure we have today is good - its a matter of how to expose it.

Aaron and I both want to see a iter_files_bytes that rocks really hard
on Packs. I think that the stuff I describe above could form a useful
part of it, but that migrating the code base from VersionedFile to
VersionedFiles, which can be done cleanly without the changes I have
described here, is likely the easiest way.

Here's how I see a VersionedFiles coming about:

 - we create a VersionedFiles interface that has only the methods we
really commonly use from VersionedFile. It takes (file_id, revision_id)
keys in its API's.
 - initial implementation of VersionedFiles just groups all the things
its asked to do by file_id, then:
   for file_id, versions in fileid_groups:
       vf = self._get_fileid_callback_id(file_id)
       yield vf.operation(versions)

or whatever.

 - we'll want inventory, revision and signature 'knits' to stay as
single-key things, but we could do a single VersionedFiles instance on
Repository which has a callback to convert 'revisions' to the revisions
knit etc.

 - then, when all our commonly used internal code is using the
VersionedFiles interface rather than VersionedFile, we invert the logic.
That is, we migrate the code within knit.py (we can leave weaves using
the thunk layer I described above indefinitely) so that the delta
combining code, component-mapping code etc etc all work on arbitrary key
tuples. (its nearly already done in fact). At this point we have
VersionedFile being a thunk from 'version' style keys to
(self._my_prefix, 'version') keys and calling into the more generic
layer. *as a result of doing this* the pack index layer will become
simpler, as its already doing that exact thunk, so it can essentially
become pass through, improving performance and batching operations
across fileids with no explicit changes.


-Rob
-- 
GPG key available at: <http://www.robertcollins.net/keys.txt>.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20080211/25d831cf/attachment-0001.pgp