[PACKS] Pack disk format changing, updated dogfood instructions.

[Robert Collins]

If you don't know about the pack repository format, please skip down to
'Pack repositories' in this mail. If you are already aware of it and
dogfooding, please read the entire thing.

The pack repository format is being changed to have no cached
annotations in the knit records used within the pack.

This saves space and time for the majority of operations, at the price
of making 'annotate' slower, so the first release is probably not
suitable for a default format.

I expect to make this change in revno 2784.

Pack repositories
-----------------

Pack repositories offer some significant benefits over knit repositories
for dumb protocol access. The key benefit is the ability to cap the
total latency vs knits where each separate file causes additional round
trips. We also reduce the total VFS operations required - we no longer
need to append to files, and that makes both SFTP and FTP support
easier.

Pack repositories also reduce the overhead required during commit
operations by removing the mandatory annotation cache from knits; we
plan to reinstate an optional annotation cache in a later edition of the
pack repository format.

Status
------

Pack repositories are an experimental format aimed at 0.92. They
currently work fully and are undergoing performance tuning - part of
which is in the repository format, and part of which is the use of the
repository API from the rest of bzr - working through the size(history)
operations we do and removing them.

Martin is working on the inventory layer, with the goal of integrating a
more scalable inventory into the pack format. The two things in
combination - the performance tuning, and the plans to change
representation, mean that the disk format for packs is subject to change
- all such changes will be announced here with instructions for
migrating test repositories.

Dogfooding
----------

In order to find rough edges, I'm using the pack format full time myself
at the moment. There is a copy in knit format at 
http://people.ubuntu.com/~robertc/pack-repository.knits. This is kept
reasonably up to date, but the very latest code is only available via
the pack-based format branch at 
http://people.ubuntu.com/~robertc/baz2.0/repository. I update the knits
copy when format-related changes occur, but not for e.g. performance
tweaks.

When dogfooding, be aware that folk who are not also dogfooding will not
be able to read pack based repositories. You may need to keep some
branches in knit format to interoperate. (I have kept my integration
branch like that :)).

To dogfood:
 - pull a copy of the pack supporting branch - 'bzr branch 
http://people.ubuntu.com/~robertc/pack-repository.knits packs.knits'
 - Then you can create a branch or repository in experimental format:
   '$ packs.packs/bzr init --experimental my-pack-branch'
   or
   '$ packs.packs/bzr init-repo --experimental my-pack-repo'
 - Now just pull in content from a knits branch and bzr will convert the
data on the fly:
   - for a branch: 'cd my-pack-branch && ../packs.packs/bzr pull URL'
   - for a repo 'packs.packs/bzr branch URL my-pack-repo/PATH'

***NOTE***: The bzr.dev repository has an knit delta/index mismatch with
the inventory content pointer that causes pull to not grab enough data;
this is currently being addressed in bzr.dev. So don't test packs on bzr
itself unless you seed your copy by wgetting the contents of
http://people.ubuntu.com/~robertc/baz2.0/.bzr/repository. 'bzr
branching' from my pack repository will *NOT* work because of this data
mismatch.

I have put the packs.knits directory on my path, so that I use that bzr
all the time, but this isn't a requirement.

I will announce all changes to the format with a [PACKS] email to the
list, so you can take whatever action is needed to upgrade. In general
this will be straight forward.

Upgrading an existing dogfood repository/branch
-----------------------------------------------

1). Convert it to knits:
$ bzr upgrade --dirstate, or 
$ bzr upgrade --dirstate-with-subtree (if you are using the subtree
format already - but don't do this if you are not, or do not know what I
am referring to).
2) Get the latest copy of the packs code (see above)
3) Convert it to packs:
$ bzr upgrade --experimental or
$ bzr upgrade --experimental-subtree (as above, only do this if you are
already using a subtree format).


Known issues
------------

At the moment remote access will read the entire text index rather than
just the needed contents; this will be fixed in a future change.

-Rob
-- 
GPG key available at: <http://www.robertcollins.net/keys.txt>.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20070926/882a4764/attachment.pgp