2.0 upgrade experiences

[Neil Martinsen-Burrell]

On 2009-08-11 16:36 , Robert Collins wrote:
> On Tue, 2009-08-11 at 16:20 -0500, Neil Martinsen-Burrell wrote:
>
>> 1.  Running ``bzr check`` in a standalone branch that was originally in
>> format 1.9 gives me a load of output like:
>>
>> [...]
>> Missing inventory {('TREE_ROOT',
>> 'svn-v2:1179 at 8f9f66e6-09d4-0310-8052-96bf1e615463-')}
>> Missing inventory {('TREE_ROOT',
>> 'nmb at wartburg.edu-20090417201757-chtr18w6ddnpcal3')}
>> Missing inventory {('TREE_ROOT',
>> 'nmb at quaggy.local-20071106145233-vr33wd75mx91n97l')}
>> Missing inventory {('TREE_ROOT',
>> 'svn-v2:724 at 8f9f66e6-09d4-0310-8052-96bf1e615463-')}
>> Missing inventory {('TREE_ROOT',
>> 'nmb at wartburg.edu-20071109170840-o838ndsrs1prwl1m')}
>> Missing inventory {('TREE_ROOT',
>> 'nmb at wartburg.edu-20071115142153-pkeip1evaq353ubc')}
>
> What is the format of your source branch? you say 1.9, but those errors
> (and the svn revids) make me think it was 1.9-rich-root.

This is the output of ``bzr info -v`` in the original branch::

nmb at rameses[~/repos.bak/career]$ bzr info -v
Standalone tree (format: 1.9)
Location:
   branch root: .

Related branches:
   parent branch: /Users/nmb/Career-new

Format:
        control: Meta directory format 1
   working tree: Working tree format 4
         branch: Branch format 7
     repository: Packs 6 (uses btree indexes, requires bzr 1.9)

Working tree is out of date: missing 13 revisions.

In the working tree:
         93 unchanged
          0 modified
          0 added
          0 removed
          0 renamed
          3 unknown
         11 ignored
         10 versioned subdirectories

Branch history:
        154 revisions
       1366 days old
    first revision: Mon 2005-11-14 19:37:54 +0000
   latest revision: Wed 2009-07-15 08:39:10 -0500

Repository:
        157 revisions

I don't think that these branches have ever been rich roots, even though 
they were conversions from SVN using bzr-svn (at some old version).

>> It is not clear from ``bzr help check`` whether or not a missing
>> inventory for the tree root "indicates a problem".
>
> Its a problem - any output from check other than summary data is a
> problem. We should make this clearer.
>
>> 2. The space savings is not as big as my wildest dreams (This is a
>> standalone branch, even though the parent directory is called "repos".)
>>
>> nmb at rameses[~/repos/career]$ du -sk .
>> 6848	.
>> nmb at rameses[~/repos/career]$ du -sk ../../repos.bak/career
>> 7028	../../repos.bak/career
>
> You will have content in .bzr/repository/obsolete-packs that will get
> cleaned up when you make subsequent commits. We do this to deal with
> various edge cases without requiring fsync/fdatasync operations (which
> are either a) expensive, or b) [on XFS and MacOSX] don't work).
>
>> When I thought that a pack operation might clear up additional space, I
>> got the following error:
>
> bzr auto packs on conversion, there is no need to run pack. The general
> rule for running 'bzr pack' is 'do not run this command'.
>
>> nmb at rameses[~/repos/career]$ bzr pack
>> bzr: ERROR: Pack '6ffbb186d5539f27e54609f64e5fb92e' already exists in
>> <bzrlib.repofmt.groupcompress_repo.GCRepositoryPackCollection object at
>> 0x126c3d0>
>>
>> Is that error serious?
>
> Its a known bug with 2a, and a fix is in-progress at the moment in a
> branch of Andrews.
>
>>    Does it mean that I cannot pack this branch
>> further?
>
> It means that unlike 1.9 bzr is willing to attempt a pack when there is
> only one pack present, and that it doesn't cleanly report the case when
> the repository is already fully packed.
>
>>    Are there obsolete packs still sitting around somewhere?
>
> Possibly, you can look in .bzr/repository/obsolete_packs. The contents
> of this directory can be removed (but be sure that your filesystem has
> recorded the bzr dir to disk before doing the removal). Leave the
> directory in place though.
>
>>   Can
>> these be gotten rid of somehow?  This suggests that the space savings
>> could be quite a bit larger:
>>
>> nmb at rameses[~/repos/career]$ du -sk .bzr/repository/obsolete_packs/
>> 2260	.bzr/repository/obsolete_packs/
>>
>> What processes can be used to empty out these obsolete packs?  When is
>> that process safe?
>
> Once your new packs have hit disk, which is _extremely_ hard to tell
> reliably. On some protocols we simply can't tell (SFTP/FTP/HTTP[S]), on
> others asking for them to hit disk causes very expensive operations
> globally rather than locally to that file (ext3), and on yet others the
> operations required to make it hit disk are stubbed out (because tools
> that didn't really need to ask for things to hit disk did so so often
> the operating system vendor stubbed out the call and added new separate
> calls for programs that really want to be sure data has hit disk).

I see the reason that such a directory is necessary.  I guess my big 
question is whether a human can ever know that it is safe to remove the 
contents of this directory, or if I need to wait for Bazaar to choose to 
remove those files at a later date.

>> 3.  In a different branch, ``bzr check`` tells me that I have "sha1
>> mismatch"
> ...
>> errors.  How does one go about fixing these SHA1 errors?  What will be
>> the future impact of them?  How did they get in there?  (This branch is
>> a bzr-svn conversion of part of an old SVN repository.)
>
> Please file a bug on this one; on 'bzr' though it may be a bzr-svn
> issue.

Done. #412210

>
>> 4.  The progress bar for upgrade goes 70% of the way to finished
>> immediately and then takes all of its time working on "Copying content
>> into repository.:Transferring revisions".  Is there any way to make the
>> progress on the progress bar corrspond more nearly to the time that
>> various parts of the process actually take?
>
> Progress is hard :(. Feel free to file a bug on this, but I don't think
> that we're likely to get time to really improve it before 2.0 - we have
> lots of functional issues to fix.

Agreed.

>> 5. Standalone branches are much easier to upgrade than those living in
>> shared repositories.  ...I hope that shared repository usage and policy will
>> be back on the table, because a single repository that large is taking a
>> *long* time to check before even starting an upgrade.
>
> I use similar shared repositories. My recent check landing should have
> made check faster for all repositories, but there is more that can be
> done. Once the repository is checked, you have checked all the branches
> as far as repository content goes - so there is no need to check the
> repository. bzr check --no-repository will help there. (I think thats
> how the option is spelt).

Did this commit land in 1.18rc1?  Is it worth including in next weeks 
1.18whatever?

-Neil