2.0 upgrade experiences

Martin Pool mbp at canonical.com
Wed Aug 12 11:49:02 BST 2009


2009/8/12 Robert Collins <robert.collins at canonical.com>:

>> 2. The space savings is not as big as my wildest dreams (This is a
>> standalone branch, even though the parent directory is called "repos".)
>>
>> nmb at rameses[~/repos/career]$ du -sk .
>> 6848  .
>> nmb at rameses[~/repos/career]$ du -sk ../../repos.bak/career
>> 7028  ../../repos.bak/career
>
> You will have content in .bzr/repository/obsolete-packs that will get
> cleaned up when you make subsequent commits. We do this to deal with
> various edge cases without requiring fsync/fdatasync operations (which
> are either a) expensive, or b) [on XFS and MacOSX] don't work).

I'd like to improve this by

1- Making the pack UI say something like this:

  Packing repository ..../career
  5454MB in 12 packs....
  Thinking hard.....
  Now 1230MB in 1 pack, and 5454MB in 12 obsolete packs that will be
deleted later

2- Adding an option either to pack or to another command to remove
obsolete packs, and mentioning this from the help for pack.  (There is
a bug for this.)

The text I propose for #1 is perhaps not ideal, but at least it would
give a clue that there's a reason why this is happening and it's not
just something broken, and adding it may be less controversial than
deleting these files.

I think if fsync was expensive but reliable and was only run from an
explicitly requested repack that would be reasonable.  However, if
it's not reliable and there's a chance that a crash near the end of
packing could lose all your data, we can at least make it more obvious
why this is happening.  I would note though that there's no guarantee
that the fs will be able to safely read the renamed files.

The pattern in bug reports seems to be:

 * The space bloat is a real cause for concern.
 * People do suffer fs corruption - either lost files, or files with
missing blocks.   There is a limit to how much we can work around this
but not deleting files when they're immediately obsolete will probably
help.
 * I think the code that recovers from an unexpectedly-removed pack
file is working well now, so we don't need the obsolete packs for the
sake of supporting concurrent clients.

>> When I thought that a pack operation might clear up additional space, I
>> got the following error:
>
> bzr auto packs on conversion, there is no need to run pack. The general
> rule for running 'bzr pack' is 'do not run this command'.
>
>> nmb at rameses[~/repos/career]$ bzr pack
>> bzr: ERROR: Pack '6ffbb186d5539f27e54609f64e5fb92e' already exists in
>> <bzrlib.repofmt.groupcompress_repo.GCRepositoryPackCollection object at
>> 0x126c3d0>
>>
>> Is that error serious?
>
> Its a known bug with 2a, and a fix is in-progress at the moment in a
> branch of Andrews.
>
>>   Does it mean that I cannot pack this branch
>> further?
>
> It means that unlike 1.9 bzr is willing to attempt a pack when there is
> only one pack present, and that it doesn't cleanly report the case when
> the repository is already fully packed.
>
>>   Are there obsolete packs still sitting around somewhere?
>
> Possibly, you can look in .bzr/repository/obsolete_packs. The contents
> of this directory can be removed (but be sure that your filesystem has
> recorded the bzr dir to disk before doing the removal). Leave the
> directory in place though.

I think there's also an easy open bug that if the user deletes the
whole directory we fail; they have to recreate it by hand.

>> 4.  The progress bar for upgrade goes 70% of the way to finished
>> immediately and then takes all of its time working on "Copying content
>> into repository.:Transferring revisions".  Is there any way to make the
>> progress on the progress bar corrspond more nearly to the time that
>> various parts of the process actually take?
>
> Progress is hard :(. Feel free to file a bug on this, but I don't think
> that we're likely to get time to really improve it before 2.0 - we have
> lots of functional issues to fix.

I don't object to filing a bug but this may be the kind of thing on
which it's hard to get traction from a bunch of reports for every
occurrence.

Is the progress bar still spinning there, or counting up, or is it
just totally still?  In other words, is the objection just that
there's progress but it's not proportional to elapsed time, or that
there's no progress?

> Thanks! I really appreciate this feedback.

Yes, me too, thanks for taking the time.

-- 
Martin <http://launchpad.net/~mbp/>



More information about the bazaar mailing list