2.0 upgrade experiences

[Neil Martinsen-Burrell]

On 2009-08-12 05:49 , Martin Pool wrote:
> 2009/8/12 Robert Collins<robert.collins at canonical.com>:
>
>>> 2. The space savings is not as big as my wildest dreams (This is a
>>> standalone branch, even though the parent directory is called "repos".)
>>>
>>> nmb at rameses[~/repos/career]$ du -sk .
>>> 6848  .
>>> nmb at rameses[~/repos/career]$ du -sk ../../repos.bak/career
>>> 7028  ../../repos.bak/career
>>
>> You will have content in .bzr/repository/obsolete-packs that will get
>> cleaned up when you make subsequent commits. We do this to deal with
>> various edge cases without requiring fsync/fdatasync operations (which
>> are either a) expensive, or b) [on XFS and MacOSX] don't work).
>
> I'd like to improve this by
>
> 1- Making the pack UI say something like this:
>
>    Packing repository ..../career
>    5454MB in 12 packs....
>    Thinking hard.....
>    Now 1230MB in 1 pack, and 5454MB in 12 obsolete packs that will be
> deleted later
>
> 2- Adding an option either to pack or to another command to remove
> obsolete packs, and mentioning this from the help for pack.  (There is
> a bug for this.)
>
> The text I propose for #1 is perhaps not ideal, but at least it would
> give a clue that there's a reason why this is happening and it's not
> just something broken, and adding it may be less controversial than
> deleting these files.
>
> I think if fsync was expensive but reliable and was only run from an
> explicitly requested repack that would be reasonable.  However, if
> it's not reliable and there's a chance that a crash near the end of
> packing could lose all your data, we can at least make it more obvious
> why this is happening.  I would note though that there's no guarantee
> that the fs will be able to safely read the renamed files.
>
> The pattern in bug reports seems to be:
>
>   * The space bloat is a real cause for concern.
>   * People do suffer fs corruption - either lost files, or files with
> missing blocks.   There is a limit to how much we can work around this
> but not deleting files when they're immediately obsolete will probably
> help.
>   * I think the code that recovers from an unexpectedly-removed pack
> file is working well now, so we don't need the obsolete packs for the
> sake of supporting concurrent clients.

I think that it is important that upgrading to our new, highly-touted 
default format should have clear savings in disk space.  If fsync is 
expensive but reliable, then another option would be to add the 
expensive clear-obsolete-packs process to the end of upgrade, which 
people already expect to be an expensive process.  If reliability cannot 
be guaranteed, I think that the current situation is optimal, but needs 
better communication about why it's necessary for safety.

[...]

>>> 4.  The progress bar for upgrade goes 70% of the way to finished
>>> immediately and then takes all of its time working on "Copying content
>>> into repository.:Transferring revisions".  Is there any way to make the
>>> progress on the progress bar corrspond more nearly to the time that
>>> various parts of the process actually take?
>>
>> Progress is hard :(. Feel free to file a bug on this, but I don't think
>> that we're likely to get time to really improve it before 2.0 - we have
>> lots of functional issues to fix.
>
> I don't object to filing a bug but this may be the kind of thing on
> which it's hard to get traction from a bunch of reports for every
> occurrence.
>
> Is the progress bar still spinning there, or counting up, or is it
> just totally still?  In other words, is the objection just that
> there's progress but it's not proportional to elapsed time, or that
> there's no progress?

It is spinning (although slowly on my laptop which seems to have to work 
quite hard at this process).  The only complaint is that the left-right 
progress is not proportional to elapsed time.

-Neil