Difficulty of reconstructing a branch whilst deleting/fixing a broken revision

[Jules Bean]

Hi,

I have just spent rather a long time reconstructing a damaged branch,
with a lot of trial and error and a lot of help from the IRC channel
#bzr.

I think there may be some interesting things in this mess for the bzr
developers to think about, so here is an abridged report. The bzr
version I was using is 2.0.3 and the repo format of the broken repo
was pack-0.92 but I upgraded to 2a before fixing anything.

At the end of the report I've put together a few things which seem, to
me, to be bugs or feature requests. I'm happy to open them in the BTS
if you like.

The problem was an accidentally committed very large text file
(600M). The file was removed a couple of revisions later but it lived
on in the repository for that branch, which continued for another 40
or 50 commits before we became aware of the problem.

This is obviously a problem in a trivial way, in that a repository
which should have been about 100M in size was 300M in size. However it
turned out to be a problem in a more substantial way, in that the
large file made certain bzr operations crash, and made other
operations extremely slow and memory hungry.

It wasn't acceptable to lose the work in those 40 or 50 commits, and
unfortunately some of those commits were merges from another branch,
and they needed to be preserved too so that merging back would work.

So, to start with it's easy enough to pull the first broken revision,
uncommit, remove the offending file, and recommit. The hard part is to
graft the old history back on top preserving all subsequent work and
merges and conflict resolution.

I tried quite a few different approaches to fix the problem and they
failed for various reasons. The one which eventually worked was:

For all commits which are just simple commits rather than merges, just
replay them one at a time with

(working directory is reconstructed branch)
bzr replay -rR.A.B ../broken-repo

For reasons not clear to me bzr replay (provided by the rebase plugin)
can only replay single, simple commits. So to replay multiple ones I
just looped in my shell.

For merges with other sub-branches of the *broken* revision, I
couldn't merge (that would bring the large file back in), so I just
use 'bzr shelve' in a scratch branch to shelve the changeset, copied
the shelf over to the reconstructed branch, and unshelved it. (Thanks
to vila in #bzr for that tip). Of course this loses any subhistory in
that branch but, as it happens there was only one and it was shallow
so I decided to ignore that problem. Obviously you could reconstruct
the sub-branch in full in a scratch dir and then merge that.

For merges with non-broken branches I had to preserve the merge. Worse
still these merges conflicted, one of them very badly, and I had to
preserve the conflict resolution. In the end I did the merge (with the
exact revid of the parent in the non-broken branch), resolved any file
name conflicts by hand, using bzr inventory --show-ids when in doubt
to see how the conflicts had been resolved in the other branch, and
then synced up the working tree of the reconstructed branch with a
working tree of a broken branch (at the corresponding revision) using
rsync like this:

rsync  -i -r  --delete --checksum --stats --exclude=".bzr" 
--exclude="*~" ../broken-repo .

...and committed that.

I incidentally note that I tried to do one of these complex changesets
with 'bzr shelve' and it didn't work :

bzr: ERROR: Tree transform is malformed [('versioning no contents', 
'new-150')]

Actually I think this was misguided anyway since bzr shelve would have
lost the file-ids of added files, and that's one of the things that
you have to preserve about the merges - the revision parents and the
file-ids.

Although it took me a while to understand the problem well enough to
attempt this solution, it was fairly mechanical once I got it
working.

Some aproaches which didn't work:

bzr-fast-export everything and fast-import-filter to cut out the
offending file. This didn't work most significantly because
bzr-fast-export crashed on the revision with the big file. Even if it
*had* worked I believe it would have rewritten all the file-ids so I
would have been forces to bzr fast-export *all* my branches and import
them into the same repo using the same mark file, to get the file-ids
in sync. (Even then, would that have handled merges which renamed
files correctly?)

bzr-fast-export just that last portion of history and replay it. This
sounded promising but I couldn't work out what commands to use and in
any case, would it have preserved file-ids through merges?

Fixing merges by doing "bzr merge -rrevid:xyz ../broken-repo; bzr revert ."
followed by applying a diff or shelf by hand. Again, file-ids kill
you. A hand-applied diff will never do the right thing with newly
added files brought in by the merge.


Probable bug:

bzr performs very badly on large text files. It crashes some operations
and makes other operations appear to crash (displaying a malloc error)
before eventually succeeding. I quite see that huge text files are not
considered a key use case, but if they don't work well, they should be
forbidden by the tools. I suggest a "max-file-size" limit, overridable
at the commandline. If that had been in place for us, we would have
noticed the mistake and never committed the file in the first place.

Probable bug:

On some sufficiently complex changesets, bzr shelve doesn't work
even though bzr commit does.

Probable documentation bug:

bzr shelve docs imply it is text changes only, but actually that
change (AIUI) when shelve went core.

Feature request:

Explicitly support an externalisable 'rich diff' which includes file
renames, additions, deletions and binary changes. This is probably
just the 'shelve' format, and not very different from the external
patchset format.

Feature request:

Have a "bzr log" option or format (or something) to extract 'just' the
commit message, so that a shell script could grab a commit message and
a diff and re-apply.


I'm sorry this email is so long and rambling, I hope some of it is
useful feedback,

Jules