'bzr reconcile' *really* slow
Andrew Bennetts
andrew at canonical.com
Mon Oct 15 01:23:11 BST 2007
John Arbash Meinel wrote:
[...]
> I'm guessing this will help, but I'm worried about:
>
> 1) Memory overflow on large trees. I have an LRUCache implementation
> that (for whatever reason) never made it into core. I'll probably try to
> separate it out again, and submit it.
I'm a bit worried about this too. reconcile is hopefully not a common
operation, so I think it's acceptable to consume large amounts of memory if
that's the expedient thing to do, so long as it doesn't consume impossibly large
amounts. What exactly that means is hard to know.
> 2) It may not be quite enough. I'm currently at just shy of 48 hours of
> runtime. And I'm at 533/2068. Which puts me at 186 total runtime. Or
> just shy of 8 days of operation. A 5-fold improvement makes this 1.5
> days. Which is quite a bit better, but I certainly would think something
> like this should be in the hours range, not in the days range.
I've now got a version of that patch that caches get_text_version, get_parents
and get_heads as well. It does the first 2000 mainline revisions of bzr.dev
(7410 revisions) in 15 minutes on my laptop. It consumed about 700MB of memory
in the process. I'm guessing this makes current bzr.dev (2906 mainline
revisions, ~14000 repository revisions) practical to reconcile, although perhaps
not on this laptop with only 1GB of RAM...
[...]
> 3) Is there any way that this stuff could be stopped and resumed, or is
> most of the time spent just reassuring itself that it really is correct.
> (I have a feeling it is the latter.)
Good question. It always rechecks everything, and our knit files don't have a
“verified to be clean” flag. It might be nice to support this somehow (perhaps
by making it possible to specify an individual file-id (or pack name?) to
reconcile?).
-Andrew.
More information about the bazaar
mailing list