Merge eats memory

[Michael Ellerman]

On Wed, 1 Jun 2005 17:29, Michael Ellerman wrote:
> On Wed, 1 Jun 2005 01:28, John A Meinel wrote:
> > John A Meinel wrote:
> > Do you have this repository somewhere we can get at it. I would be
> > interested in profiling it a little bit if possible.
>
> It's pretty easy to reproduce though. You just pull Linus' git repository,
> then bzr init; bzr add .; bzr ci -m "Import everything".

Actually you don't need git at all.

Just using a tar ball of the kernel (or probably any large tree) is enough to 
see the behaviour.

This is from merging a 30K patch between two kernel trees (no .git), it took 
about 8m44s.

The last entry in the refs log (attached) comes out as:

Refcount	Class
-------------------------------------------------------
55319		bzrlib.merge.SourceFile
55317		bzrlib.inventory.InventoryEntry
134		bzrlib.commands.Command
48		exceptions.Exception
34		random.Random
32		bzrlib.changeset.ChangesetEntry
28		bzrlib.changeset.ChangeUnixPermissions
27		threading._Verbose
26		exceptions.StandardError
20		bzrlib.tree.Tree

The destination tree is 339M, 17,361 source files. The source tree is 340M 
17,363 source files.

So perhaps the merge code is keeping a SourceFile instance for each source 
file in both trees, and one for the merged result? Same for InventoryEntries?

cheers

-- 
Michael Ellerman
IBM OzLabs

email: michael:ellerman.id.au
inmsg: mpe:jabber.org
wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person
-------------- next part --------------
<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.branch.Branch'>:exceptions.Warning:<class 'threading.Thread'>:<class 'bzrlib.tree.Tree'>:<class 'sets.BaseSet'>:
133:47:33:26:25:18:17:16:16:15:

bzrlib.inventory.InventoryEntry:<class 'bzrlib.merge.SourceFile'>:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55316:36881:134:48:34:27:26:20:18:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:exceptions.Warning:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:<class 'bzrlib.changeset.ChangesetEntry'>:
55319:55317:134:48:34:27:26:20:19:18:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:
55319:55317:134:48:34:27:26:20:20:19:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:
55319:55317:134:48:34:27:26:20:20:19:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:
55319:55317:134:48:34:27:26:20:20:19:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.store.ImmutableStore'>:
55319:55317:134:48:34:27:26:22:20:19:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'bzrlib.tree.Tree'>:<class 'bzrlib.changeset.ChangeUnixPermissions'>:
55319:55317:134:48:34:27:26:23:20:19:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:27:26:23:20:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:27:26:23:20:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:27:26:24:21:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:27:26:24:21:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:<class 'bzrlib.changeset.ChangesetEntry'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:27:26:26:22:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:27:27:26:23:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'threading._Verbose'>:<class 'bzrlib.changeset.ChangesetEntry'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:27:26:26:23:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:27:27:26:24:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:27:27:26:24:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:27:27:26:24:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:28:27:26:24:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:28:27:26:24:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:28:27:26:24:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:27:27:26:24:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:29:27:26:25:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:28:27:26:25:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:28:27:26:25:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:28:27:26:25:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:29:27:26:25:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:31:27:27:26:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:32:28:27:26:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:32:28:27:26:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:31:28:27:26:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:31:28:27:26:20:

<class 'bzrlib.merge.SourceFile'>:bzrlib.inventory.InventoryEntry:<class 'bzrlib.commands.Command'>:exceptions.Exception:<class 'random.Random'>:<class 'bzrlib.changeset.ChangesetEntry'>:<class 'bzrlib.changeset.ChangeUnixPermissions'>:<class 'threading._Verbose'>:exceptions.StandardError:<class 'bzrlib.tree.Tree'>:
55319:55317:134:48:34:32:28:27:26:20:

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20050603/6bc19b77/attachment.pgp