RFC: get_record_stream and groupcompress

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
> On Thu, 2008-08-14 at 15:34 -0500, John Arbash Meinel wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Robert Collins wrote:
>>> So I've a thorny little tradeoff to encapsulate in code.
>>>
>>> The best approach I've thought of to date is a parameter to
>>> VF.get_record_stream called 'sort_is_hint'. If you think that that
>>> sounds fine and dandy, say so now and skip the detail below.
>>>
>> It is okay, but I'm not sure it is the best way. Why can't gc and packs
>> just pass 'unordered' if the sort is a hint anyway.
> 
> its not a hint for knits or weaves; its correctness for them. knit based
> packs don't strictly care [though they do remember what is being
> inserted in case it's compression basis is not satisfied]. If its a hint
> it doesn't stop it being _useful_.
> 

I feel like the heuristics for determining when to pay attention to the
hint are somewhat ill-defined, and would require a non-trivial amount of
tweaking to make sure they are correct. I'd rather postpone than wait a
long time to get this api right on top of everything else.

> 
>>> We could insert in any order from remote repositories, this would tend
>>> to convert from packs poorly, but a 'pack' operation would fix things
>>> up. Fetching from a gc repository over SFTP or the smart server would
>>> tend to behave well because they would preserve as much ordering as the
>>> base repository has.
>> That is the route I would take.
> 
> This feels like its going to end up with a LOT of uses going 'my
> repository got bigger; I thought this was an upgrade'. And that will
> confuse people, and also be rather nasty for interoperation with knits,
> knitpacks etc.
> 

With non-optimal ordering, groupcompress was 2:1, with optimal, it was
3:1. Even with optimally bad ordering I would expect a small benefit. At
a minimum we are running zlib over multiple texts (20MB?) rather than a
single file text.

>>> We could buffer to a local store - for instance we could do like git
>>> does some form of file-per-object store [but still using atomic
>>> insertion to make data visible to other readers] and then compress that
>>> store.
>> Sounds like a lot of work to avoid doing "bzr pack".
> 
> The difference is that pack has to consider the whole repo; doing a
> short term buffer operation (heck even a totally uncompressed temporary
> pack) would only consider transmitted data, so be O(change) not
> O(history).
>  

I think this is an interesting idea. I'm not very convinced about
needing visibility before it gets repacked. I'm *not* convinced that it
needs to be there before we allow groupcompress to actually get into
bzr.dev so that it is tracked properly, or even before it is a non-dev
format to be used. It feels a lot like something we can add later.

> 
>> My feeling is to cross the bridge when we get there. Let's get basic
>> groupcompress working.
> 
> It is.

I don't know what updates you've done and published. Last I checked
there wasn't anything for me to pull. It may be sitting on your disk.

'time bzr branch gc-repo1/bzr.dev gc-repo2/bzr.dev'

is not strictly "usable" at the moment. Without the LRU text cache, it
has been 8 min+ and it hasn't gotten past "Transferring 0/4".

With an LRU cache (which at least simulates transferring in a better
order) the last time I tested it took 7 min to transfer from a pack repo
to a gc repo. I'm currently at 9m30s between gc repos and still on 0/4.

It does work, but it takes far too long to be considered something I
would ever have people actually use.

> 
>>  Get fetch working such that it doesn't transmit
>> all full-texts and require a full recompression of everything.
> 
> Half done. In fact this is why I'm working on the ordering issue.

I can't say I've heard much about how you plan to avoid the
full-recompression problem.

> 
>> And then get a "bzr pack" that can figure out how to put everything
>> "optimally".
> 
> Sure.
> 
>> I realized the other day that "topo_sort" *doesn't* guarantee grouping
>> across the whole file_id.
> 
> Yes, thats why the new sort order is called reverse_topo_grouped,
> because it wants to be grouped by file id.
> 
> 
>> It would also be really neat if we could find a way to do appropriate
>> cross-file grouping. I can't really think of much from just a file-id
>> stand-point, though. Given file size, we might try to insert large ones
>> first. Though I would be careful to not insert all the large files into
>> one chunk, and have only tiny ones for the next.
> 
> basename is likely the best hint to use for that; I would use it to
> group files named the same, with all of a given id then all of the next
> etc. But we don't currently have that pushed down to the text layer.

I would actually guess extension is one of the better bits to use, but I
don't know that for sure.

Anyway, the first portion of a file-id (before the first '-') is
actually the basename. So just sorting by file_id is probably going to
get us close. Sure, it doesn't change with a rename, etc, etc, but it is
still a decent hint.

> 
>> So I think the hint is a nice thing, but I feel it is a bit premature.
> 
> I don't :).
> 
> -Rob
> 

We don't have gc in bzr.dev, and I don't think hinting is a strict
requirement for getting it in. Thus I don't think the hinting should be
the current highest priority. Maybe I'm just on my "lets get things
moving" from being a release manager.

Getting a way to not have to spend 7 minutes of CPU to branch bzr.dev
locally would be something more important than passing something other
than "unordered" to get_record_stream(), IMO.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkikrz4ACgkQJdeBCYSNAANMrQCeIVx+vx8w3Zoug4gHiZRFHccZ
NV8Ani3z6FA+UCleiYldNPcaa59VMA+Y
=umMw
-----END PGP SIGNATURE-----