Next Steps for hpss, etc

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


...

>> (2) seems like something that could use a really good HPSS verb for
>> "give me everything I need to start hacking on the tip revision of this
>> branch".
> 
> Agreed.  Happily I'm currently working on improving the “fetch spec”
> code (as part of “fetching a branch should by default fetch all the
> branch's tags too” bug) which will help with this I think.
> 
>> I've considered making it a branch that will cache anything it has to
>> access (so any time it falls back to the stacked-on repository, it will
>> save that data locally.) The primary reason I would skip that, is
>> because inserting into a repository is transactional. We require both a
>> write lock, and a write group that is then committed. And defining the
>> transaction boundaries during an otherwise read-only operation is
>> difficult at best.
>> Also, you get into problems of data model violation. (log needs to look
>> at only the revision texts, but the presence of a revision text implies
>> the presence of the associated inventory and texts.)
> 
> We can always tackle this later.  I agree it sounds a bit hairy... but
> when we get there perhaps we'll see some ways to deal with that.  We'll
> see, eventually :)

Yeah, I did try, a couple of different ways. But the transactional
nature meant we needed something different. (One option is to take a
write lock and a write group for the lifetime of the read lock, and
auto-commit during unlock)

> 
>> I think the two of these would be very good for UDD stuff. Since it
>> allows people to download ~tarball sized content, and still gives them a
>> branch they can hack on and upload.
> 
> I believe Launchpad's build-from-recipe infrastructure would like to use
> this as well.
> 
>> I also think most of the bits are there, so we can migrate
>> incrementally, rather than having to do a large amount of work to get
>> anything that can land.
>>
>> I also think these two things could be worked on in parallel. You may
>> end up with a shallow branch that you can't commit to yet, but it would
>> still be a productive thing to get done.
> 
> Yes, I think the verb can be done at least somewhat in parallel... and
> on that note: precisely which records should a hypothetical
> “get_shallow_stream for revid X” HPSS operation fetch?
> 
> Off the top of my head you want:
> 
>  * revision X
>  * inventory X
>  * at least one parent inventory (as usual for stacking), unless of
>    course the parent is null:
>  * all texts referenced in inventory X (available as fulltexts, of
>    course)
>  * signature X (if present)
> 
> -Andrew.

The stacking invariants say that if we have all of the texts, then we
don't *have* to have the parent inventory. And arguably, if we branched
that revision, then it is unlikely we'll need to push it back up (it
will exist in the location we branched from.)

However, filling in a parent inventory should be a trivial amount of
transfer, and allow us to push up a lot less data if we ever did need to
create a repo with the current tip.

If we are going to do this, it should get *all* parent inventories, though.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkzqiXcACgkQJdeBCYSNAAMIwQCfQN+wF6TGJGfr+SjkvWjdXaCk
H5kAn0Z/O8O603x+z7+hUQ13bQ1SI4TL
=MAm+
-----END PGP SIGNATURE-----