store migration

[Ian Booth]

Hi

We are looking to migrate juju-core across to github Real Soon Now™. The
timeframe is hopefully by the end of next week, but depends on how soon we can
get documentation finished, everything tested, and agreement to move ahead. It
will be the entire juju-core codebase, and from there we plan to split out
reusable packages like cmd etc as required.

It makes sense to me to split out store after the github migration.

With regard to the thirdparty package in core, I don't see that we need it any
more. The only thing in it is pbkdf2, which is available from go.crypto which we
import from anyway for ssh authorised keys functionality. So I say we nuke it.

On 20/05/14 23:58, David Cheney wrote:
> On Tue, May 20, 2014 at 11:55 PM, Francesco Banconi
> <francesco.banconi at canonical.com> wrote:
>> Hi all,
>>
>> FYI this morning the UI team discussed about possibile paths to migrate the store code from lp:juju-core to Github.
>> We are sharing this plan so that we can synchronize/collaborate with other teams with similar needs in their todo list.
>>
>> Here are the store package dependencies as obtained with go list:
>>
>> Deps:
>> $ go list -f "{{range .Deps}}{{println .}}{{end}}" launchpad.net/juju-core/store | grep juju-core
>> launchpad.net/juju-core/charm
>> launchpad.net/juju-core/charm/hooks
>> launchpad.net/juju-core/juju/osenv
>> launchpad.net/juju-core/schema
>> launchpad.net/juju-core/thirdparty/pbkdf2
>> launchpad.net/juju-core/utils
>> launchpad.net/juju-core/utils/set
>> launchpad.net/juju-core/utils/zip
>>
>> Tests deps:
>> $ for i in `go list -f "{{range .Deps}}{{println .}}{{end}}" launchpad.net/juju-core/store`; do go list -f "{{range .XTestImports}}{{println .}}{{end}}" $i; done | grep juju-core | sort | uniq
>> launchpad.net/juju-core/charm
>> launchpad.net/juju-core/charm/testing
>> launchpad.net/juju-core/environs/config
>> launchpad.net/juju-core/juju/osenv
>> launchpad.net/juju-core/schema
>> launchpad.net/juju-core/testing
>> launchpad.net/juju-core/testing/filetesting
>> launchpad.net/juju-core/testing/testbase
>> launchpad.net/juju-core/utils
>> launchpad.net/juju-core/utils/set
>> launchpad.net/juju-core/utils/zip
>>
>> As you can see, there are some incremental steps we will need to follow to achieve our goal, I’ll try to describe them below including notes from William. I suppose we can encounter complications in this path, but hopefully at the end we’ll have a good starting point for the store.
>> William agreed on the goals of this migration (having a common/separate module which can be reused by both juju-core and the GUI).
>>
>> Just two notes before sketching a possible plan:
>> - the store must be able to be configured to use its own db or the juju-core one based on the context it is used;
>> - we need a way to migrate partial Bazaar commit history to git (perhaps someone has already experience with this?).
>>
>> A possible plan is as follow:
>>
>> 1) Migrate juju-core/thirdparty to juju/thirdparty (Github).
>> Since there are no dependencies here, this seems to be a good first candidate.
> 
> nac, these are out horrors. The code we forked should either be pushed
> back upstream, and/or captured by godeps. I think the whole thirdparty
> thing happened before we had godeps.
> 
>>
>> 2) Migrate juju-core/utils to juju/utils (Github).
>> package:
>>  launchpad.net/juju-core/utils
>> deps:
>>  launchpad.net/juju-core/juju/osenv
>>  launchpad.net/juju-core/thirdparty/pbkdf2
>> tests deps:
>>  launchpad.net/juju-core/juju/osenv
>>  launchpad.net/juju-core/testing/testbase
> 
> Sounds like the best candidate to start with.
> 
>> I did not look at juju/osenv: we might want to migrate that as well, or just refactor the code to remove the dependency. Note that each time we move a package we need to also move the relevant code in juju-core/testing to juju/testing. The latter already exists on Github. The juju-core/testing module has lots of dependencies on other packages in juju-core, so if we encounter those we will need to handle that (I presume by refactoring test code). In the utils case, juju-core/testing/testbase does not seem to depend on anything, so we should be ok.
>>
>> 3) Migrate juju-core/schema to juju/utils/schema. The schema package has no juju-core dependencies.
> 
> Yes to migrating it, no to renaming it. Schema is useful as a top
> level project, rather than buried in an amorphous utils repo.
> 
>>
>> 4) Migrate juju-core/charm. This has some pre-requisites. Basically IIUC charm defines config and meta and those definitions are tangled with the underlaying Mongo documents. William suggested to decouple that, implementing separate data structures to be used to (de)serialize data. This way, changes to charm database structure can be detected earlier and core developers are able to react accordingly. Soon this could also involve actions data.
> 
> I don't see the value in splitting off this repo unless you need it
> for the store.
> 
>>
>> 5) Move the store.
>>
>> For each step AFAICT what we need to do is similar to the following:
>> 1) possible preliminary work to move the testing stuff;
>> 2) create Github project (if it does not exist);
>> 3) add readme, copying, license files etc;
>> 4) notify developers we are locking the package;
>> 5) migrate the code;
>> 6) fix package imports if required (e.g. for sub-packages);
>> 7) fix package tests;
>> 8) land a juju-core branch which:
>>    - removes the package;
>>    - fixes all the imports;
>>    - includes the new dependency info in the dependencies.tsv file;
>> 9) notify developers about the new external dependency.
>>
>> Thanks!
>>
>> --
>> Francesco
>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev at lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
>