About plainbox-provider-certification-{client,server}

[Zygmunt Krynicki]

On Mon, Aug 17, 2015 at 5:10 PM, Jeffrey Lane
<jeffrey.lane at canonical.com> wrote:
>
>
> On Mon, Aug 17, 2015 at 7:20 AM Zygmunt Krynicki
> <zygmunt.krynicki at canonical.com> wrote:
>>
>> Hey
>>
>> tl;dr; plainbox-provider-certification-{client,server} now live in
>> https://code.launchpad.net/plainbox-provider-canonical-certification
>> and releases are using git-dpm. Read on for details.
>>
>> I wanted to explain my actions affecting both certification providers
>> that occurred last week. As you know, the last maintenance release did
>> happen, but it was made outside of lp:checkbox. This happened due to a
>> combination of several factors. You can read about those below. Now
>> that it is done, with [1] being merged, let me tell how how the two
>> providers are going to live on.
>>
>> First of all, this is the first time we're going to use git on a
>> plainbox provider natively. There might be some learning curve though
>> it should be related only to git-on-launchpad and UI issues rather to
>> git itself.
>
>
> That's a bit presumptive.  I know of at least one or two people who had and
> continue to have a very steep learning curve because of git itself.  Not
> everyone is part of the cult of git, nor cares to be and only use git when
> absolutely necessary. That's something that should be kept in mind,
> especially for those of us who have been using other things exclusively
> (BZR) for years and have only seldomly ever touched git.  Couple that with
> the weirdness of using LP with git and that learning curve grows steeper
> yet.
>
> Not saying this should hamper any progress, however, these sorts of major
> paradigm shifts should consider that as well.  I really miss UDS, because
> this sort of major change is what would have been discussed and planned and
> shared well ahead of things just happening.
>
> Is there some sort of roadmap for these changes that we could track?  Not
> sure if your team's trello board is appropriate or not, but if it is, can I
> take a peek?
>
>> The workflow is conceptually the same as before, where we propose a
>> merge request to a given branch an after it lands, it's done. The only
>> visible difference here now is that there are two branches
>> (master-client and master-server), each holding exactly one of the two
>> providers. This is strictly related to simplicity, tagging and
>> management of releases. I wanted to move to a model where each
>> provider can essentially live in a small repository and everything
>> related to the life cycle of that provider would be no different than
>> any other provider that follows the same pattern.
>
>
> Makes sense, I was never a fan of having all the providers in one place, and
> then all the packaging stuff in a separate place that may or may not also
> house other bits of the provider that are non-packaging-related like
> canonical-certification.conf.
>
>>
>> The reason there are two branches is very simple, I didn't want to
>> split the Launchpad project in two, given how complex and time
>> consuming that is. With git it's equally easy to track and work on one
>> or one hundred branches, as long as they share one repository. So in
>> practice, everything relating to the *client* provider is in the
>> master-client branch while the *server* provider is in the
>> master-server branch. If you find that confusing we can trivially
>> rename that to just client and server, there's no problem in doing
>> that.
>
>
> Easy is a relative term.  I've spent 15 minutes now after cloning this
> trying to figure out how to get access to the origin-master branch.  It
> doesn't seem to exist, or at least I'm not smart enough to figure out how to
> access it.
>
> bladernr at sulaco:~/development/git/plainbox-provider-canonical-certification$
> git branch -a
> * master-client
>   remotes/origin/HEAD -> origin/master-client
>   remotes/origin/master-client
>   remotes/origin/master-server
>
> Again, please do not presume that anyone working on this beyond yourself
> knows anything at all about git.  I have yet to sort out how to actually get
> this master-server branch.  I would really, really appreciate, for my sanity
> and for the sanity of others who have limited or no git experience or
> knowledge a useful getting-started doc specifically to cover all these
> things that I'm being told are simple, easy and fast.  Because I still find
> that anything I do with git, in general, takes me 2 - 3 times as long as
> with bazaar and have yet to see any real benefit from all this change, but
> that's just my personal opinion, which I've voiced before.
>
>> Next thing is merge requests. Apart from the slight UI change (you
>> have to target a branch _in_ a repository rather than branch alone)
>> everything works as before. I'm sure there will be a few WTF moments
>> as we all learn the new Launchpad user interface but we'll get over
>> that soon and the knowledge is useful for all the other projects we
>> have, or are moving to git. After the merge request is approved a CI
>> automaton will merge it. This is still a bit up in the air. I'll work
>> with a few people to enhance our CI system to work with git and bzr
>> together (i.e. switch to the new "tarmac" prototype). This will happen
>> over the next few weeks. We need this for a number of other projects
>> that started off with git. This will be no different and I will send
>> updates as we progress on this effort.
>
>
> Another layer of complexity for people that are new to all this... at least
> in this one, everyone is at the same disadvantage.  I've made one MR for a
> git branch so far, and it took me 30 minutes, plus asking and finally
> getting an answer on IRC to figure out how to to do so.  I could have done
> it the traditional way in less than 5 minutes...
>
>> The only thing that *is* significantly different is how any kind of
>> "transition" is made. In the past we could have easily made a job
>> rename in one commit, that touches all providers at the same time
>> (well, except for those that are already out-of-tree, like CDTS). Now
>> that will need to be coordinated.
>
>
> I would suggest the following:
>
> Any "common core" job definitiions exist in plainbox-provider-checkbox.
> Any "provider specific" job definitions exist in
> plainbox-provider-certification in the relevant branch.  Thus the client
> specific jobs like wifi, bluetooth, graphics, etc, would live in the client
> branch, and would be maintained and owned by the client team.  Server
> specific jobs that client doesn't use, such as multi_nic,
> multi-path(coming), large-volume iperf runs, etc live in the server provider
> and are maintained by us.
>
> Thus the only things that need to be "coordinated" would be major name
> changes to the "Common Core" jobs that apply to both client and server
> situations, and the rest of us would be free to change our own provider
> specific jobs at our leisure.
>
> I would presume that changes to the common jobs are rare enough that this
> would be workable as most/all the job development is on NEW jobs and a name
> change is very seldom used.
>
> The only catch there is that we'd need to be better about sharing new job
> definitions in case something we create is useful to you and vice-versa.  A
> good example of this is TRIM testing on SSDs.  We have a story coming up in
> the future to write a test that tests the TRIM function of SSDs because of a
> kernel bug discovered where TRIM causes SSDs to end up remounted read-only.
>
>>
>> This also leads us to the greater
>> importance of "API" stability. For a provider, the API is the set of
>> jobs it contains. I don't think this is a major issue, we already have
>> that problem with our downstream project/commercial derivatives. We
>> will just have to tread ourselves the same way as we treat those
>> projects. With planning, consideration, tooling and communication.
>>
>> The last thing, and also the biggest change, is how releases work. The
>> key reason I moved away from lp:checkbox was the broken release
>> support for this pair providers. Rather than add more complex tooling
>> I wanted to removed the excessive complexity that made the tools
>> needed in the first place. Releases are simple: they are based on
>> signed tags and tarballs (generated with ./manage.py sdist). Tarballs
>> are uploaded to Launchpad (to the
>> plainbox-provider-canonical-certification project). This is exactly
>> how we already treat all of the other providers. We now have less
>> divergence as as result of this. This tarball is then combined with a
>> packaging branch, like we do in all the Debian and Ubuntu releases
>> (though not as we do in our PPAs) and the resulting source package is
>> uploaded to a PPA for building.
>>
>> The major thing here, is that we have now adopted git-dpm. Git DPM is
>> the preferred form of modification of all the packages in Debian and
>> in Ubuntu. All our Debian packaging will be automatically converted to
>> git-dpm in the next few days (or week). This is already done in
>> testing but the final trigger will be pulled at around after Debconf.
>> This is a Debian-wide change and we are just following the transition.
>> Git DPM is *fantastic*. If you ever plan to, or need to, work on a
>> release you need to learn this tool. The manual page is a great start.
>> There are also several online tutorials but I think there are
>> overcomplicated for what this tool is doing. Your mileage may vary.
>> Git DPM lets us work, in one tree, with our upstream branches, our PPA
>> releases and our Debian releases. I'm sure that we're just getting
>> started and as we explore and try we will simplify our release process
>> greatly.
>>
>> There is a downside though, as Launchpad git auto-builder is not yet
>> functional we have lost the ability to do daily builds straight from
>> the repository. This is not a major change, because this is a project
>> that sees less updates than others, but it is a loss non the less. As
>> Launchpad evolves this functionality will be restored. For the moment,
>> and for our other projects that are also on Git I will be working on
>> simple tooling (thanks to git DPM) that lets us upload daily source
>> packages from a cron job somewhere.
>
>
> How will this, if at all, affect hot-fixes?  E.g. a bug is discovered in a
> test script, fixed and merged?  Will this delay getting that bug fix into a
> release or into the daily builds?
>
>> The one thing that you must remember from this, is that our packaging
>> is no longer based on the tree export from the bzr repository. This is
>> a *big* change as now packaging for all kinds of releases (Ubuntu,
>> Debian, PPA, daily PPA, etc) for this and all projects that follow the
>> same pattern is _identical_. This is a major improvement over our
>> prior model, where packaging was totally different an never kept
>> up-to-date. The certification provider does not have Debian or Ubuntu
>> releases (and likely won't ever have) but the same release tooling
>> (git-dpm) applies equally to all of those. I strongly encourage you to
>> look at git-dpm, at the packaging branch [2]. I recommend that you
>> clone it, and follow the git-dpm manual page to build it. There are
>> helpful instructions in the debian/README.Debian file.
>>
>> If you have any questions about this please don't hesitate to contact me.
>
>
> I understand change is necessary, please understand my concerns are because:
>
> 1:  This change to git seemed (to me) to be completely arbitrary and was
> never really discussed with anyone outside the client team who also works on
> this code
> 2: More importantly, I have very, VERY limited dev resources on my team, and
> a HUGE, almost insurmountable amount of development work left to do for
> 16.04, in addition to having a large amount of dev work to do on OCP
> Checkbox as well.
> 2a: This forced change to git, for me, at least, means I will be spending
> vast more amounts of time trying to use ALL of this new complexity (whether
> it is less complex or no, it is a fundamental shift in tools and methods
> which have a very steep learning curve) and I do not have that kind of time,
> nor resources.  So the net, for me, for now, in the short term, is a huge
> time sink.
> 3:  I'm not blaming anyone, just a bit disappointed in what is a least a
> perceived (on my part) lack of communications about major changes to a
> shared code base.
>
> Finally, I'm all for change.  SO please take this as just my concerns, I
> don't speak for anyone but myself, except for where I see the potential to
> delay dev work for my team on new tests and other improvements.  It's partly
> frustration, and partly, as I said, because my team and I have very limited
> resources and major tooling and packaging shifts eat into those very limited
> resources.
>
> I'm sure this is just an immediate growing pain and will ease as time moves
> on.
>
> I do appreciate all the very hard work and time you've put into this, and
> into explaining in this very descriptive email (as with the previous
> announcement about hwcert-tools as well).

Thanks for your honest and clear message Jeff.

I want to show you, and anyone else that needs to, how to work with
the certification provider in its current location. I will gladly
record a tutorial, with step-by-step instructions that show how one
can propose changes to the server provider. I will also, gladly, spend
any amount of time required with hands-on sessions and make sure that
everyone that needs to or wants to can use the tools effectively. What
I'm trying to say is that I'm not willing to let people behind. This
also extends to packaging, to releases, to all the things that we need
to do on a daily basis.

I'll address each of the specific issues you've raised in a separate
message (it's a bit late today).

Best regards
ZK