[Bug 1267393] Re: [MIR] juju-core, juju-mongodb, gccgo, golang
Jamie Strandboge
jamie at ubuntu.com
Fri Sep 11 15:58:59 UTC 2015
Yesterday slangasek and I met with the juju team regarding the path
forward for transitioning to shared libraries and Alexis will be
commenting on that later.
Unrelated to that, juju is using many embedded code copies and juju-core will need to be adjusted to build with many of these broken out into golang-*-dev packages (see https://wiki.ubuntu.com/MIRTeam for details, and use dh-make-golang to make this easy). Before I give the list, a couple of things:
* it is acknowledged that the Go community has traditionally embedded sources in their trees and then managed imports themselves. This works fine for the app model where bundling is the norm (eg, for Ubuntu Touch clicks and Ubuntu Core snaps). However, for the reasons outlined in the MIRteam wiki, this does not work well with the Linux distribution model and applying for main inclusion in Ubuntu by definition means working within the distribution model. Also, the Go community has acknowledged the benefits of shared libraries with regard to code reuse, project management, the distribution model, etc, which is why Go 1.5 has shared library capability (that doesn't mean the app bundling model doesn't still have its place)
* it is also acknowledged that juju is different in some ways to other software delivered via the archive in that its current packaging of client, server, agents, etc is for multiple platforms, not just Ubuntu. However, juju is being delivered to Ubuntu juju clients via apt and therefore for those clients there isn't a technical reason why those embedded libraries can't be delivered as separate packages (in other words, it's fine if juju-core has embedded sources in the orig.tar.gz if that helps the juju team, it is just that (many of) those embedded sources should not be used during archive builds (where instead juju-core should use Build-Depends on golang-*-dev packages)). This is all standard Debian and Ubuntu archive practice and that is why the Debian Go team has been working so hard on dh-golang, Built-Using, etc
* it should be acknowledged by go developers seeking main inclusion that the MIR, foundations and security teams have already made many concessions for golang and these changes to the process alone will result in increased maintenance effort for officially supported packages (but at a level we believe can be supported, unlike if we ignored embedded sources/shared libraries)
* creating golang-*-dev packages for embedded sources doesn't mean the juju team can't still control the code, it is just that the lib will live in a different place. Eg, suppose that github.com/joyent/* is pulled out into golang-joyent-dev. A MIR would be required for golang-joyent-dev and the juju team would be the team committed to maintaining the package and fixing those bugs. The PPU acls would be adjusted to allow the juju team to upload this package, just like with juju-core. The SRU process would apply to it, etc. (Ie, nothing fundamentally changes wrt control if the juju team wants to maintain that control (ie, perhaps they wouldn't mind relinquishing control of some of them))
* all the embedded packages do not necessarily need to be pulled out. Eg, github.com/juju and gopkg.in/juju seem to be internal to juju and if nothing in the archive would ever need to use them, there is no reason to break them out
* juju is currently suffering from the problems that can happen with the bundling model in that it is using different embedded sources for the same functionality. Eg, code.google.com/p/go.crypto and golang.org/x/crypto. Picking one and breaking it out into a single -dev package will ease the maintenance burden
With that said, here is a list to start the conversation for *Ubuntu
archive builds* and using embedded sources (again, if it helps to leave
the sources in the orig.tar.gz for other reasons, fine):
* embedded that seem obviously ok to leave alone and use during the build:
* github.com/juju/* (do pull out anything that's reused in other go sources though)
* gopkg.in/juju/* (same here)
* embedded that seems clear should be cleaned up/pulled out/use existing archive -dev package:
* code.google.com/p/go.crypto (use golang-go.crypto-dev, why this and golang.org/x/crypto? pick one)
* golang.org/x/crypto (why this and code.google.com/p/go.crypto? pick one)
* golang.org/x/net (use golang-golang-x-net-dev)
* gopkg.in/check.v1 (use golang-check.v1-dev)
* gopkg.in/mgo.v2 (use golang-gopkg-mgo.v2-dev)
* gopkg.in/yaml.v1 (use golang-goyaml-dev, but consider golang-yaml.v2-dev cause that is what snappy uses and the MIR process advocates for one package per functionality as much as possible)
* embedded that may need other Canonical upstream involvement to create golang-*-dev packages (note, LXD is applying for MIR in 15.10 so this golxc might fall under that work):
* launchpad.net/golxc (seems our lxc packages should be adjusted to provide golang-golxc-dev?)
* launchpad.net/gomaasapi (seems like our maas packages should be adjusted to provide golang-gomaasapi-dev?)
* embedded sources where is is unclear if it is juju-specific or something that should be broken out
* bitbucket.org/kardianos/osext (seems like something useful to others? ie, break out?)
* bitbucket.org/kardianos/service (seems like something useful to others? ie, break out?)
* code.google.com/p/winsvc (seems like something useful to others? ie, break out?)
* github.com/bmizerany/pat (??)
* github.com/joyent/* (juju-specific?)
* gopkg.in/natefinch/lumberjack.v2 (??)
* gopkg.in/natefinch/npipe.v2 (seems like something useful to others? ie, break out?)
* launchpad.net/gnuflag (seems like something useful to others? ie, break out?)
* launchpad.net/goamz/* (juju-specific? (amazon))
* launchpad.net/goose/* (juju-specific? (openstack))
* launchpad.net/gwacl/* (seems like something useful to others? ie, break out?)
* launchpad.net/tomb (??)
Questions:
* juju team: can you comment on the package breakdown? For items requiring further discussion, it might be worthwhile understanding how often you are updating the embedded package (useful for the SRU question, below)
* SRU team: juju-core already has a release exception. For packages that are being broken out that were formerly part of the juju-core package and that the juju team will now maintain, can those just be given a release exception?
* Ubuntu Archive team: juju-core will likely need a PPU for members of the juju team when it goes to main. Can we extend the acl to include the packages that are being broken out that they are going to maintain?
* MIR team: for the packages that are being broken out, I propose that they don't get extended MIR review, but rather simply the packaging review to make sure they are following the Go standards as outlined in the MIRteam document
* juju team (/security team): the juju team has said that they would like coordination of security updates for juju-core and golang-*-dev packages for which they maintain. I propose the security team maintains a list of packages and when we triage a CVE against a package in that list, we file a bug for the juju team to fix, and sponsor their uploads (like for other Canonical upstreams). juju team-- does that address your concerns?
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to golang in Ubuntu.
https://bugs.launchpad.net/bugs/1267393
Title:
[MIR] juju-core, juju-mongodb, gccgo, golang
Status in gccgo-5 package in Ubuntu:
Fix Released
Status in gccgo-go package in Ubuntu:
Invalid
Status in golang package in Ubuntu:
Incomplete
Status in juju-core package in Ubuntu:
Confirmed
Status in juju-mongodb package in Ubuntu:
Incomplete
Bug description:
>> golang <<
Availability
------------
In universe, limited to amd64, i386 and armhf archs.
Rationale
---------
golang is the primary focus of Go toolchain development; scale testing
of juju with gccgo and golang gc built binaries revealed that the
gccgo built binaries consume a significant amount of memory compared
to golang gc built versions.
As juju is focussed on building scale out service deployments,
choosing the toolchain that produces the most scalable binaries on the
architectures that most users are going to be using would make sense.
Security
--------
Can't find any interesting security history.
QA
--
OK; the toolchain does have a test suite but its not run by default
(yet).
Dependencies
------------
All build-deps are in main; some non-core packages depend on package
in universe (kate, vim addons) - recommend that these are left in
universe.
golang-go.tools can be demoted to a suggested to limit scope of main
inclusion.
Standards compliance
--------------------
OK
Maintenance
-----------
>> gccgo-go <<
Availability
------------
In universe, available on all required architectures (x86, armhf,
arm64, ppc64el).
Rationale
---------
'go' build tool built using gccgo, avoiding the need to promote two
golang toolchains to Ubuntu main.
Security
--------
Searching for golang CVE's turned up nothing (this package is a rename
of the golang 1.2 source package).
Quality assurance
-----------------
Package installs cleanly, go tool installed using alternatives at
higher priority that golang-go version in universe.
Dependencies
------------
gccgo is in universe, all other dependencies in main.
Standards compliance
--------------------
OK
Maintenance
-----------
Some bugs expected upfront but should stabilize before release.
Probably picked up by ubuntu-server if foundations team don't want to.
Background information
----------------------
This package is a re-cut of the golang upstream codebase with selected
cherry-picks from upstream VCS for gccgo support, along with a patch
to support building using gccgo + Make.
The only code actually used is in src/cmd/go.
>> juju-mongodb <<
Availability
------------
In universe, available on all required architectures (x86, armhf,
arm64, ppc64el).
Rationale
---------
MongoDB is a dependency for operating a Juju deployed Ubuntu
environment.
Security
--------
MongoDB has had some CVE's in the past, related to the use of the V8
and Spidermonkey Javascript engine in the Mongo Shell; however juju-
mongodb builds without support for Javascript scripting, avoiding the
historic CVE's (which where fixed upstream anyway).
Quality assurance
-----------------
Package installs cleanly, package build process runs upstream smoke
tests (minus jstests due to disabling javascript support). Tests
pass on all architectures.
Dependencies
------------
All in main already
Standards compliance
--------------------
OK (well is scons but we won't hold that against it)
Maintenance
-----------
Upstream MongoDB run stable releases with point updates; its intended
that a MRE is applied for this package so point releases can be pushed
as SRU's.
Its also possible that we might need to bump a major version (2.4.x ->
2.6.x); as this package is specific to Juju, we can constrain the
impact and regression testing to Juju only.
Background information
----------------------
Why a separate package? it was agreed at the last vUDS that having a
separate package allows us to limit a) use of v8 (disabled) which was
a security concern and b) allows us to potentially update at a later
date if need be only impacting juju itself.
>> juju-core <<
Availability
------------
In universe.
Rationale
---------
Juju is the recommended service orchestration tool for Ubuntu; as such
it really needs to be a core part of Ubuntu.
Security
--------
No security history, but it was agreed that a security review would be
undertaken as part of the MIR process.
Quality assurance
-----------------
No tests are run as part of the package build process; however
upstream do run these tests for all target series (12.04 -> 14.04)
prior to release so the overall quality of the codebase it pretty
good.
The package has some basic DEP-8 tests that bootstrap a local Juju
environment to ensure everything hangs together OK.
Dependencies
------------
juju-mongodb (see above)
gccgo + gccgo-go
Currently all required go dependencies are snapshotted at specific
upstream commits and bundled with Juju.
Standards compliance
--------------------
OK
Maintenance
-----------
Upstream Juju team intend to manage stable point releases against the
version shipped in 14.04. Ubuntu Server team will own the package in
distro.
Background information
----------------------
Some decisions still need to be made, mainly around toolchain.
Specifically the aim is to support a single Go toolchain in Ubuntu
main for all architectures; golang-go does not support arm64 or
ppc64el yet, whereas the gccgo implementation does.
Required changes to support gccgo have been upstreamed into the Juju
codebase.
Its also worth noting that the package and binaries in Ubuntu are used
for:
client tool (juju)
juju agent (jujud) - but only for local provider and where --upload-tools is used
Upstream released jujud binaries are/will be distributed officially
via simplestreams using a documented build process (details TBC). The
juju client will use these tools on public clouds and potentially in
private cloud deployments where tools are synced into the cloud using
the juju client tool (juju sync-tools).
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/gccgo-5/+bug/1267393/+subscriptions
More information about the foundations-bugs
mailing list