kernel-series-info.yaml redux

[Andy Whitcroft]

We have been talking various forums about issues we have with the
kernel-series-info.yaml file.  This file is nominally meant to describe
all series of Ubuntu and the kernels contained therein.  We are starting
to want to record more information about the kernels such as adding
repository information for mainline-builds and cve-autotriage, adding snap
tracking information for shankbot etc.  As we try and shoehorn these into
the existing data format it is clear that the file is the wrong shape to
make this easy and extensible.

Currently the data really only considers series.  We then are storing
package specific information as per series dictionaries further keyed by
the package.  With multiple attributes this is just becoming hard to read
and even harder to validate.  We have multiple lists which contain
either debian packages, or primary source packages.  There are many
subtle interactions such as if a package is in derivative-packages and
not in packages then the source is not supported; nice and obvious.
Packages which are rebased on sources in the same series are specified
in a completely different manner to those in a different series; ugg.

I have taken a step back and looked at what we are trying to encode.  We
are trying to store attributes of a series; name, development, supported
etc.  We are also trying to store attributes for a source; supported,
version, related debian packges, related snap packages, and what we are
derived from.

To this end I defined a new kernel-series.yaml format and converted over
the existing data to the new form.  Below is an example record for
17.10.  Note that each kernel is referred to by the primary source
package name.  Everything related to that source package sits below it.
We have one list of dependent packages (.debs), there is also a snap
package list when needed.  The relationship between packages is always
expressed as a derived-from attribute pointing to a series and source
pair:

    # 17.10 (artful)
    '17.10':
	name: artful
	development: false
	supported: true
	sources:
	    linux:
		versions: ['4.11.0', '4.12.0', '4.13.0']
		packages:
		    linux:
			repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/artful']
		    linux-meta:
			type: meta
			repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta/+git/artful']
		    linux-signed:
			type: signed
			repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-signed/+git/artful']
		supported: true

	    linux-raspi2:
		packages:
		    linux-raspi2:
			repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-raspi2/+git/artful', 'raspi2']
		    linux-meta-raspi2:
			type: meta
			repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta-raspi2/+git/artful', 'raspi2']
		supported: true
		derived-from: [ '17.10', 'linux' ]

Snaps are represented in a similar form to packages:

    sources:
        linux:
            versions: ['4.4.0']
            packages:
                linux:
                    repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial']
                linux-meta:
                    type: meta
                    repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta/+git/xenial']
                linux-signed:
                    type: signed
                    repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-signed/+git/xenial']
            snaps:
                caracalla-kernel:
                pc-kernel:
                    primary: true
                    repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-snap/+git/xenial', 'pc']
                    gated: false
                    stable: true
                    qa: false
                    hw-cert: true
                    arches: ['amd64', 'i386']
                stlouis-kernel:
            supported: true

The full new format file is available[1] and contains full documentation
on the meaning of the current fields.  The aim is to make this a sensible
form to add new attributes as we need them, such as the proposed-only
blocker for shanky is clearly a source related attribute and would get
into the sources/linux attribute set; proposed-only: true.

We have a few wrinkles with any attempt to change the format of this file.
The biggest of which is that the live ubuntu.py library consumes the
existing ktl/kernel-series-info.yaml from the launchpad git repository
directly.  This means it is essentially impossible to change the
original file without first changing all of the consumers to have new
code.  For this reason I am proposing to place the new V2 format file in
the (slightly more sensibly named) info/kernel-series.yaml in the same
repository.  We have converters to produce the old V1 format file from
that to keep un-upgraded consumers working in the short term.

The second wrinkle is that the current implementation of ubuntu.py
hands out the entire data structure for direct inspection.  This is
something else I would like to improve but in the short term it means
that even with an updated ubuntu.py we would need to expose the data in
the old form.  I am proposing to use the converter above to convert the
file as it is loaded into the old form so unconverted consumers will
still function but would be unable to access new attributes.

I am further proposing to start a small project to create proper
accessors for the various data elements in this data so that we can
convert consumers over to those.  This would be ongoing once we have
concensus that the new format makes sense to us.

In parallel I would like to start converting some of the other edge code
over to this data, such as mainline builds which have their own out of
date lists of repositories and the like.

Comments appreciated on the new format, its extensibility, and the
overall plan and direction.

-apw

[1] https://git.launchpad.net/~canonical-kernel/+git/kteam-tools/plain/info/kernel-series.yaml