kernel-series-info.yaml redux

Tue Nov 14 15:40:30 UTC 2017

On Tue, Nov 14, 2017 at 03:02:25PM +0000, Andy Whitcroft wrote:
> We have been talking various forums about issues we have with the
> kernel-series-info.yaml file.  This file is nominally meant to describe
> all series of Ubuntu and the kernels contained therein.  We are starting
> to want to record more information about the kernels such as adding
> repository information for mainline-builds and cve-autotriage, adding snap
> tracking information for shankbot etc.  As we try and shoehorn these into
> the existing data format it is clear that the file is the wrong shape to
> make this easy and extensible.
> 
> Currently the data really only considers series.  We then are storing
> package specific information as per series dictionaries further keyed by
> the package.  With multiple attributes this is just becoming hard to read
> and even harder to validate.  We have multiple lists which contain
> either debian packages, or primary source packages.  There are many
> subtle interactions such as if a package is in derivative-packages and
> not in packages then the source is not supported; nice and obvious.
> Packages which are rebased on sources in the same series are specified
> in a completely different manner to those in a different series; ugg.
> 
> I have taken a step back and looked at what we are trying to encode.  We
> are trying to store attributes of a series; name, development, supported
> etc.  We are also trying to store attributes for a source; supported,
> version, related debian packges, related snap packages, and what we are
> derived from.
> 
> To this end I defined a new kernel-series.yaml format and converted over
> the existing data to the new form.  Below is an example record for
> 17.10.  Note that each kernel is referred to by the primary source
> package name.  Everything related to that source package sits below it.
> We have one list of dependent packages (.debs), there is also a snap
> package list when needed.  The relationship between packages is always
> expressed as a derived-from attribute pointing to a series and source
> pair:
> 
>     # 17.10 (artful)
>     '17.10':
> 	name: artful
> 	development: false
> 	supported: true
> 	sources:
> 	    linux:
> 		versions: ['4.11.0', '4.12.0', '4.13.0']
> 		packages:
> 		    linux:
> 			repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/artful']
> 		    linux-meta:
> 			type: meta
> 			repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta/+git/artful']
> 		    linux-signed:
> 			type: signed
> 			repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-signed/+git/artful']
> 		supported: true
> 
> 	    linux-raspi2:
> 		packages:
> 		    linux-raspi2:
> 			repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-raspi2/+git/artful', 'raspi2']
> 		    linux-meta-raspi2:
> 			type: meta
> 			repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta-raspi2/+git/artful', 'raspi2']
> 		supported: true
> 		derived-from: [ '17.10', 'linux' ]
> 
> Snaps are represented in a similar form to packages:
> 
>     sources:
>         linux:
>             versions: ['4.4.0']
>             packages:
>                 linux:
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial']
>                 linux-meta:
>                     type: meta
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta/+git/xenial']
>                 linux-signed:
>                     type: signed
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-signed/+git/xenial']
>             snaps:
>                 caracalla-kernel:
>                 pc-kernel:
>                     primary: true
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-snap/+git/xenial', 'pc']
>                     gated: false
>                     stable: true
>                     qa: false
>                     hw-cert: true
>                     arches: ['amd64', 'i386']
>                 stlouis-kernel:
>             supported: true
> 
> The full new format file is available[1] and contains full documentation
> on the meaning of the current fields.  The aim is to make this a sensible
> form to add new attributes as we need them, such as the proposed-only
> blocker for shanky is clearly a source related attribute and would get
> into the sources/linux attribute set; proposed-only: true.
> 
> We have a few wrinkles with any attempt to change the format of this file.
> The biggest of which is that the live ubuntu.py library consumes the
> existing ktl/kernel-series-info.yaml from the launchpad git repository
> directly.  This means it is essentially impossible to change the
> original file without first changing all of the consumers to have new
> code.  For this reason I am proposing to place the new V2 format file in
> the (slightly more sensibly named) info/kernel-series.yaml in the same
> repository.  We have converters to produce the old V1 format file from
> that to keep un-upgraded consumers working in the short term.

What do you think about defining a new class KernelSeries (for example)
which consumes the new format directly? That class defines the new accessors.
As we go through our existing code base it will be easy to see what remaining
code still imports the old code. In the end, after everything has been
converted over we simply delete ktl/ubuntu.py and move on.

> 
> The second wrinkle is that the current implementation of ubuntu.py
> hands out the entire data structure for direct inspection.  This is
> something else I would like to improve but in the short term it means
> that even with an updated ubuntu.py we would need to expose the data in
> the old form.  I am proposing to use the converter above to convert the
> file as it is loaded into the old form so unconverted consumers will
> still function but would be unable to access new attributes.
> 
> I am further proposing to start a small project to create proper
> accessors for the various data elements in this data so that we can
> convert consumers over to those.  This would be ongoing once we have
> concensus that the new format makes sense to us.
> 
> In parallel I would like to start converting some of the other edge code
> over to this data, such as mainline builds which have their own out of
> date lists of repositories and the like.
> 
> Comments appreciated on the new format, its extensibility, and the
> overall plan and direction.
> 
> -apw
> 
> [1] https://git.launchpad.net/~canonical-kernel/+git/kteam-tools/plain/info/kernel-series.yaml
> 
> -- 
> kernel-team mailing list
> kernel-team at lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/kernel-team

-- 
Brad Figg brad.figg at canonical.com http://www.canonical.com