kernel-series-info.yaml redux

[Stefan Bader]

On 14.11.2017 16:02, Andy Whitcroft wrote:
> We have been talking various forums about issues we have with the
> kernel-series-info.yaml file.  This file is nominally meant to describe
> all series of Ubuntu and the kernels contained therein.  We are starting
> to want to record more information about the kernels such as adding
> repository information for mainline-builds and cve-autotriage, adding snap
> tracking information for shankbot etc.  As we try and shoehorn these into
> the existing data format it is clear that the file is the wrong shape to
> make this easy and extensible.
> 
> Currently the data really only considers series.  We then are storing
> package specific information as per series dictionaries further keyed by
> the package.  With multiple attributes this is just becoming hard to read
> and even harder to validate.  We have multiple lists which contain
> either debian packages, or primary source packages.  There are many
> subtle interactions such as if a package is in derivative-packages and
> not in packages then the source is not supported; nice and obvious.
> Packages which are rebased on sources in the same series are specified
> in a completely different manner to those in a different series; ugg.
> 
> I have taken a step back and looked at what we are trying to encode.  We
> are trying to store attributes of a series; name, development, supported
> etc.  We are also trying to store attributes for a source; supported,
> version, related debian packges, related snap packages, and what we are
> derived from.
> 
> To this end I defined a new kernel-series.yaml format and converted over
> the existing data to the new form.  Below is an example record for
> 17.10.  Note that each kernel is referred to by the primary source
> package name.  Everything related to that source package sits below it.
> We have one list of dependent packages (.debs), there is also a snap
> package list when needed.  The relationship between packages is always
> expressed as a derived-from attribute pointing to a series and source
> pair:

The layout feels sensible from the layering. At this time of day, the only thing
that trips me a bit is the naming of "sources" but would not know a better name
either. "kernels" is probably not clearer as "sources"...

-Stefan
> 
>     # 17.10 (artful)
>     '17.10':
> 	name: artful
> 	development: false
> 	supported: true
> 	sources:
> 	    linux:
> 		versions: ['4.11.0', '4.12.0', '4.13.0']
> 		packages:
> 		    linux:
> 			repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/artful']
> 		    linux-meta:
> 			type: meta
> 			repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta/+git/artful']
> 		    linux-signed:
> 			type: signed
> 			repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-signed/+git/artful']
> 		supported: true
> 
> 	    linux-raspi2:
> 		packages:
> 		    linux-raspi2:
> 			repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-raspi2/+git/artful', 'raspi2']
> 		    linux-meta-raspi2:
> 			type: meta
> 			repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta-raspi2/+git/artful', 'raspi2']
> 		supported: true
> 		derived-from: [ '17.10', 'linux' ]
> 
> Snaps are represented in a similar form to packages:
> 
>     sources:
>         linux:
>             versions: ['4.4.0']
>             packages:
>                 linux:
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial']
>                 linux-meta:
>                     type: meta
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-meta/+git/xenial']
>                 linux-signed:
>                     type: signed
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-signed/+git/xenial']
>             snaps:
>                 caracalla-kernel:
>                 pc-kernel:
>                     primary: true
>                     repo: ['git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-snap/+git/xenial', 'pc']
>                     gated: false
>                     stable: true
>                     qa: false
>                     hw-cert: true
>                     arches: ['amd64', 'i386']
>                 stlouis-kernel:
>             supported: true
> 
> The full new format file is available[1] and contains full documentation
> on the meaning of the current fields.  The aim is to make this a sensible
> form to add new attributes as we need them, such as the proposed-only
> blocker for shanky is clearly a source related attribute and would get
> into the sources/linux attribute set; proposed-only: true.
> 
> We have a few wrinkles with any attempt to change the format of this file.
> The biggest of which is that the live ubuntu.py library consumes the
> existing ktl/kernel-series-info.yaml from the launchpad git repository
> directly.  This means it is essentially impossible to change the
> original file without first changing all of the consumers to have new
> code.  For this reason I am proposing to place the new V2 format file in
> the (slightly more sensibly named) info/kernel-series.yaml in the same
> repository.  We have converters to produce the old V1 format file from
> that to keep un-upgraded consumers working in the short term.
> 
> The second wrinkle is that the current implementation of ubuntu.py
> hands out the entire data structure for direct inspection.  This is
> something else I would like to improve but in the short term it means
> that even with an updated ubuntu.py we would need to expose the data in
> the old form.  I am proposing to use the converter above to convert the
> file as it is loaded into the old form so unconverted consumers will
> still function but would be unable to access new attributes.
> 
> I am further proposing to start a small project to create proper
> accessors for the various data elements in this data so that we can
> convert consumers over to those.  This would be ongoing once we have
> concensus that the new format makes sense to us.
> 
> In parallel I would like to start converting some of the other edge code
> over to this data, such as mainline builds which have their own out of
> date lists of repositories and the like.
> 
> Comments appreciated on the new format, its extensibility, and the
> overall plan and direction.
> 
> -apw
> 
> [1] https://git.launchpad.net/~canonical-kernel/+git/kteam-tools/plain/info/kernel-series.yaml
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20171114/4662a112/attachment.sig>