ACK: [SRU bionic/focal/groovy/hirsure LRM] [PATCH 0/9] LP#1918134 -- LRMv4 switch to signing with Ubuntu Kernel Modules signing key
Seth Forshee
seth.forshee at canonical.com
Mon Mar 8 22:33:06 UTC 2021
On Mon, Mar 08, 2021 at 02:59:55PM +0000, Andy Whitcroft wrote:
> Currently we build Nvidia packages in the main kernel package (linux,
> linux-aws etc). We do this to sign the resulting .ko's into the kernel
> ephemeral modules signing key. We package up only the added signature
> data (as .sig files) and destroy the remainder. We later rebuild the .o's
> (and temporarily the .ko's) in linux-restricted-modules* and include the
> .o's so produced and the .sig's as produced by the main kernel build in
> LRM's binary packages. These are then reconstructed on installation on
> the end-user system.
>
> Now this has several unfortunate properties. Firstly we have to access
> and build dkms packages from the restricted component in the main kernel
> package build (which is hard and error prone). Secondly, we must be able
> to reproducibly build the .ko's as we build them in both the main package
> and LRM. (Recently a toolchain change seems to have made this build no
> longer reproducible leading to modules which do not match the produced
> signatures and are thus not loadable.) Finally, in order to bump the
> version of any particular Nvidia stream (or add or remove a stream) we
> have to rebuild the main kernel package in order to get signatures which
> is very costly and leads to unnecessary kernel updates for people who do
> not even use Nvidia modules.
>
> As background, in recent kernel uploads we have included two new static
> keys into the kernel `.builtin_trusted_keys`; the Canonical Livepatch key,
> and the Ubuntu Kernel Modules key. This second key is designed to allow
> signing of driver .ko's into the kernel even after the build is completed.
> With this in place it is no longer necessary for us to build the Nvidia
> packages in the main kernel package to obtain signatures.
>
> Signing of .ko's with the Ubuntu Drivers Key (UDK) is performed via a
> signing custom binary upload containing those .ko's. This signing either
> produces a signed .ko file, or optionally a detached .sig signature.
> For our purposes the latter is our desired form as we cannot distribute
> the combined .ko files. It is very important to note that as we
> cannot distribute the combined .ko files for these Nvidia packages
> (due to liscencing constraints) that we also cannot allow the resultant
> custom binary upload to be distributed either; ie it cannot end up on the
> launchpad librarian or published into the public archive. This constraint
> is a key design feature if the implementation described below.
>
> In the new world order (LRMv4) the linux-restricted-modules (LRM) package
> is responsible for the only build of the nvidia .o's and the helpers which
> allow those to be combined on the end-user system to produce final loadable
> .ko's; this avoids the build reproducibility issues. Two new packages are
> introduced, linux-restricted-generate (LRG) and linux-restricted-signatures
> (LRS). LRG consumes the raw .o's and helpers and uses those to reconstruct
> the unsigned .ko's and places those into the signing custom binary upload.
> LRS simply downloads the resulting signatures from the archive and
> expresses those directly into the filesystem in the same directory as the
> .o's expressed by LRM. LRM's binary packages then directly depend on these
> signatures such that when the postinst there runs we are able to add those
> signatures to the unsigned .ko's forming loadable .ko's for the kernel.
>
> In order to allow LRG to simply consume the raw .o's and build helpers via
> a Build-Depend: we have restructured LRM's binary packages. We split the
> raw .o's and build helpers from the build dependancies and postinst/prerm
> support. The former is needed both by LRG and on the end-user system,
> the latter is only needed on the end-user system to construct and
> destroy the signed .ko's. The raw .o's and build scripts are moved
> into new linux-objects-nvidia-460-ABI-FLAVOUR packages. The existing
> linux-modules-nvidia-460-ABI-FLAVOUR packages simply depend on that and
> the associated signature package, and carry the postinst/prerm support.
>
> Building this package set is complex as the LRG package cannot be built
> in a public PPA as that would cause the signing custom binary upload to
> be redistributed by the launchpad librarian. We therefore must build
> LRG and LRS in a private PPA. The built LRG and LRS must then be signed
> using the Ubuntu Kernel Modules key (again privatly) and the resultant
> LRS returned for distribution; this occurs via a private signing PPA
> containing the appropriate key; LRG is abandoned at this stage and takes
> no futher part. As LRM and LRS are redistributable (and indeed both are
> required to install the drivers) LRS is returned to the public build PPA
> where LRM was built and enters the normal pipeline.
>
> As LRG depends on binary packages produced by LRM the private PPA in which
> LRG will build must depend on the PPA in which LRM builds. To support
> this we introduce an adjunct PPA for each public build PPA which will
> produce kernels which contain an LRM component. These are needed for
> both of the stable PPAs (ppa and ppa2) and the development PPAs (bootstrap
> and unstable); each is simply named using the primary PPA name with a -ps
> (Private Signing) suffix. Below is a flow-chart showing the lifecycles
> for these packages:
>
> LRM LRS LRG
> | .LRS(-b). | |
> v v | v v
> +------------+ | +------------+
> | ppa | | | ppa-ps |
> +------------+ | +------------+
> | | | |
> | | LRS LRG(-b)
> v | v v
> +------------+ | +------------+
> | -proposed | `---| signing |
> +------------+ +------------+
>
> This new proceedure is complex and error prone as (nominally) this requires
> the addition and maintenance of two new packages. Also of the three
> packages in this set one is initially uploaded to the public PPA with the
> rest of the kernel (linux, linux-signed, and linux-meta) and the other two
> must be uploaded to a specific related private PPA (it must depend on the
> associated public PPA); indeed if LRG is uploaded to the public PPA we may
> end up redistributing nvidia .ko's which must be prevented at all costs.
>
> To avoid the complexity and associated risk of misplacing these new
> packages the LRG and LRS package source are folded into the LRM package.
> This package will continue to be maintained in the normal way and uploaded
> to the public build PPA as normal, meaning no process change for crankers.
> LRS and LRG are (intentionally) generable from LRM simply by changing the
> name of the source package in debian/changelog. A cron based helper will
> monitor the primary PPAs for new LRM uploads and automatically convert
> that into the necessary LRG/LRS pairing and upload it to the correct
> adjunct PPA. The tracking bugs associated with our uploads will drive
> the process from there to sign and return LRS to the primary build PPA.
>
> The patch stack below represents the nominal conversion of the
> hirsute:linux package from LRMv3 to LRMv4. However (as we did with LRMv3)
> I proposed to apply this update programatically to all LRM packages
> once this is reviewed.
>
> This stack does not contain the (further) changes we will apply to the main
> kernel packages to stop building the nvidia signatures there. We also
> continue to assume that the primary kernel package for a version-series
> (bionic:linux) still carries the definative debian/dkms-versions file so
> that we retain consistent versioning on these packages[1]; though this
> does not preclude LRMs being respun independently during a cycle.
>
> -apw
>
> [1] the next phase of this work will revisit the home for this version
> data going forward, and the aquisition of the DKMS packages during
> the builds, and later.
I think I've got a pretty good handle now on of how this all works, and
it makes sense to me. The process is convoluted, with sort of a circular
chain of dependencies, but that's unfortunately necessary and it brings
maintenance benefits that I think are worthwhile. Having looked at a
tree and a test build with these changes, the results look reasonable.
Acked-by: Seth Forshee <seth.forshee at canonical.com>
More information about the kernel-team
mailing list