ACK: [SRU bionic/focal/groovy/hirsure LRM] [PATCH 0/9] LP#1918134 -- LRMv4 switch to signing with Ubuntu Kernel Modules signing key

Seth Forshee seth.forshee at canonical.com
Mon Mar 8 22:33:06 UTC 2021


On Mon, Mar 08, 2021 at 02:59:55PM +0000, Andy Whitcroft wrote:
> Currently we build Nvidia packages in the main kernel package (linux,
> linux-aws etc).  We do this to sign the resulting .ko's into the kernel
> ephemeral modules signing key.  We package up only the added signature
> data (as .sig files) and destroy the remainder.  We later rebuild the .o's
> (and temporarily the .ko's) in linux-restricted-modules* and include the
> .o's so produced and the .sig's as produced by the main kernel build in
> LRM's binary packages.  These are then reconstructed on installation on
> the end-user system.
> 
> Now this has several unfortunate properties.  Firstly we have to access
> and build dkms packages from the restricted component in the main kernel
> package build (which is hard and error prone).  Secondly, we must be able
> to reproducibly build the .ko's as we build them in both the main package
> and LRM.  (Recently a toolchain change seems to have made this build no
> longer reproducible leading to modules which do not match the produced
> signatures and are thus not loadable.)  Finally, in order to bump the
> version of any particular Nvidia stream (or add or remove a stream) we
> have to rebuild the main kernel package in order to get signatures which
> is very costly and leads to unnecessary kernel updates for people who do
> not even use Nvidia modules.
> 
> As background, in recent kernel uploads we have included two new static
> keys into the kernel `.builtin_trusted_keys`; the Canonical Livepatch key,
> and the Ubuntu Kernel Modules key.  This second key is designed to allow
> signing of driver .ko's into the kernel even after the build is completed.
> With this in place it is no longer necessary for us to build the Nvidia
> packages in the main kernel package to obtain signatures.
> 
> Signing of .ko's with the Ubuntu Drivers Key (UDK) is performed via a
> signing custom binary upload containing those .ko's.  This signing either
> produces a signed .ko file, or optionally a detached .sig signature.
> For our purposes the latter is our desired form as we cannot distribute
> the combined .ko files.  It is very important to note that as we
> cannot distribute the combined .ko files for these Nvidia packages
> (due to liscencing constraints) that we also cannot allow the resultant
> custom binary upload to be distributed either; ie it cannot end up on the
> launchpad librarian or published into the public archive.  This constraint
> is a key design feature if the implementation described below.
> 
> In the new world order (LRMv4) the linux-restricted-modules (LRM) package
> is responsible for the only build of the nvidia .o's and the helpers which
> allow those to be combined on the end-user system to produce final loadable
> .ko's; this avoids the build reproducibility issues.  Two new packages are
> introduced, linux-restricted-generate (LRG) and linux-restricted-signatures
> (LRS).  LRG consumes the raw .o's and helpers and uses those to reconstruct
> the unsigned .ko's and places those into the signing custom binary upload.
> LRS simply downloads the resulting signatures from the archive and
> expresses those directly into the filesystem in the same directory as the
> .o's expressed by LRM.  LRM's binary packages then directly depend on these
> signatures such that when the postinst there runs we are able to add those
> signatures to the unsigned .ko's forming loadable .ko's for the kernel.
> 
> In order to allow LRG to simply consume the raw .o's and build helpers via
> a Build-Depend: we have restructured LRM's binary packages.  We split the
> raw .o's and build helpers from the build dependancies and postinst/prerm
> support.  The former is needed both by LRG and on the end-user system,
> the latter is only needed on the end-user system to construct and
> destroy the signed .ko's.  The raw .o's and build scripts are moved
> into new linux-objects-nvidia-460-ABI-FLAVOUR packages.  The existing
> linux-modules-nvidia-460-ABI-FLAVOUR packages simply depend on that and
> the associated signature package, and carry the postinst/prerm support.
> 
> Building this package set is complex as the LRG package cannot be built
> in a public PPA as that would cause the signing custom binary upload to
> be redistributed by the launchpad librarian.  We therefore must build
> LRG and LRS in a private PPA.  The built LRG and LRS must then be signed
> using the Ubuntu Kernel Modules key (again privatly) and the resultant
> LRS returned for distribution; this occurs via a private signing PPA
> containing the appropriate key; LRG is abandoned at this stage and takes
> no futher part.  As LRM and LRS are redistributable (and indeed both are
> required to install the drivers) LRS is returned to the public build PPA
> where LRM was built and enters the normal pipeline.
> 
> As LRG depends on binary packages produced by LRM the private PPA in which
> LRG will build must depend on the PPA in which LRM builds.  To support
> this we introduce an adjunct PPA for each public build PPA which will
> produce kernels which contain an LRM component.  These are needed for
> both of the stable PPAs (ppa and ppa2) and the development PPAs (bootstrap
> and unstable); each is simply named using the primary PPA name with a -ps
> (Private Signing) suffix.  Below is a flow-chart showing the lifecycles
> for these packages:
> 
>          LRM                 LRS LRG
>           |   .LRS(-b).       |   |
>           v   v       |       v   v
>     +------------+    |   +------------+
>     |    ppa     |    |   |   ppa-ps   |
>     +------------+    |   +------------+
>           |           |       |   |
>           |           |      LRS LRG(-b)
>           v           |       v   v
>     +------------+    |   +------------+
>     | -proposed  |    `---|  signing   |
>     +------------+        +------------+
> 
> This new proceedure is complex and error prone as (nominally) this requires
> the addition and maintenance of two new packages.  Also of the three
> packages in this set one is initially uploaded to the public PPA with the
> rest of the kernel (linux, linux-signed, and linux-meta) and the other two
> must be uploaded to a specific related private PPA (it must depend on the
> associated public PPA); indeed if LRG is uploaded to the public PPA we may
> end up redistributing nvidia .ko's which must be prevented at all costs.
> 
> To avoid the complexity and associated risk of misplacing these new
> packages the LRG and LRS package source are folded into the LRM package.
> This package will continue to be maintained in the normal way and uploaded
> to the public build PPA as normal, meaning no process change for crankers.
> LRS and LRG are (intentionally) generable from LRM simply by changing the
> name of the source package in debian/changelog.  A cron based helper will
> monitor the primary PPAs for new LRM uploads and automatically convert
> that into the necessary LRG/LRS pairing and upload it to the correct
> adjunct PPA.  The tracking bugs associated with our uploads will drive
> the process from there to sign and return LRS to the primary build PPA.
> 
> The patch stack below represents the nominal conversion of the
> hirsute:linux package from LRMv3 to LRMv4.  However (as we did with LRMv3)
> I proposed to apply this update programatically to all LRM packages
> once this is reviewed.
> 
> This stack does not contain the (further) changes we will apply to the main
> kernel packages to stop building the nvidia signatures there.  We also
> continue to assume that the primary kernel package for a version-series
> (bionic:linux) still carries the definative debian/dkms-versions file so
> that we retain consistent versioning on these packages[1]; though this
> does not preclude LRMs being respun independently during a cycle.
> 
> -apw
> 
> [1] the next phase of this work will revisit the home for this version
>     data going forward, and the aquisition of the DKMS packages during
>     the builds, and later.

I think I've got a pretty good handle now on of how this all works, and
it makes sense to me. The process is convoluted, with sort of a circular
chain of dependencies, but that's unfortunately necessary and it brings
maintenance benefits that I think are worthwhile. Having looked at a
tree and a test build with these changes, the results look reasonable.

Acked-by: Seth Forshee <seth.forshee at canonical.com>



More information about the kernel-team mailing list