linux-headers-5.15.0-1027-gke is removed from jammy last night]

Phil Roche phil.roche at canonical.com
Thu May 11 09:17:29 UTC 2023 

@tailinchu at gmail.com Hi,

I work on the Canonical Public Cloud team, and I work with the GKE team to
build and publish the Ubuntu GKE node images.


> ---------- Forwarded message ----------
> From: Tai-Lin Chu <tailinchu at gmail.com>
> To: Steve Langasek <steve.langasek at canonical.com>, Dimitri John Ledkov <
> dimitri.ledkov at canonical.com>, Tai-Lin Chu <tailinchu at gmail.com>, Stefan
> Bader <stefan.bader at canonical.com>, kernel-team at lists.ubuntu.com
> Cc:
> Bcc:
> Date: Wed, 10 May 2023 12:16:52 -0700
> Subject: Re: linux-headers-5.15.0-1027-gke is removed from jammy last night
> Thanks for replying.
> Let me clarify the use case: we use gke ubuntu containerd node image.
> Because we cannot know which kernel version the node is using, on vm
> creation completed, we will compile and inject a kernel module so that
> pods can use that kernel module:
>
> apt-get install -y "linux-headers-$(uname -r)"
> "linux-modules-extra-$(uname -r)"
>
> > Is 5.15.0-1027-gke the version of a running kernel on your GKE instance?
> yes.
>
> >  If so, why is the headers package not installed already?
> I don't think the kernel header package exists on the node image
>
> https://cloud.google.com/kubernetes-engine/docs/concepts/node-images#ubuntu-variants
> .
>

The headers are pre-installed in all the Ubuntu node images

```
~$ apt list --installed | grep headers
linux-gke-headers-5.15.0-1028/now 5.15.0-1028.33 amd64 [installed,local]
linux-headers-5.15.0-1028-gke/now 5.15.0-1028.33 amd64 [installed,local]
linux-headers-gke-5.15/now 5.15.0.1028.27 amd64 [installed,local]
```


> From that mailing list post, that timeline was not publicly announced
> on gke updates, so we were completely unaware of it, and caused
> incidents. Our current workaround is using 20.04 with the older gke
> version because the new auto cleanup process is for jammy and later,
> but I hope that older kernel can be kept around.
>

Aside from any discussions about the pruning of the headers from the
archive, another workaround or  - permanent solution - is to bind mount
what you need from the host to the container. The headers are present on
the node so this should work.


> Does canonical collaborate gcp to create node image?


Yes, we do.


> If so, I can also
> forward this to our gcp dedicated account team.
>

Yes, I see there was an internal case raised now which I will respond to.

Phil


>
> Thanks!
>
> On Wed, May 10, 2023 at 8:29 AM Steve Langasek
> <steve.langasek at canonical.com> wrote:
> >
> > On Wed, May 10, 2023 at 08:12:50AM -0700, Steve Langasek wrote:
> > > When Andy and I looked at this, the analysis had showed that all our
> images
> > > except for minimal images were being built with linux-$flavor
> installed,
> > > rather than linux-image-$flavor, so that the headers would already be
> > > present and removal of the old ABIs from the archive would have no
> impact on
> > > users of these images.
> > >
> > > The GKE images are being built on a minimal base, which I did not
> recall.
> > >
> > > It is not otherwise an issue for cloud images per above, with the
> exception
> > > of the cloud-minimal images.
> > >
> > > Since the linux-gke flavor is used only for minimal GKE images, we
> could
> > > reasonably exclude these from the NBS cleaning going forward.
> >
> > > I've reached out to our Public Cloud team to see what their preference
> is.
> >
> > I've clarified with the Public Cloud team that, although the GKE images
> use
> > a minimal base, the images are built using linux-gke-$version, NOT
> > linux-image-gke-$version.
> >
> > So it's unclear that what Tai-Lin is doing is a use case that the Public
> > Cloud team is concerned with supporting.
> >
> > Is 5.15.0-1027-gke the version of a running kernel on your GKE
> instance?  If
> > so, why is the headers package not installed already?  If not, why are
> you
> > trying to compile kernel modules for this version?
> >
> > > > Separately, whilst this NBS cleanup is in place, you can use the tool
> > > > `pull-lp-debs` from ubuntu-dev-tools which should allow you to
> > > > securely fetch any of the packages you require out of Launchpad
> > > > Librarian archival service. (note that pull-lp-debs is part of
> > > > collection of tools pull-ppa-ddebs pull-ppa-debs pull-ppa-source -
> > > > which are all wrappers around the swiss army knife type of tool
> > > > pull-pkg which can pull anything and everything out of Launchpad,
> > > > PPAs, Debian)
> > >
> > >
> > > > > Best,
> > > > >
> > > > > On Wed, May 10, 2023 at 12:51 AM Stefan Bader
> > > > > <stefan.bader at canonical.com> wrote:
> > > > > >
> > > > > > On 09.05.23 22:28, Tai-Lin Chu wrote:
> > > > > > > hi,
> > > > > > > I received alerts about linux-headers-5.15.0-1027-gke being
> removed last night.
> > > > > > > What might be the reason for that? thanks!
> > > > > > >
> > > > > > > Get:3 http://security.ubuntu.com/ubuntu
> jammy-security/restricted
> > > > > > > amd64 Packages [1077 kB]
> > > > > > > Get:4 http://archive.ubuntu.com/ubuntu jammy-updates
> InRelease [119 kB]
> > > > > > > Get:5 http://archive.ubuntu.com/ubuntu jammy-backports
> InRelease [108 kB]
> > > > > > > Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64
> Packages [1792 kB]
> > > > > > > Get:7 http://security.ubuntu.com/ubuntu jammy-security/main
> amd64
> > > > > > > Packages [585 kB]
> > > > > > > Get:8 http://security.ubuntu.com/ubuntu
> jammy-security/universe amd64
> > > > > > > Packages [898 kB]
> > > > > > > Get:9 http://security.ubuntu.com/ubuntu
> jammy-security/multiverse
> > > > > > > amd64 Packages [41.2 kB]
> > > > > > > Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64
> Packages [17.5 MB]
> > > > > > > Get:11 http://archive.ubuntu.com/ubuntu jammy/multiverse
> amd64 Packages [266 kB]
> > > > > > > Get:12 http://archive.ubuntu.com/ubuntu jammy/restricted
> amd64 Packages [164 kB]
> > > > > > > Get:13 http://archive.ubuntu.com/ubuntu
> jammy-updates/multiverse amd64
> > > > > > > Packages [46.6 kB]
> > > > > > > Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main
> amd64
> > > > > > > Packages [992 kB]
> > > > > > > Get:15 http://archive.ubuntu.com/ubuntu
> jammy-updates/restricted amd64
> > > > > > > Packages [1137 kB]
> > > > > > > Get:16 http://archive.ubuntu.com/ubuntu
> jammy-updates/universe amd64
> > > > > > > Packages [1143 kB]
> > > > > > > Get:17 http://archive.ubuntu.com/ubuntu
> jammy-backports/universe amd64
> > > > > > > Packages [25.6 kB]
> > > > > > > Get:18 http://archive.ubuntu.com/ubuntu jammy-backports/main
> amd64
> > > > > > > Packages [49.4 kB]
> > > > > > > Fetched 26.3 MB in 2s (10.6 MB/s)
> > > > > > > Reading package lists...
> > > > > > > + apt-get install -y linux-headers-5.15.0-1027-gke
> > > > > > > linux-modules-extra-5.15.0-1027-gke
> > > > > > > Reading package lists...
> > > > > > > Building dependency tree...
> > > > > > > Reading state information...
> > > > > > > E: Unable to locate package linux-headers-5.15.0-1027-gke
> > > > > > > E: Couldn't find any package by glob
> 'linux-headers-5.15.0-1027-gke'
> > > > > > > E: Couldn't find any package by regex
> 'linux-headers-5.15.0-1027-gke'
> > > > > > >
> > > > > >
> > > > > > That is just the normal way things change with updates. The
> archive will
> > > > > > not find older version which have been replaced by newer ones.
> You
> > > > > > should never try to install specific versions for that reason.
> Try "apt
> > > > > > install linux-headers-gke".
> > > > > > --
> > > > > > - Stefan
> > > > > >
> > > > >
> > > > > --
> > > > > kernel-team mailing list
> > > > > kernel-team at lists.ubuntu.com
> > > > > https://lists.ubuntu.com/mailman/listinfo/kernel-team
> > > >
> > > >
> > > >
> > > > --
> > > > okurrr,
> > > >
> > > > Dimitri
> > > >
> > >
> > > --
> > > Steve Langasek                   Give me a lever long enough and a
> Free OS
> > > Debian Developer                   to set it on, and I can move the
> world.
> > > Ubuntu Developer
> https://www.debian.org/
> > > slangasek at ubuntu.com
> vorlon at debian.org
> >
> > --
> > Steve Langasek                   Give me a lever long enough and a Free
> OS
> > Debian Developer                   to set it on, and I can move the
> world.
> > Ubuntu Developer
> https://www.debian.org/
> > slangasek at ubuntu.com
> vorlon at debian.org
>
>

-- 
Phil Roche
Staff Software Engineer
Canonical Public Cloud
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20230511/d444e1dc/attachment.html>

More information about the kernel-team mailing list