linux-headers-5.15.0-1027-gke is removed from jammy last night]

Fri May 12 10:30:40 UTC 2023

Apologies I had missed the comment regarding modules-extra as this wasn't
mentioned in the internal case.

I am not willing to share the internal case URL on this public list but it
was filed by porterdavid at google.com and is issue ID 281872648.

Yesterday myself and steve.langasek at canonical.com discussed no longer
pruning the gke and gkeop kernel headers but this didn't extend to
`modules-extra`.  I will get clarification on this today.

Phil

On Thu, 11 May 2023 at 18:33, Tai-Lin Chu <tailinchu at gmail.com> wrote:

> The kernel header is installed but linux-modules-extra is not.
> $ dpkg -L linux-modules-extra-$(uname -r)
> dpkg-query: package 'linux-modules-extra-5.15.0-1027' is not installed
> Use dpkg --contents (= dpkg-deb --contents) to list archive files contents.
>
> > Yes, I see there was an internal case raised now which I will respond to.
>
> could you send me the link to that case?
>
> Thanks!
>
> On Thu, May 11, 2023 at 2:17 AM Phil Roche <phil.roche at canonical.com>
> wrote:
> >
> > @tailinchu at gmail.com Hi,
> >
> > I work on the Canonical Public Cloud team, and I work with the GKE team
> to build and publish the Ubuntu GKE node images.
> >
> >>
> >> ---------- Forwarded message ----------
> >> From: Tai-Lin Chu <tailinchu at gmail.com>
> >> To: Steve Langasek <steve.langasek at canonical.com>, Dimitri John Ledkov
> <dimitri.ledkov at canonical.com>, Tai-Lin Chu <tailinchu at gmail.com>, Stefan
> Bader <stefan.bader at canonical.com>, kernel-team at lists.ubuntu.com
> >> Cc:
> >> Bcc:
> >> Date: Wed, 10 May 2023 12:16:52 -0700
> >> Subject: Re: linux-headers-5.15.0-1027-gke is removed from jammy last
> night
> >> Thanks for replying.
> >> Let me clarify the use case: we use gke ubuntu containerd node image.
> >> Because we cannot know which kernel version the node is using, on vm
> >> creation completed, we will compile and inject a kernel module so that
> >> pods can use that kernel module:
> >>
> >> apt-get install -y "linux-headers-$(uname -r)"
> "linux-modules-extra-$(uname -r)"
> >>
> >> > Is 5.15.0-1027-gke the version of a running kernel on your GKE
> instance?
> >> yes.
> >>
> >> >  If so, why is the headers package not installed already?
> >> I don't think the kernel header package exists on the node image
> >>
> https://cloud.google.com/kubernetes-engine/docs/concepts/node-images#ubuntu-variants
> .
> >
> >
> > The headers are pre-installed in all the Ubuntu node images
> >
> > ```
> > ~$ apt list --installed | grep headers
> > linux-gke-headers-5.15.0-1028/now 5.15.0-1028.33 amd64 [installed,local]
> > linux-headers-5.15.0-1028-gke/now 5.15.0-1028.33 amd64 [installed,local]
> > linux-headers-gke-5.15/now 5.15.0.1028.27 amd64 [installed,local]
> > ```
> >
> >>
> >> From that mailing list post, that timeline was not publicly announced
> >> on gke updates, so we were completely unaware of it, and caused
> >> incidents. Our current workaround is using 20.04 with the older gke
> >> version because the new auto cleanup process is for jammy and later,
> >> but I hope that older kernel can be kept around.
> >
> >
> > Aside from any discussions about the pruning of the headers from the
> archive, another workaround or  - permanent solution - is to bind mount
> what you need from the host to the container. The headers are present on
> the node so this should work.
> >
> >>
> >> Does canonical collaborate gcp to create node image?
> >
> >
> > Yes, we do.
> >
> >>
> >> If so, I can also
> >> forward this to our gcp dedicated account team.
> >
> >
> > Yes, I see there was an internal case raised now which I will respond to.
> >
> > Phil
> >
> >>
> >>
> >> Thanks!
> >>
> >> On Wed, May 10, 2023 at 8:29 AM Steve Langasek
> >> <steve.langasek at canonical.com> wrote:
> >> >
> >> > On Wed, May 10, 2023 at 08:12:50AM -0700, Steve Langasek wrote:
> >> > > When Andy and I looked at this, the analysis had showed that all
> our images
> >> > > except for minimal images were being built with linux-$flavor
> installed,
> >> > > rather than linux-image-$flavor, so that the headers would already
> be
> >> > > present and removal of the old ABIs from the archive would have no
> impact on
> >> > > users of these images.
> >> > >
> >> > > The GKE images are being built on a minimal base, which I did not
> recall.
> >> > >
> >> > > It is not otherwise an issue for cloud images per above, with the
> exception
> >> > > of the cloud-minimal images.
> >> > >
> >> > > Since the linux-gke flavor is used only for minimal GKE images, we
> could
> >> > > reasonably exclude these from the NBS cleaning going forward.
> >> >
> >> > > I've reached out to our Public Cloud team to see what their
> preference is.
> >> >
> >> > I've clarified with the Public Cloud team that, although the GKE
> images use
> >> > a minimal base, the images are built using linux-gke-$version, NOT
> >> > linux-image-gke-$version.
> >> >
> >> > So it's unclear that what Tai-Lin is doing is a use case that the
> Public
> >> > Cloud team is concerned with supporting.
> >> >
> >> > Is 5.15.0-1027-gke the version of a running kernel on your GKE
> instance?  If
> >> > so, why is the headers package not installed already?  If not, why
> are you
> >> > trying to compile kernel modules for this version?
> >> >
> >> > > > Separately, whilst this NBS cleanup is in place, you can use the
> tool
> >> > > > `pull-lp-debs` from ubuntu-dev-tools which should allow you to
> >> > > > securely fetch any of the packages you require out of Launchpad
> >> > > > Librarian archival service. (note that pull-lp-debs is part of
> >> > > > collection of tools pull-ppa-ddebs pull-ppa-debs pull-ppa-source -
> >> > > > which are all wrappers around the swiss army knife type of tool
> >> > > > pull-pkg which can pull anything and everything out of Launchpad,
> >> > > > PPAs, Debian)
> >> > >
> >> > >
> >> > > > > Best,
> >> > > > >
> >> > > > > On Wed, May 10, 2023 at 12:51 AM Stefan Bader
> >> > > > > <stefan.bader at canonical.com> wrote:
> >> > > > > >
> >> > > > > > On 09.05.23 22:28, Tai-Lin Chu wrote:
> >> > > > > > > hi,
> >> > > > > > > I received alerts about linux-headers-5.15.0-1027-gke being
> removed last night.
> >> > > > > > > What might be the reason for that? thanks!
> >> > > > > > >
> >> > > > > > > Get:3 http://security.ubuntu.com/ubuntu
> jammy-security/restricted
> >> > > > > > > amd64 Packages [1077 kB]
> >> > > > > > > Get:4 http://archive.ubuntu.com/ubuntu jammy-updates
> InRelease [119 kB]
> >> > > > > > > Get:5 http://archive.ubuntu.com/ubuntu jammy-backports
> InRelease [108 kB]
> >> > > > > > > Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64
> Packages [1792 kB]
> >> > > > > > > Get:7 http://security.ubuntu.com/ubuntu
> jammy-security/main amd64
> >> > > > > > > Packages [585 kB]
> >> > > > > > > Get:8 http://security.ubuntu.com/ubuntu
> jammy-security/universe amd64
> >> > > > > > > Packages [898 kB]
> >> > > > > > > Get:9 http://security.ubuntu.com/ubuntu
> jammy-security/multiverse
> >> > > > > > > amd64 Packages [41.2 kB]
> >> > > > > > > Get:10 http://archive.ubuntu.com/ubuntu jammy/universe
> amd64 Packages [17.5 MB]
> >> > > > > > > Get:11 http://archive.ubuntu.com/ubuntu jammy/multiverse
> amd64 Packages [266 kB]
> >> > > > > > > Get:12 http://archive.ubuntu.com/ubuntu jammy/restricted
> amd64 Packages [164 kB]
> >> > > > > > > Get:13 http://archive.ubuntu.com/ubuntu
> jammy-updates/multiverse amd64
> >> > > > > > > Packages [46.6 kB]
> >> > > > > > > Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main
> amd64
> >> > > > > > > Packages [992 kB]
> >> > > > > > > Get:15 http://archive.ubuntu.com/ubuntu
> jammy-updates/restricted amd64
> >> > > > > > > Packages [1137 kB]
> >> > > > > > > Get:16 http://archive.ubuntu.com/ubuntu
> jammy-updates/universe amd64
> >> > > > > > > Packages [1143 kB]
> >> > > > > > > Get:17 http://archive.ubuntu.com/ubuntu
> jammy-backports/universe amd64
> >> > > > > > > Packages [25.6 kB]
> >> > > > > > > Get:18 http://archive.ubuntu.com/ubuntu
> jammy-backports/main amd64
> >> > > > > > > Packages [49.4 kB]
> >> > > > > > > Fetched 26.3 MB in 2s (10.6 MB/s)
> >> > > > > > > Reading package lists...
> >> > > > > > > + apt-get install -y linux-headers-5.15.0-1027-gke
> >> > > > > > > linux-modules-extra-5.15.0-1027-gke
> >> > > > > > > Reading package lists...
> >> > > > > > > Building dependency tree...
> >> > > > > > > Reading state information...
> >> > > > > > > E: Unable to locate package linux-headers-5.15.0-1027-gke
> >> > > > > > > E: Couldn't find any package by glob
> 'linux-headers-5.15.0-1027-gke'
> >> > > > > > > E: Couldn't find any package by regex
> 'linux-headers-5.15.0-1027-gke'
> >> > > > > > >
> >> > > > > >
> >> > > > > > That is just the normal way things change with updates. The
> archive will
> >> > > > > > not find older version which have been replaced by newer
> ones. You
> >> > > > > > should never try to install specific versions for that
> reason. Try "apt
> >> > > > > > install linux-headers-gke".
> >> > > > > > --
> >> > > > > > - Stefan
> >> > > > > >
> >> > > > >
> >> > > > > --
> >> > > > > kernel-team mailing list
> >> > > > > kernel-team at lists.ubuntu.com
> >> > > > > https://lists.ubuntu.com/mailman/listinfo/kernel-team
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > okurrr,
> >> > > >
> >> > > > Dimitri
> >> > > >
> >> > >
> >> > > --
> >> > > Steve Langasek                   Give me a lever long enough and a
> Free OS
> >> > > Debian Developer                   to set it on, and I can move the
> world.
> >> > > Ubuntu Developer
> https://www.debian.org/
> >> > > slangasek at ubuntu.com
> vorlon at debian.org
> >> >
> >> > --
> >> > Steve Langasek                   Give me a lever long enough and a
> Free OS
> >> > Debian Developer                   to set it on, and I can move the
> world.
> >> > Ubuntu Developer
> https://www.debian.org/
> >> > slangasek at ubuntu.com
> vorlon at debian.org
> >>
> >
> >
> > --
> > Phil Roche
> > Staff Software Engineer
> > Canonical Public Cloud
>

-- 
Phil Roche
Staff Software Engineer
Canonical Public Cloud
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20230512/a1af0829/attachment-0001.html>