linux-headers-5.15.0-1027-gke is removed from jammy last night]

Wed May 17 19:15:00 UTC 2023

@Tai-Lin Chu <tailinchu at gmail.com>

Kernel headers, modules and modules-extras for GKE kernel versions
5.15.0.1027, 5.15.0.1028 and 5.15.0.1030 have now restored to the archive.

On Fri, 12 May 2023 at 17:07, Tai-Lin Chu <tailinchu at gmail.com> wrote:

> Great, thanks for the update.
>
> On Fri, May 12, 2023 at 8:13 AM Phil Roche <phil.roche at canonical.com>
> wrote:
> >
> > @tailinchu at gmail.com  steve.langasek at canonical.com has now confirmed
> that the exception for gke and gkeop kernels has been extended to include
> the modules-extras packages.
> >
> > I will urge GKE now to promote the latest images with the latest kernels
> so this will no longer be an issues for you or other GKE users.
> >
> > Phil
> >
> > On Fri, 12 May 2023 at 11:30, Phil Roche <phil.roche at canonical.com>
> wrote:
> >>
> >> Apologies I had missed the comment regarding modules-extra as this
> wasn't mentioned in the internal case.
> >>
> >> I am not willing to share the internal case URL on this public list but
> it was filed by porterdavid at google.com and is issue ID 281872648.
> >>
> >> Yesterday myself and steve.langasek at canonical.com discussed no longer
> pruning the gke and gkeop kernel headers but this didn't extend to
> `modules-extra`.  I will get clarification on this today.
> >>
> >> Phil
> >>
> >> On Thu, 11 May 2023 at 18:33, Tai-Lin Chu <tailinchu at gmail.com> wrote:
> >>>
> >>> The kernel header is installed but linux-modules-extra is not.
> >>> $ dpkg -L linux-modules-extra-$(uname -r)
> >>> dpkg-query: package 'linux-modules-extra-5.15.0-1027' is not installed
> >>> Use dpkg --contents (= dpkg-deb --contents) to list archive files
> contents.
> >>>
> >>> > Yes, I see there was an internal case raised now which I will
> respond to.
> >>>
> >>> could you send me the link to that case?
> >>>
> >>> Thanks!
> >>>
> >>> On Thu, May 11, 2023 at 2:17 AM Phil Roche <phil.roche at canonical.com>
> wrote:
> >>> >
> >>> > @tailinchu at gmail.com Hi,
> >>> >
> >>> > I work on the Canonical Public Cloud team, and I work with the GKE
> team to build and publish the Ubuntu GKE node images.
> >>> >
> >>> >>
> >>> >> ---------- Forwarded message ----------
> >>> >> From: Tai-Lin Chu <tailinchu at gmail.com>
> >>> >> To: Steve Langasek <steve.langasek at canonical.com>, Dimitri John
> Ledkov <dimitri.ledkov at canonical.com>, Tai-Lin Chu <tailinchu at gmail.com>,
> Stefan Bader <stefan.bader at canonical.com>, kernel-team at lists.ubuntu.com
> >>> >> Cc:
> >>> >> Bcc:
> >>> >> Date: Wed, 10 May 2023 12:16:52 -0700
> >>> >> Subject: Re: linux-headers-5.15.0-1027-gke is removed from jammy
> last night
> >>> >> Thanks for replying.
> >>> >> Let me clarify the use case: we use gke ubuntu containerd node
> image.
> >>> >> Because we cannot know which kernel version the node is using, on vm
> >>> >> creation completed, we will compile and inject a kernel module so
> that
> >>> >> pods can use that kernel module:
> >>> >>
> >>> >> apt-get install -y "linux-headers-$(uname -r)"
> "linux-modules-extra-$(uname -r)"
> >>> >>
> >>> >> > Is 5.15.0-1027-gke the version of a running kernel on your GKE
> instance?
> >>> >> yes.
> >>> >>
> >>> >> >  If so, why is the headers package not installed already?
> >>> >> I don't think the kernel header package exists on the node image
> >>> >>
> https://cloud.google.com/kubernetes-engine/docs/concepts/node-images#ubuntu-variants
> .
> >>> >
> >>> >
> >>> > The headers are pre-installed in all the Ubuntu node images
> >>> >
> >>> > ```
> >>> > ~$ apt list --installed | grep headers
> >>> > linux-gke-headers-5.15.0-1028/now 5.15.0-1028.33 amd64
> [installed,local]
> >>> > linux-headers-5.15.0-1028-gke/now 5.15.0-1028.33 amd64
> [installed,local]
> >>> > linux-headers-gke-5.15/now 5.15.0.1028.27 amd64 [installed,local]
> >>> > ```
> >>> >
> >>> >>
> >>> >> From that mailing list post, that timeline was not publicly
> announced
> >>> >> on gke updates, so we were completely unaware of it, and caused
> >>> >> incidents. Our current workaround is using 20.04 with the older gke
> >>> >> version because the new auto cleanup process is for jammy and later,
> >>> >> but I hope that older kernel can be kept around.
> >>> >
> >>> >
> >>> > Aside from any discussions about the pruning of the headers from the
> archive, another workaround or  - permanent solution - is to bind mount
> what you need from the host to the container. The headers are present on
> the node so this should work.
> >>> >
> >>> >>
> >>> >> Does canonical collaborate gcp to create node image?
> >>> >
> >>> >
> >>> > Yes, we do.
> >>> >
> >>> >>
> >>> >> If so, I can also
> >>> >> forward this to our gcp dedicated account team.
> >>> >
> >>> >
> >>> > Yes, I see there was an internal case raised now which I will
> respond to.
> >>> >
> >>> > Phil
> >>> >
> >>> >>
> >>> >>
> >>> >> Thanks!
> >>> >>
> >>> >> On Wed, May 10, 2023 at 8:29 AM Steve Langasek
> >>> >> <steve.langasek at canonical.com> wrote:
> >>> >> >
> >>> >> > On Wed, May 10, 2023 at 08:12:50AM -0700, Steve Langasek wrote:
> >>> >> > > When Andy and I looked at this, the analysis had showed that
> all our images
> >>> >> > > except for minimal images were being built with linux-$flavor
> installed,
> >>> >> > > rather than linux-image-$flavor, so that the headers would
> already be
> >>> >> > > present and removal of the old ABIs from the archive would have
> no impact on
> >>> >> > > users of these images.
> >>> >> > >
> >>> >> > > The GKE images are being built on a minimal base, which I did
> not recall.
> >>> >> > >
> >>> >> > > It is not otherwise an issue for cloud images per above, with
> the exception
> >>> >> > > of the cloud-minimal images.
> >>> >> > >
> >>> >> > > Since the linux-gke flavor is used only for minimal GKE images,
> we could
> >>> >> > > reasonably exclude these from the NBS cleaning going forward.
> >>> >> >
> >>> >> > > I've reached out to our Public Cloud team to see what their
> preference is.
> >>> >> >
> >>> >> > I've clarified with the Public Cloud team that, although the GKE
> images use
> >>> >> > a minimal base, the images are built using linux-gke-$version, NOT
> >>> >> > linux-image-gke-$version.
> >>> >> >
> >>> >> > So it's unclear that what Tai-Lin is doing is a use case that the
> Public
> >>> >> > Cloud team is concerned with supporting.
> >>> >> >
> >>> >> > Is 5.15.0-1027-gke the version of a running kernel on your GKE
> instance?  If
> >>> >> > so, why is the headers package not installed already?  If not,
> why are you
> >>> >> > trying to compile kernel modules for this version?
> >>> >> >
> >>> >> > > > Separately, whilst this NBS cleanup is in place, you can use
> the tool
> >>> >> > > > `pull-lp-debs` from ubuntu-dev-tools which should allow you to
> >>> >> > > > securely fetch any of the packages you require out of
> Launchpad
> >>> >> > > > Librarian archival service. (note that pull-lp-debs is part of
> >>> >> > > > collection of tools pull-ppa-ddebs pull-ppa-debs
> pull-ppa-source -
> >>> >> > > > which are all wrappers around the swiss army knife type of
> tool
> >>> >> > > > pull-pkg which can pull anything and everything out of
> Launchpad,
> >>> >> > > > PPAs, Debian)
> >>> >> > >
> >>> >> > >
> >>> >> > > > > Best,
> >>> >> > > > >
> >>> >> > > > > On Wed, May 10, 2023 at 12:51 AM Stefan Bader
> >>> >> > > > > <stefan.bader at canonical.com> wrote:
> >>> >> > > > > >
> >>> >> > > > > > On 09.05.23 22:28, Tai-Lin Chu wrote:
> >>> >> > > > > > > hi,
> >>> >> > > > > > > I received alerts about linux-headers-5.15.0-1027-gke
> being removed last night.
> >>> >> > > > > > > What might be the reason for that? thanks!
> >>> >> > > > > > >
> >>> >> > > > > > > Get:3 http://security.ubuntu.com/ubuntu
> jammy-security/restricted
> >>> >> > > > > > > amd64 Packages [1077 kB]
> >>> >> > > > > > > Get:4 http://archive.ubuntu.com/ubuntu jammy-updates
> InRelease [119 kB]
> >>> >> > > > > > > Get:5 http://archive.ubuntu.com/ubuntu jammy-backports
> InRelease [108 kB]
> >>> >> > > > > > > Get:6 http://archive.ubuntu.com/ubuntu jammy/main
> amd64 Packages [1792 kB]
> >>> >> > > > > > > Get:7 http://security.ubuntu.com/ubuntu
> jammy-security/main amd64
> >>> >> > > > > > > Packages [585 kB]
> >>> >> > > > > > > Get:8 http://security.ubuntu.com/ubuntu
> jammy-security/universe amd64
> >>> >> > > > > > > Packages [898 kB]
> >>> >> > > > > > > Get:9 http://security.ubuntu.com/ubuntu
> jammy-security/multiverse
> >>> >> > > > > > > amd64 Packages [41.2 kB]
> >>> >> > > > > > > Get:10 http://archive.ubuntu.com/ubuntu jammy/universe
> amd64 Packages [17.5 MB]
> >>> >> > > > > > > Get:11 http://archive.ubuntu.com/ubuntu
> jammy/multiverse amd64 Packages [266 kB]
> >>> >> > > > > > > Get:12 http://archive.ubuntu.com/ubuntu
> jammy/restricted amd64 Packages [164 kB]
> >>> >> > > > > > > Get:13 http://archive.ubuntu.com/ubuntu
> jammy-updates/multiverse amd64
> >>> >> > > > > > > Packages [46.6 kB]
> >>> >> > > > > > > Get:14 http://archive.ubuntu.com/ubuntu
> jammy-updates/main amd64
> >>> >> > > > > > > Packages [992 kB]
> >>> >> > > > > > > Get:15 http://archive.ubuntu.com/ubuntu
> jammy-updates/restricted amd64
> >>> >> > > > > > > Packages [1137 kB]
> >>> >> > > > > > > Get:16 http://archive.ubuntu.com/ubuntu
> jammy-updates/universe amd64
> >>> >> > > > > > > Packages [1143 kB]
> >>> >> > > > > > > Get:17 http://archive.ubuntu.com/ubuntu
> jammy-backports/universe amd64
> >>> >> > > > > > > Packages [25.6 kB]
> >>> >> > > > > > > Get:18 http://archive.ubuntu.com/ubuntu
> jammy-backports/main amd64
> >>> >> > > > > > > Packages [49.4 kB]
> >>> >> > > > > > > Fetched 26.3 MB in 2s (10.6 MB/s)
> >>> >> > > > > > > Reading package lists...
> >>> >> > > > > > > + apt-get install -y linux-headers-5.15.0-1027-gke
> >>> >> > > > > > > linux-modules-extra-5.15.0-1027-gke
> >>> >> > > > > > > Reading package lists...
> >>> >> > > > > > > Building dependency tree...
> >>> >> > > > > > > Reading state information...
> >>> >> > > > > > > E: Unable to locate package
> linux-headers-5.15.0-1027-gke
> >>> >> > > > > > > E: Couldn't find any package by glob
> 'linux-headers-5.15.0-1027-gke'
> >>> >> > > > > > > E: Couldn't find any package by regex
> 'linux-headers-5.15.0-1027-gke'
> >>> >> > > > > > >
> >>> >> > > > > >
> >>> >> > > > > > That is just the normal way things change with updates.
> The archive will
> >>> >> > > > > > not find older version which have been replaced by newer
> ones. You
> >>> >> > > > > > should never try to install specific versions for that
> reason. Try "apt
> >>> >> > > > > > install linux-headers-gke".
> >>> >> > > > > > --
> >>> >> > > > > > - Stefan
> >>> >> > > > > >
> >>> >> > > > >
> >>> >> > > > > --
> >>> >> > > > > kernel-team mailing list
> >>> >> > > > > kernel-team at lists.ubuntu.com
> >>> >> > > > > https://lists.ubuntu.com/mailman/listinfo/kernel-team
> >>> >> > > >
> >>> >> > > >
> >>> >> > > >
> >>> >> > > > --
> >>> >> > > > okurrr,
> >>> >> > > >
> >>> >> > > > Dimitri
> >>> >> > > >
> >>> >> > >
> >>> >> > > --
> >>> >> > > Steve Langasek                   Give me a lever long enough
> and a Free OS
> >>> >> > > Debian Developer                   to set it on, and I can move
> the world.
> >>> >> > > Ubuntu Developer
> https://www.debian.org/
> >>> >> > > slangasek at ubuntu.com
> vorlon at debian.org
> >>> >> >
> >>> >> > --
> >>> >> > Steve Langasek                   Give me a lever long enough and
> a Free OS
> >>> >> > Debian Developer                   to set it on, and I can move
> the world.
> >>> >> > Ubuntu Developer
> https://www.debian.org/
> >>> >> > slangasek at ubuntu.com
> vorlon at debian.org
> >>> >>
> >>> >
> >>> >
> >>> > --
> >>> > Phil Roche
> >>> > Staff Software Engineer
> >>> > Canonical Public Cloud
> >>
> >>
> >>
> >> --
> >> Phil Roche
> >> Staff Software Engineer
> >> Canonical Public Cloud
> >
> >
> >
> > --
> > Phil Roche
> > Staff Software Engineer
> > Canonical Public Cloud
>

-- 
Phil Roche
Staff Software Engineer
Canonical Public Cloud
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20230517/8c40702f/attachment-0001.html>