linux-headers-5.15.0-1027-gke is removed from jammy last night]
Tai-Lin Chu
tailinchu at gmail.com
Thu May 11 17:32:55 UTC 2023
The kernel header is installed but linux-modules-extra is not.
$ dpkg -L linux-modules-extra-$(uname -r)
dpkg-query: package 'linux-modules-extra-5.15.0-1027' is not installed
Use dpkg --contents (= dpkg-deb --contents) to list archive files contents.
> Yes, I see there was an internal case raised now which I will respond to.
could you send me the link to that case?
Thanks!
On Thu, May 11, 2023 at 2:17 AM Phil Roche <phil.roche at canonical.com> wrote:
>
> @tailinchu at gmail.com Hi,
>
> I work on the Canonical Public Cloud team, and I work with the GKE team to build and publish the Ubuntu GKE node images.
>
>>
>> ---------- Forwarded message ----------
>> From: Tai-Lin Chu <tailinchu at gmail.com>
>> To: Steve Langasek <steve.langasek at canonical.com>, Dimitri John Ledkov <dimitri.ledkov at canonical.com>, Tai-Lin Chu <tailinchu at gmail.com>, Stefan Bader <stefan.bader at canonical.com>, kernel-team at lists.ubuntu.com
>> Cc:
>> Bcc:
>> Date: Wed, 10 May 2023 12:16:52 -0700
>> Subject: Re: linux-headers-5.15.0-1027-gke is removed from jammy last night
>> Thanks for replying.
>> Let me clarify the use case: we use gke ubuntu containerd node image.
>> Because we cannot know which kernel version the node is using, on vm
>> creation completed, we will compile and inject a kernel module so that
>> pods can use that kernel module:
>>
>> apt-get install -y "linux-headers-$(uname -r)" "linux-modules-extra-$(uname -r)"
>>
>> > Is 5.15.0-1027-gke the version of a running kernel on your GKE instance?
>> yes.
>>
>> > If so, why is the headers package not installed already?
>> I don't think the kernel header package exists on the node image
>> https://cloud.google.com/kubernetes-engine/docs/concepts/node-images#ubuntu-variants.
>
>
> The headers are pre-installed in all the Ubuntu node images
>
> ```
> ~$ apt list --installed | grep headers
> linux-gke-headers-5.15.0-1028/now 5.15.0-1028.33 amd64 [installed,local]
> linux-headers-5.15.0-1028-gke/now 5.15.0-1028.33 amd64 [installed,local]
> linux-headers-gke-5.15/now 5.15.0.1028.27 amd64 [installed,local]
> ```
>
>>
>> From that mailing list post, that timeline was not publicly announced
>> on gke updates, so we were completely unaware of it, and caused
>> incidents. Our current workaround is using 20.04 with the older gke
>> version because the new auto cleanup process is for jammy and later,
>> but I hope that older kernel can be kept around.
>
>
> Aside from any discussions about the pruning of the headers from the archive, another workaround or - permanent solution - is to bind mount what you need from the host to the container. The headers are present on the node so this should work.
>
>>
>> Does canonical collaborate gcp to create node image?
>
>
> Yes, we do.
>
>>
>> If so, I can also
>> forward this to our gcp dedicated account team.
>
>
> Yes, I see there was an internal case raised now which I will respond to.
>
> Phil
>
>>
>>
>> Thanks!
>>
>> On Wed, May 10, 2023 at 8:29 AM Steve Langasek
>> <steve.langasek at canonical.com> wrote:
>> >
>> > On Wed, May 10, 2023 at 08:12:50AM -0700, Steve Langasek wrote:
>> > > When Andy and I looked at this, the analysis had showed that all our images
>> > > except for minimal images were being built with linux-$flavor installed,
>> > > rather than linux-image-$flavor, so that the headers would already be
>> > > present and removal of the old ABIs from the archive would have no impact on
>> > > users of these images.
>> > >
>> > > The GKE images are being built on a minimal base, which I did not recall.
>> > >
>> > > It is not otherwise an issue for cloud images per above, with the exception
>> > > of the cloud-minimal images.
>> > >
>> > > Since the linux-gke flavor is used only for minimal GKE images, we could
>> > > reasonably exclude these from the NBS cleaning going forward.
>> >
>> > > I've reached out to our Public Cloud team to see what their preference is.
>> >
>> > I've clarified with the Public Cloud team that, although the GKE images use
>> > a minimal base, the images are built using linux-gke-$version, NOT
>> > linux-image-gke-$version.
>> >
>> > So it's unclear that what Tai-Lin is doing is a use case that the Public
>> > Cloud team is concerned with supporting.
>> >
>> > Is 5.15.0-1027-gke the version of a running kernel on your GKE instance? If
>> > so, why is the headers package not installed already? If not, why are you
>> > trying to compile kernel modules for this version?
>> >
>> > > > Separately, whilst this NBS cleanup is in place, you can use the tool
>> > > > `pull-lp-debs` from ubuntu-dev-tools which should allow you to
>> > > > securely fetch any of the packages you require out of Launchpad
>> > > > Librarian archival service. (note that pull-lp-debs is part of
>> > > > collection of tools pull-ppa-ddebs pull-ppa-debs pull-ppa-source -
>> > > > which are all wrappers around the swiss army knife type of tool
>> > > > pull-pkg which can pull anything and everything out of Launchpad,
>> > > > PPAs, Debian)
>> > >
>> > >
>> > > > > Best,
>> > > > >
>> > > > > On Wed, May 10, 2023 at 12:51 AM Stefan Bader
>> > > > > <stefan.bader at canonical.com> wrote:
>> > > > > >
>> > > > > > On 09.05.23 22:28, Tai-Lin Chu wrote:
>> > > > > > > hi,
>> > > > > > > I received alerts about linux-headers-5.15.0-1027-gke being removed last night.
>> > > > > > > What might be the reason for that? thanks!
>> > > > > > >
>> > > > > > > Get:3 http://security.ubuntu.com/ubuntu jammy-security/restricted
>> > > > > > > amd64 Packages [1077 kB]
>> > > > > > > Get:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
>> > > > > > > Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB]
>> > > > > > > Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1792 kB]
>> > > > > > > Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64
>> > > > > > > Packages [585 kB]
>> > > > > > > Get:8 http://security.ubuntu.com/ubuntu jammy-security/universe amd64
>> > > > > > > Packages [898 kB]
>> > > > > > > Get:9 http://security.ubuntu.com/ubuntu jammy-security/multiverse
>> > > > > > > amd64 Packages [41.2 kB]
>> > > > > > > Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]
>> > > > > > > Get:11 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [266 kB]
>> > > > > > > Get:12 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [164 kB]
>> > > > > > > Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64
>> > > > > > > Packages [46.6 kB]
>> > > > > > > Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64
>> > > > > > > Packages [992 kB]
>> > > > > > > Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64
>> > > > > > > Packages [1137 kB]
>> > > > > > > Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64
>> > > > > > > Packages [1143 kB]
>> > > > > > > Get:17 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64
>> > > > > > > Packages [25.6 kB]
>> > > > > > > Get:18 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64
>> > > > > > > Packages [49.4 kB]
>> > > > > > > Fetched 26.3 MB in 2s (10.6 MB/s)
>> > > > > > > Reading package lists...
>> > > > > > > + apt-get install -y linux-headers-5.15.0-1027-gke
>> > > > > > > linux-modules-extra-5.15.0-1027-gke
>> > > > > > > Reading package lists...
>> > > > > > > Building dependency tree...
>> > > > > > > Reading state information...
>> > > > > > > E: Unable to locate package linux-headers-5.15.0-1027-gke
>> > > > > > > E: Couldn't find any package by glob 'linux-headers-5.15.0-1027-gke'
>> > > > > > > E: Couldn't find any package by regex 'linux-headers-5.15.0-1027-gke'
>> > > > > > >
>> > > > > >
>> > > > > > That is just the normal way things change with updates. The archive will
>> > > > > > not find older version which have been replaced by newer ones. You
>> > > > > > should never try to install specific versions for that reason. Try "apt
>> > > > > > install linux-headers-gke".
>> > > > > > --
>> > > > > > - Stefan
>> > > > > >
>> > > > >
>> > > > > --
>> > > > > kernel-team mailing list
>> > > > > kernel-team at lists.ubuntu.com
>> > > > > https://lists.ubuntu.com/mailman/listinfo/kernel-team
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > okurrr,
>> > > >
>> > > > Dimitri
>> > > >
>> > >
>> > > --
>> > > Steve Langasek Give me a lever long enough and a Free OS
>> > > Debian Developer to set it on, and I can move the world.
>> > > Ubuntu Developer https://www.debian.org/
>> > > slangasek at ubuntu.com vorlon at debian.org
>> >
>> > --
>> > Steve Langasek Give me a lever long enough and a Free OS
>> > Debian Developer to set it on, and I can move the world.
>> > Ubuntu Developer https://www.debian.org/
>> > slangasek at ubuntu.com vorlon at debian.org
>>
>
>
> --
> Phil Roche
> Staff Software Engineer
> Canonical Public Cloud
More information about the kernel-team
mailing list