linux-headers-5.15.0-1027-gke is removed from jammy last night]
Tai-Lin Chu
tailinchu at gmail.com
Wed May 17 19:22:26 UTC 2023
thank you!
On Wed, May 17, 2023 at 12:15 PM Phil Roche <phil.roche at canonical.com> wrote:
>
> @Tai-Lin Chu
>
> Kernel headers, modules and modules-extras for GKE kernel versions 5.15.0.1027, 5.15.0.1028 and 5.15.0.1030 have now restored to the archive.
>
> On Fri, 12 May 2023 at 17:07, Tai-Lin Chu <tailinchu at gmail.com> wrote:
>>
>> Great, thanks for the update.
>>
>> On Fri, May 12, 2023 at 8:13 AM Phil Roche <phil.roche at canonical.com> wrote:
>> >
>> > @tailinchu at gmail.com steve.langasek at canonical.com has now confirmed that the exception for gke and gkeop kernels has been extended to include the modules-extras packages.
>> >
>> > I will urge GKE now to promote the latest images with the latest kernels so this will no longer be an issues for you or other GKE users.
>> >
>> > Phil
>> >
>> > On Fri, 12 May 2023 at 11:30, Phil Roche <phil.roche at canonical.com> wrote:
>> >>
>> >> Apologies I had missed the comment regarding modules-extra as this wasn't mentioned in the internal case.
>> >>
>> >> I am not willing to share the internal case URL on this public list but it was filed by porterdavid at google.com and is issue ID 281872648.
>> >>
>> >> Yesterday myself and steve.langasek at canonical.com discussed no longer pruning the gke and gkeop kernel headers but this didn't extend to `modules-extra`. I will get clarification on this today.
>> >>
>> >> Phil
>> >>
>> >> On Thu, 11 May 2023 at 18:33, Tai-Lin Chu <tailinchu at gmail.com> wrote:
>> >>>
>> >>> The kernel header is installed but linux-modules-extra is not.
>> >>> $ dpkg -L linux-modules-extra-$(uname -r)
>> >>> dpkg-query: package 'linux-modules-extra-5.15.0-1027' is not installed
>> >>> Use dpkg --contents (= dpkg-deb --contents) to list archive files contents.
>> >>>
>> >>> > Yes, I see there was an internal case raised now which I will respond to.
>> >>>
>> >>> could you send me the link to that case?
>> >>>
>> >>> Thanks!
>> >>>
>> >>> On Thu, May 11, 2023 at 2:17 AM Phil Roche <phil.roche at canonical.com> wrote:
>> >>> >
>> >>> > @tailinchu at gmail.com Hi,
>> >>> >
>> >>> > I work on the Canonical Public Cloud team, and I work with the GKE team to build and publish the Ubuntu GKE node images.
>> >>> >
>> >>> >>
>> >>> >> ---------- Forwarded message ----------
>> >>> >> From: Tai-Lin Chu <tailinchu at gmail.com>
>> >>> >> To: Steve Langasek <steve.langasek at canonical.com>, Dimitri John Ledkov <dimitri.ledkov at canonical.com>, Tai-Lin Chu <tailinchu at gmail.com>, Stefan Bader <stefan.bader at canonical.com>, kernel-team at lists.ubuntu.com
>> >>> >> Cc:
>> >>> >> Bcc:
>> >>> >> Date: Wed, 10 May 2023 12:16:52 -0700
>> >>> >> Subject: Re: linux-headers-5.15.0-1027-gke is removed from jammy last night
>> >>> >> Thanks for replying.
>> >>> >> Let me clarify the use case: we use gke ubuntu containerd node image.
>> >>> >> Because we cannot know which kernel version the node is using, on vm
>> >>> >> creation completed, we will compile and inject a kernel module so that
>> >>> >> pods can use that kernel module:
>> >>> >>
>> >>> >> apt-get install -y "linux-headers-$(uname -r)" "linux-modules-extra-$(uname -r)"
>> >>> >>
>> >>> >> > Is 5.15.0-1027-gke the version of a running kernel on your GKE instance?
>> >>> >> yes.
>> >>> >>
>> >>> >> > If so, why is the headers package not installed already?
>> >>> >> I don't think the kernel header package exists on the node image
>> >>> >> https://cloud.google.com/kubernetes-engine/docs/concepts/node-images#ubuntu-variants.
>> >>> >
>> >>> >
>> >>> > The headers are pre-installed in all the Ubuntu node images
>> >>> >
>> >>> > ```
>> >>> > ~$ apt list --installed | grep headers
>> >>> > linux-gke-headers-5.15.0-1028/now 5.15.0-1028.33 amd64 [installed,local]
>> >>> > linux-headers-5.15.0-1028-gke/now 5.15.0-1028.33 amd64 [installed,local]
>> >>> > linux-headers-gke-5.15/now 5.15.0.1028.27 amd64 [installed,local]
>> >>> > ```
>> >>> >
>> >>> >>
>> >>> >> From that mailing list post, that timeline was not publicly announced
>> >>> >> on gke updates, so we were completely unaware of it, and caused
>> >>> >> incidents. Our current workaround is using 20.04 with the older gke
>> >>> >> version because the new auto cleanup process is for jammy and later,
>> >>> >> but I hope that older kernel can be kept around.
>> >>> >
>> >>> >
>> >>> > Aside from any discussions about the pruning of the headers from the archive, another workaround or - permanent solution - is to bind mount what you need from the host to the container. The headers are present on the node so this should work.
>> >>> >
>> >>> >>
>> >>> >> Does canonical collaborate gcp to create node image?
>> >>> >
>> >>> >
>> >>> > Yes, we do.
>> >>> >
>> >>> >>
>> >>> >> If so, I can also
>> >>> >> forward this to our gcp dedicated account team.
>> >>> >
>> >>> >
>> >>> > Yes, I see there was an internal case raised now which I will respond to.
>> >>> >
>> >>> > Phil
>> >>> >
>> >>> >>
>> >>> >>
>> >>> >> Thanks!
>> >>> >>
>> >>> >> On Wed, May 10, 2023 at 8:29 AM Steve Langasek
>> >>> >> <steve.langasek at canonical.com> wrote:
>> >>> >> >
>> >>> >> > On Wed, May 10, 2023 at 08:12:50AM -0700, Steve Langasek wrote:
>> >>> >> > > When Andy and I looked at this, the analysis had showed that all our images
>> >>> >> > > except for minimal images were being built with linux-$flavor installed,
>> >>> >> > > rather than linux-image-$flavor, so that the headers would already be
>> >>> >> > > present and removal of the old ABIs from the archive would have no impact on
>> >>> >> > > users of these images.
>> >>> >> > >
>> >>> >> > > The GKE images are being built on a minimal base, which I did not recall.
>> >>> >> > >
>> >>> >> > > It is not otherwise an issue for cloud images per above, with the exception
>> >>> >> > > of the cloud-minimal images.
>> >>> >> > >
>> >>> >> > > Since the linux-gke flavor is used only for minimal GKE images, we could
>> >>> >> > > reasonably exclude these from the NBS cleaning going forward.
>> >>> >> >
>> >>> >> > > I've reached out to our Public Cloud team to see what their preference is.
>> >>> >> >
>> >>> >> > I've clarified with the Public Cloud team that, although the GKE images use
>> >>> >> > a minimal base, the images are built using linux-gke-$version, NOT
>> >>> >> > linux-image-gke-$version.
>> >>> >> >
>> >>> >> > So it's unclear that what Tai-Lin is doing is a use case that the Public
>> >>> >> > Cloud team is concerned with supporting.
>> >>> >> >
>> >>> >> > Is 5.15.0-1027-gke the version of a running kernel on your GKE instance? If
>> >>> >> > so, why is the headers package not installed already? If not, why are you
>> >>> >> > trying to compile kernel modules for this version?
>> >>> >> >
>> >>> >> > > > Separately, whilst this NBS cleanup is in place, you can use the tool
>> >>> >> > > > `pull-lp-debs` from ubuntu-dev-tools which should allow you to
>> >>> >> > > > securely fetch any of the packages you require out of Launchpad
>> >>> >> > > > Librarian archival service. (note that pull-lp-debs is part of
>> >>> >> > > > collection of tools pull-ppa-ddebs pull-ppa-debs pull-ppa-source -
>> >>> >> > > > which are all wrappers around the swiss army knife type of tool
>> >>> >> > > > pull-pkg which can pull anything and everything out of Launchpad,
>> >>> >> > > > PPAs, Debian)
>> >>> >> > >
>> >>> >> > >
>> >>> >> > > > > Best,
>> >>> >> > > > >
>> >>> >> > > > > On Wed, May 10, 2023 at 12:51 AM Stefan Bader
>> >>> >> > > > > <stefan.bader at canonical.com> wrote:
>> >>> >> > > > > >
>> >>> >> > > > > > On 09.05.23 22:28, Tai-Lin Chu wrote:
>> >>> >> > > > > > > hi,
>> >>> >> > > > > > > I received alerts about linux-headers-5.15.0-1027-gke being removed last night.
>> >>> >> > > > > > > What might be the reason for that? thanks!
>> >>> >> > > > > > >
>> >>> >> > > > > > > Get:3 http://security.ubuntu.com/ubuntu jammy-security/restricted
>> >>> >> > > > > > > amd64 Packages [1077 kB]
>> >>> >> > > > > > > Get:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
>> >>> >> > > > > > > Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB]
>> >>> >> > > > > > > Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1792 kB]
>> >>> >> > > > > > > Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64
>> >>> >> > > > > > > Packages [585 kB]
>> >>> >> > > > > > > Get:8 http://security.ubuntu.com/ubuntu jammy-security/universe amd64
>> >>> >> > > > > > > Packages [898 kB]
>> >>> >> > > > > > > Get:9 http://security.ubuntu.com/ubuntu jammy-security/multiverse
>> >>> >> > > > > > > amd64 Packages [41.2 kB]
>> >>> >> > > > > > > Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]
>> >>> >> > > > > > > Get:11 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [266 kB]
>> >>> >> > > > > > > Get:12 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [164 kB]
>> >>> >> > > > > > > Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64
>> >>> >> > > > > > > Packages [46.6 kB]
>> >>> >> > > > > > > Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64
>> >>> >> > > > > > > Packages [992 kB]
>> >>> >> > > > > > > Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64
>> >>> >> > > > > > > Packages [1137 kB]
>> >>> >> > > > > > > Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64
>> >>> >> > > > > > > Packages [1143 kB]
>> >>> >> > > > > > > Get:17 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64
>> >>> >> > > > > > > Packages [25.6 kB]
>> >>> >> > > > > > > Get:18 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64
>> >>> >> > > > > > > Packages [49.4 kB]
>> >>> >> > > > > > > Fetched 26.3 MB in 2s (10.6 MB/s)
>> >>> >> > > > > > > Reading package lists...
>> >>> >> > > > > > > + apt-get install -y linux-headers-5.15.0-1027-gke
>> >>> >> > > > > > > linux-modules-extra-5.15.0-1027-gke
>> >>> >> > > > > > > Reading package lists...
>> >>> >> > > > > > > Building dependency tree...
>> >>> >> > > > > > > Reading state information...
>> >>> >> > > > > > > E: Unable to locate package linux-headers-5.15.0-1027-gke
>> >>> >> > > > > > > E: Couldn't find any package by glob 'linux-headers-5.15.0-1027-gke'
>> >>> >> > > > > > > E: Couldn't find any package by regex 'linux-headers-5.15.0-1027-gke'
>> >>> >> > > > > > >
>> >>> >> > > > > >
>> >>> >> > > > > > That is just the normal way things change with updates. The archive will
>> >>> >> > > > > > not find older version which have been replaced by newer ones. You
>> >>> >> > > > > > should never try to install specific versions for that reason. Try "apt
>> >>> >> > > > > > install linux-headers-gke".
>> >>> >> > > > > > --
>> >>> >> > > > > > - Stefan
>> >>> >> > > > > >
>> >>> >> > > > >
>> >>> >> > > > > --
>> >>> >> > > > > kernel-team mailing list
>> >>> >> > > > > kernel-team at lists.ubuntu.com
>> >>> >> > > > > https://lists.ubuntu.com/mailman/listinfo/kernel-team
>> >>> >> > > >
>> >>> >> > > >
>> >>> >> > > >
>> >>> >> > > > --
>> >>> >> > > > okurrr,
>> >>> >> > > >
>> >>> >> > > > Dimitri
>> >>> >> > > >
>> >>> >> > >
>> >>> >> > > --
>> >>> >> > > Steve Langasek Give me a lever long enough and a Free OS
>> >>> >> > > Debian Developer to set it on, and I can move the world.
>> >>> >> > > Ubuntu Developer https://www.debian.org/
>> >>> >> > > slangasek at ubuntu.com vorlon at debian.org
>> >>> >> >
>> >>> >> > --
>> >>> >> > Steve Langasek Give me a lever long enough and a Free OS
>> >>> >> > Debian Developer to set it on, and I can move the world.
>> >>> >> > Ubuntu Developer https://www.debian.org/
>> >>> >> > slangasek at ubuntu.com vorlon at debian.org
>> >>> >>
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Phil Roche
>> >>> > Staff Software Engineer
>> >>> > Canonical Public Cloud
>> >>
>> >>
>> >>
>> >> --
>> >> Phil Roche
>> >> Staff Software Engineer
>> >> Canonical Public Cloud
>> >
>> >
>> >
>> > --
>> > Phil Roche
>> > Staff Software Engineer
>> > Canonical Public Cloud
>
>
>
> --
> Phil Roche
> Staff Software Engineer
> Canonical Public Cloud
More information about the kernel-team
mailing list