<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Apologies I had missed the comment regarding modules-extra as this wasn't mentioned in the internal case.</div><div> </div><div>I am not willing to share the internal case URL on this public list but it was filed by <a href="mailto:porterdavid@google.com">porterdavid@google.com</a> and is issue ID 281872648.</div><div> </div><div>Yesterday myself and <a href="mailto:steve.langasek@canonical.com">steve.langasek@canonical.com</a> discussed no longer pruning the gke and gkeop kernel headers but this didn't extend to `modules-extra`. I will get clarification on this today.</div><div> </div><div>Phil</div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 11 May 2023 at 18:33, Tai-Lin Chu <<a href="mailto:tailinchu@gmail.com">tailinchu@gmail.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The kernel header is installed but linux-modules-extra is not. $ dpkg -L linux-modules-extra-$(uname -r) dpkg-query: package 'linux-modules-extra-5.15.0-1027' is not installed Use dpkg --contents (= dpkg-deb --contents) to list archive files contents. > Yes, I see there was an internal case raised now which I will respond to. could you send me the link to that case? Thanks! On Thu, May 11, 2023 at 2:17 AM Phil Roche <<a href="mailto:phil.roche@canonical.com" target="_blank">phil.roche@canonical.com</a>> wrote: > > @<a href="mailto:tailinchu@gmail.com" target="_blank">tailinchu@gmail.com</a> Hi, > > I work on the Canonical Public Cloud team, and I work with the GKE team to build and publish the Ubuntu GKE node images. > >> >> ---------- Forwarded message ---------- >> From: Tai-Lin Chu <<a href="mailto:tailinchu@gmail.com" target="_blank">tailinchu@gmail.com</a>> >> To: Steve Langasek <<a href="mailto:steve.langasek@canonical.com" target="_blank">steve.langasek@canonical.com</a>>, Dimitri John Ledkov <<a href="mailto:dimitri.ledkov@canonical.com" target="_blank">dimitri.ledkov@canonical.com</a>>, Tai-Lin Chu <<a href="mailto:tailinchu@gmail.com" target="_blank">tailinchu@gmail.com</a>>, Stefan Bader <<a href="mailto:stefan.bader@canonical.com" target="_blank">stefan.bader@canonical.com</a>>, <a href="mailto:kernel-team@lists.ubuntu.com" target="_blank">kernel-team@lists.ubuntu.com</a> >> Cc: >> Bcc: >> Date: Wed, 10 May 2023 12:16:52 -0700 >> Subject: Re: linux-headers-5.15.0-1027-gke is removed from jammy last night >> Thanks for replying. >> Let me clarify the use case: we use gke ubuntu containerd node image. >> Because we cannot know which kernel version the node is using, on vm >> creation completed, we will compile and inject a kernel module so that >> pods can use that kernel module: >> >> apt-get install -y "linux-headers-$(uname -r)" "linux-modules-extra-$(uname -r)" >> >> > Is 5.15.0-1027-gke the version of a running kernel on your GKE instance? >> yes. >> >> > If so, why is the headers package not installed already? >> I don't think the kernel header package exists on the node image >> <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/node-images#ubuntu-variants" rel="noreferrer" target="_blank">https://cloud.google.com/kubernetes-engine/docs/concepts/node-images#ubuntu-variants</a>. > > > The headers are pre-installed in all the Ubuntu node images > > ``` > ~$ apt list --installed | grep headers > linux-gke-headers-5.15.0-1028/now 5.15.0-1028.33 amd64 [installed,local] > linux-headers-5.15.0-1028-gke/now 5.15.0-1028.33 amd64 [installed,local] > linux-headers-gke-5.15/now 5.15.0.1028.27 amd64 [installed,local] > ``` > >> >> From that mailing list post, that timeline was not publicly announced >> on gke updates, so we were completely unaware of it, and caused >> incidents. Our current workaround is using 20.04 with the older gke >> version because the new auto cleanup process is for jammy and later, >> but I hope that older kernel can be kept around. > > > Aside from any discussions about the pruning of the headers from the archive, another workaround or - permanent solution - is to bind mount what you need from the host to the container. The headers are present on the node so this should work. > >> >> Does canonical collaborate gcp to create node image? > > > Yes, we do. > >> >> If so, I can also >> forward this to our gcp dedicated account team. > > > Yes, I see there was an internal case raised now which I will respond to. > > Phil > >> >> >> Thanks! >> >> On Wed, May 10, 2023 at 8:29 AM Steve Langasek >> <<a href="mailto:steve.langasek@canonical.com" target="_blank">steve.langasek@canonical.com</a>> wrote: >> > >> > On Wed, May 10, 2023 at 08:12:50AM -0700, Steve Langasek wrote: >> > > When Andy and I looked at this, the analysis had showed that all our images >> > > except for minimal images were being built with linux-$flavor installed, >> > > rather than linux-image-$flavor, so that the headers would already be >> > > present and removal of the old ABIs from the archive would have no impact on >> > > users of these images. >> > > >> > > The GKE images are being built on a minimal base, which I did not recall. >> > > >> > > It is not otherwise an issue for cloud images per above, with the exception >> > > of the cloud-minimal images. >> > > >> > > Since the linux-gke flavor is used only for minimal GKE images, we could >> > > reasonably exclude these from the NBS cleaning going forward. >> > >> > > I've reached out to our Public Cloud team to see what their preference is. >> > >> > I've clarified with the Public Cloud team that, although the GKE images use >> > a minimal base, the images are built using linux-gke-$version, NOT >> > linux-image-gke-$version. >> > >> > So it's unclear that what Tai-Lin is doing is a use case that the Public >> > Cloud team is concerned with supporting. >> > >> > Is 5.15.0-1027-gke the version of a running kernel on your GKE instance? If >> > so, why is the headers package not installed already? If not, why are you >> > trying to compile kernel modules for this version? >> > >> > > > Separately, whilst this NBS cleanup is in place, you can use the tool >> > > > `pull-lp-debs` from ubuntu-dev-tools which should allow you to >> > > > securely fetch any of the packages you require out of Launchpad >> > > > Librarian archival service. (note that pull-lp-debs is part of >> > > > collection of tools pull-ppa-ddebs pull-ppa-debs pull-ppa-source - >> > > > which are all wrappers around the swiss army knife type of tool >> > > > pull-pkg which can pull anything and everything out of Launchpad, >> > > > PPAs, Debian) >> > > >> > > >> > > > > Best, >> > > > > >> > > > > On Wed, May 10, 2023 at 12:51 AM Stefan Bader >> > > > > <<a href="mailto:stefan.bader@canonical.com" target="_blank">stefan.bader@canonical.com</a>> wrote: >> > > > > > >> > > > > > On 09.05.23 22:28, Tai-Lin Chu wrote: >> > > > > > > hi, >> > > > > > > I received alerts about linux-headers-5.15.0-1027-gke being removed last night. >> > > > > > > What might be the reason for that? thanks! >> > > > > > > >> > > > > > > Get:3 <a href="http://security.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://security.ubuntu.com/ubuntu</a> jammy-security/restricted >> > > > > > > amd64 Packages [1077 kB] >> > > > > > > Get:4 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-updates InRelease [119 kB] >> > > > > > > Get:5 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-backports InRelease [108 kB] >> > > > > > > Get:6 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy/main amd64 Packages [1792 kB] >> > > > > > > Get:7 <a href="http://security.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://security.ubuntu.com/ubuntu</a> jammy-security/main amd64 >> > > > > > > Packages [585 kB] >> > > > > > > Get:8 <a href="http://security.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://security.ubuntu.com/ubuntu</a> jammy-security/universe amd64 >> > > > > > > Packages [898 kB] >> > > > > > > Get:9 <a href="http://security.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://security.ubuntu.com/ubuntu</a> jammy-security/multiverse >> > > > > > > amd64 Packages [41.2 kB] >> > > > > > > Get:10 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy/universe amd64 Packages [17.5 MB] >> > > > > > > Get:11 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy/multiverse amd64 Packages [266 kB] >> > > > > > > Get:12 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy/restricted amd64 Packages [164 kB] >> > > > > > > Get:13 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-updates/multiverse amd64 >> > > > > > > Packages [46.6 kB] >> > > > > > > Get:14 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-updates/main amd64 >> > > > > > > Packages [992 kB] >> > > > > > > Get:15 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-updates/restricted amd64 >> > > > > > > Packages [1137 kB] >> > > > > > > Get:16 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-updates/universe amd64 >> > > > > > > Packages [1143 kB] >> > > > > > > Get:17 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-backports/universe amd64 >> > > > > > > Packages [25.6 kB] >> > > > > > > Get:18 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-backports/main amd64 >> > > > > > > Packages [49.4 kB] >> > > > > > > Fetched 26.3 MB in 2s (10.6 MB/s) >> > > > > > > Reading package lists... >> > > > > > > + apt-get install -y linux-headers-5.15.0-1027-gke >> > > > > > > linux-modules-extra-5.15.0-1027-gke >> > > > > > > Reading package lists... >> > > > > > > Building dependency tree... >> > > > > > > Reading state information... >> > > > > > > E: Unable to locate package linux-headers-5.15.0-1027-gke >> > > > > > > E: Couldn't find any package by glob 'linux-headers-5.15.0-1027-gke' >> > > > > > > E: Couldn't find any package by regex 'linux-headers-5.15.0-1027-gke' >> > > > > > > >> > > > > > >> > > > > > That is just the normal way things change with updates. The archive will >> > > > > > not find older version which have been replaced by newer ones. You >> > > > > > should never try to install specific versions for that reason. Try "apt >> > > > > > install linux-headers-gke". >> > > > > > -- >> > > > > > - Stefan >> > > > > > >> > > > > >> > > > > -- >> > > > > kernel-team mailing list >> > > > > <a href="mailto:kernel-team@lists.ubuntu.com" target="_blank">kernel-team@lists.ubuntu.com</a> >> > > > > <a href="https://lists.ubuntu.com/mailman/listinfo/kernel-team" rel="noreferrer" target="_blank">https://lists.ubuntu.com/mailman/listinfo/kernel-team</a> >> > > > >> > > > >> > > > >> > > > -- >> > > > okurrr, >> > > > >> > > > Dimitri >> > > > >> > > >> > > -- >> > > Steve Langasek Give me a lever long enough and a Free OS >> > > Debian Developer to set it on, and I can move the world. >> > > Ubuntu Developer <a href="https://www.debian.org/" rel="noreferrer" target="_blank">https://www.debian.org/</a> >> > > <a href="mailto:slangasek@ubuntu.com" target="_blank">slangasek@ubuntu.com</a> <a href="mailto:vorlon@debian.org" target="_blank">vorlon@debian.org</a> >> > >> > -- >> > Steve Langasek Give me a lever long enough and a Free OS >> > Debian Developer to set it on, and I can move the world. >> > Ubuntu Developer <a href="https://www.debian.org/" rel="noreferrer" target="_blank">https://www.debian.org/</a> >> > <a href="mailto:slangasek@ubuntu.com" target="_blank">slangasek@ubuntu.com</a> <a href="mailto:vorlon@debian.org" target="_blank">vorlon@debian.org</a> >> > > > -- > Phil Roche > Staff Software Engineer > Canonical Public Cloud </blockquote></div> <div> </div>-- <div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Phil Roche </div><div>Staff Software Engineer <div>Canonical Public Cloud</div></div></div></div></div></div></div></div>