<div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div><a class="gmail_plusreply" id="gmail-plusReplyChip-0">@tailinchu@gmail.com Hi,</a><br></div><div><a class="gmail_plusreply"><br></a></div><div>I work on the Canonical Public Cloud team, and I work with the GKE team to build and publish the Ubuntu GKE node images.<a class="gmail_plusreply"><br></a></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">---------- Forwarded message ----------<br>From: Tai-Lin Chu <<a href="mailto:tailinchu@gmail.com" target="_blank">tailinchu@gmail.com</a>><br>To: Steve Langasek <<a href="mailto:steve.langasek@canonical.com" target="_blank">steve.langasek@canonical.com</a>>, Dimitri John Ledkov <<a href="mailto:dimitri.ledkov@canonical.com" target="_blank">dimitri.ledkov@canonical.com</a>>, Tai-Lin Chu <<a href="mailto:tailinchu@gmail.com" target="_blank">tailinchu@gmail.com</a>>, Stefan Bader <<a href="mailto:stefan.bader@canonical.com" target="_blank">stefan.bader@canonical.com</a>>, <a href="mailto:kernel-team@lists.ubuntu.com" target="_blank">kernel-team@lists.ubuntu.com</a><br>Cc: <br>Bcc: <br>Date: Wed, 10 May 2023 12:16:52 -0700<br>Subject: Re: linux-headers-5.15.0-1027-gke is removed from jammy last night<br>Thanks for replying.<br>
Let me clarify the use case: we use gke ubuntu containerd node image.<br>
Because we cannot know which kernel version the node is using, on vm<br>
creation completed, we will compile and inject a kernel module so that<br>
pods can use that kernel module:<br>
<br>
apt-get install -y "linux-headers-$(uname -r)" "linux-modules-extra-$(uname -r)"<br>
<br>
> Is 5.15.0-1027-gke the version of a running kernel on your GKE instance?<br>
yes.<br>
<br>
> If so, why is the headers package not installed already?<br>
I don't think the kernel header package exists on the node image<br>
<a href="https://cloud.google.com/kubernetes-engine/docs/concepts/node-images#ubuntu-variants" rel="noreferrer" target="_blank">https://cloud.google.com/kubernetes-engine/docs/concepts/node-images#ubuntu-variants</a>.<br></blockquote><div><br></div><div>The headers are pre-installed in all the Ubuntu node images<br><br>```<br><span style="font-family:monospace"><span style="font-weight:bold;color:rgb(84,84,255)">~</span><span style="color:rgb(0,0,0)">$ apt list --installed | grep headers
</span><br>linux-gke-<span style="font-weight:bold;color:rgb(255,84,84)">headers</span><span style="color:rgb(0,0,0)">-5.15.0-1028/now 5.15.0-1028.33 amd64 [installed,local]
</span><br>linux-<span style="font-weight:bold;color:rgb(255,84,84)">headers</span><span style="color:rgb(0,0,0)">-5.15.0-1028-gke/now 5.15.0-1028.33 amd64 [installed,local]
</span><br>linux-<span style="font-weight:bold;color:rgb(255,84,84)">headers</span><span style="color:rgb(0,0,0)">-gke-5.15/now 5.15.0.1028.27 amd64 [installed,local]</span><br>
```</span></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">From that mailing list post, that timeline was not publicly announced<br>
on gke updates, so we were completely unaware of it, and caused<br>
incidents. Our current workaround is using 20.04 with the older gke<br>
version because the new auto cleanup process is for jammy and later,<br>
but I hope that older kernel can be kept around.<br></blockquote><div><br></div><div>Aside from any discussions about the pruning of the headers from the archive, another workaround or - permanent solution - is to bind mount what you need from the host to the container. The headers are present on the node so this should work.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Does canonical collaborate gcp to create node image? </blockquote><div><br></div><div>Yes, we do.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">If so, I can also<br>
forward this to our gcp dedicated account team.<br></blockquote><div><br></div><div>Yes, I see there was an internal case raised now which I will respond to. </div><div><br></div><div>Phil</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Thanks!<br>
<br>
On Wed, May 10, 2023 at 8:29 AM Steve Langasek<br>
<<a href="mailto:steve.langasek@canonical.com" target="_blank">steve.langasek@canonical.com</a>> wrote:<br>
><br>
> On Wed, May 10, 2023 at 08:12:50AM -0700, Steve Langasek wrote:<br>
> > When Andy and I looked at this, the analysis had showed that all our images<br>
> > except for minimal images were being built with linux-$flavor installed,<br>
> > rather than linux-image-$flavor, so that the headers would already be<br>
> > present and removal of the old ABIs from the archive would have no impact on<br>
> > users of these images.<br>
> ><br>
> > The GKE images are being built on a minimal base, which I did not recall.<br>
> ><br>
> > It is not otherwise an issue for cloud images per above, with the exception<br>
> > of the cloud-minimal images.<br>
> ><br>
> > Since the linux-gke flavor is used only for minimal GKE images, we could<br>
> > reasonably exclude these from the NBS cleaning going forward.<br>
><br>
> > I've reached out to our Public Cloud team to see what their preference is.<br>
><br>
> I've clarified with the Public Cloud team that, although the GKE images use<br>
> a minimal base, the images are built using linux-gke-$version, NOT<br>
> linux-image-gke-$version.<br>
><br>
> So it's unclear that what Tai-Lin is doing is a use case that the Public<br>
> Cloud team is concerned with supporting.<br>
><br>
> Is 5.15.0-1027-gke the version of a running kernel on your GKE instance? If<br>
> so, why is the headers package not installed already? If not, why are you<br>
> trying to compile kernel modules for this version?<br>
><br>
> > > Separately, whilst this NBS cleanup is in place, you can use the tool<br>
> > > `pull-lp-debs` from ubuntu-dev-tools which should allow you to<br>
> > > securely fetch any of the packages you require out of Launchpad<br>
> > > Librarian archival service. (note that pull-lp-debs is part of<br>
> > > collection of tools pull-ppa-ddebs pull-ppa-debs pull-ppa-source -<br>
> > > which are all wrappers around the swiss army knife type of tool<br>
> > > pull-pkg which can pull anything and everything out of Launchpad,<br>
> > > PPAs, Debian)<br>
> ><br>
> ><br>
> > > > Best,<br>
> > > ><br>
> > > > On Wed, May 10, 2023 at 12:51 AM Stefan Bader<br>
> > > > <<a href="mailto:stefan.bader@canonical.com" target="_blank">stefan.bader@canonical.com</a>> wrote:<br>
> > > > ><br>
> > > > > On 09.05.23 22:28, Tai-Lin Chu wrote:<br>
> > > > > > hi,<br>
> > > > > > I received alerts about linux-headers-5.15.0-1027-gke being removed last night.<br>
> > > > > > What might be the reason for that? thanks!<br>
> > > > > ><br>
> > > > > > Get:3 <a href="http://security.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://security.ubuntu.com/ubuntu</a> jammy-security/restricted<br>
> > > > > > amd64 Packages [1077 kB]<br>
> > > > > > Get:4 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-updates InRelease [119 kB]<br>
> > > > > > Get:5 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-backports InRelease [108 kB]<br>
> > > > > > Get:6 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy/main amd64 Packages [1792 kB]<br>
> > > > > > Get:7 <a href="http://security.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://security.ubuntu.com/ubuntu</a> jammy-security/main amd64<br>
> > > > > > Packages [585 kB]<br>
> > > > > > Get:8 <a href="http://security.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://security.ubuntu.com/ubuntu</a> jammy-security/universe amd64<br>
> > > > > > Packages [898 kB]<br>
> > > > > > Get:9 <a href="http://security.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://security.ubuntu.com/ubuntu</a> jammy-security/multiverse<br>
> > > > > > amd64 Packages [41.2 kB]<br>
> > > > > > Get:10 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy/universe amd64 Packages [17.5 MB]<br>
> > > > > > Get:11 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy/multiverse amd64 Packages [266 kB]<br>
> > > > > > Get:12 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy/restricted amd64 Packages [164 kB]<br>
> > > > > > Get:13 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-updates/multiverse amd64<br>
> > > > > > Packages [46.6 kB]<br>
> > > > > > Get:14 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-updates/main amd64<br>
> > > > > > Packages [992 kB]<br>
> > > > > > Get:15 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-updates/restricted amd64<br>
> > > > > > Packages [1137 kB]<br>
> > > > > > Get:16 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-updates/universe amd64<br>
> > > > > > Packages [1143 kB]<br>
> > > > > > Get:17 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-backports/universe amd64<br>
> > > > > > Packages [25.6 kB]<br>
> > > > > > Get:18 <a href="http://archive.ubuntu.com/ubuntu" rel="noreferrer" target="_blank">http://archive.ubuntu.com/ubuntu</a> jammy-backports/main amd64<br>
> > > > > > Packages [49.4 kB]<br>
> > > > > > Fetched 26.3 MB in 2s (10.6 MB/s)<br>
> > > > > > Reading package lists...<br>
> > > > > > + apt-get install -y linux-headers-5.15.0-1027-gke<br>
> > > > > > linux-modules-extra-5.15.0-1027-gke<br>
> > > > > > Reading package lists...<br>
> > > > > > Building dependency tree...<br>
> > > > > > Reading state information...<br>
> > > > > > E: Unable to locate package linux-headers-5.15.0-1027-gke<br>
> > > > > > E: Couldn't find any package by glob 'linux-headers-5.15.0-1027-gke'<br>
> > > > > > E: Couldn't find any package by regex 'linux-headers-5.15.0-1027-gke'<br>
> > > > > ><br>
> > > > ><br>
> > > > > That is just the normal way things change with updates. The archive will<br>
> > > > > not find older version which have been replaced by newer ones. You<br>
> > > > > should never try to install specific versions for that reason. Try "apt<br>
> > > > > install linux-headers-gke".<br>
> > > > > --<br>
> > > > > - Stefan<br>
> > > > ><br>
> > > ><br>
> > > > --<br>
> > > > kernel-team mailing list<br>
> > > > <a href="mailto:kernel-team@lists.ubuntu.com" target="_blank">kernel-team@lists.ubuntu.com</a><br>
> > > > <a href="https://lists.ubuntu.com/mailman/listinfo/kernel-team" rel="noreferrer" target="_blank">https://lists.ubuntu.com/mailman/listinfo/kernel-team</a><br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > okurrr,<br>
> > ><br>
> > > Dimitri<br>
> > ><br>
> ><br>
> > --<br>
> > Steve Langasek Give me a lever long enough and a Free OS<br>
> > Debian Developer to set it on, and I can move the world.<br>
> > Ubuntu Developer <a href="https://www.debian.org/" rel="noreferrer" target="_blank">https://www.debian.org/</a><br>
> > <a href="mailto:slangasek@ubuntu.com" target="_blank">slangasek@ubuntu.com</a> <a href="mailto:vorlon@debian.org" target="_blank">vorlon@debian.org</a><br>
><br>
> --<br>
> Steve Langasek Give me a lever long enough and a Free OS<br>
> Debian Developer to set it on, and I can move the world.<br>
> Ubuntu Developer <a href="https://www.debian.org/" rel="noreferrer" target="_blank">https://www.debian.org/</a><br>
> <a href="mailto:slangasek@ubuntu.com" target="_blank">slangasek@ubuntu.com</a> <a href="mailto:vorlon@debian.org" target="_blank">vorlon@debian.org</a><br>
<br>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Phil Roche<br></div><div>Staff Software Engineer<br><div>Canonical Public Cloud</div></div></div></div></div></div></div>