Fwd: lxd-agent-loader/lxd-installer SRU exception policy review

Fri Oct 25 02:58:37 UTC 2024

I mistakenly replied to Robie alone, so I'm now forwarding it to the
mailing list. Sorry about that.

---------- Forwarded message ---------
From: Simon Déziel <simon.deziel at canonical.com>
Date: Fri, Oct 25, 2024 at 2:55 AM
Subject: Re: lxd-agent-loader/lxd-installer SRU exception policy review
To: Robie Basak <robie.basak at ubuntu.com>

Hello Robie,

On Thu, Oct 24, 2024 at 10:20 AM Robie Basak <robie.basak at ubuntu.com> wrote:

> Hi Simon,
>
> Thank you for working to make users' experience of LXD better for Ubuntu
> users!
>

Thanks for taking the time to analyse this request and taking a look at the
git repos of those two packages.

> On Thu, Oct 24, 2024 at 01:07:29AM +0000, Simon Déziel wrote:
> > I would like to request an SRU exception [1] for the lxd-agent-loader
> > and lxd-installer packages in all supported Ubuntu releases. I have
> > created a wiki page [2] that outlines everything I believe is relevant
> > to this SRU exception.
>
> I agree that some changes to the lxd experience probably should fit
> under our "For Long Term Support releases we sometimes want to introduce
> new features" justification for SRUs to stable Ubuntu releases[1]. Note
> though that this isn't an "exception" as such since existing SRU policy
> permits this.
>
> We do need to ensure the condition "They must not change the behaviour
> on existing installations". I would further extend this to ensure that
> we do not change behaviour of *redeployments* since this is also a
> common use case for server users (eg. CI that uses LXD).
>

That's one of the aspects we'd like to improve by having the same version
available in all supported releases. See further below.

> I note that both lxd-agent-loader and lxd-installer are pretty minimal
> "3.0 (native)" packages. Changes should be easy to review. The easiest
> way to ensure the above constraints would be a full review of the
> changes being applied during both sponsorship and upload, and SRU review
> for the same thing. To that end, I'm not sure any kind of routine
> process that skips a full code review makes sense. Can we instead review
> requested changes in SRUs on a case-by-case basis?
>
> This wouldn't preclude you from maintaining (essentially) the same
> source tree of these packages across all supported Ubuntu releases.
> You'd need to ensure no changes to behaviour for existing Ubuntu users
> whichever way you do it. I'm just saying that case-by-case review
> is probably the easiest and smoothest way of achieving that.
>
> Normally other packages bound to a regular SRU cadence but without
> changing existing user behaviour (eg. cloud-init) have to be very
> careful to maintain this in every upstream change they make. This does
> compromise engineering velocity when developing new features and new
> behaviours. Either way, you'd need to be making the same commitment. I
> would question though if this is really necessary given that engineering
> burden.
>
> For example, I see a change[2] in lxd-installer that changes some
> default behaviour to failure instead of falling back to the default snap
> channel for lxd. It may be the case that in practice this won't result
> in behaviour change to users, but that's the kind of thing that we need
> to be careful about, and therefore needs reviewing carefully. Ideally
> the code change would be written in a way that makes it easier to
> validate that there really will be no change to user behaviour.
>

That behaviour change was a conscious decision as prior to that,
lxd-installer was picking up whatever was the default snap channel. Until
not too long ago it was `latest/stable` which shipped monthly feature
releases [0]. Now with the change in place, you are right that it will
error out if no suitable channel is found but we (the LXD team) feel it is
safer than silently falling back to the default channel with a very short
support lifetime and fast moving target.

Another problem on relying on the default channel is that we'd lose the
ability to publish a quick Ubuntu/release specific fix like we now have
when publishing to the `${LTS}/stable/ubuntu-XX.YY` channel.

> To proceed, I suggest that you start by reviewing to consider if all the
> changes to both lxd-installer and lxd-agent-loader would be acceptable
> to SRU for each Ubuntu release in terms of (the lack of) changes to
> behaviour. This might involve making some code changes to make it easier
> to demonstrate that this is the case. If after careful review you think
> that this is the case and demonstrable by code review, then the SRU team
> could also review to confirm that, and then we'd be able to update both
> packages in all stable releases as a one-off exercise.
>

I re-reviewed every change made to lxd-installer and still believe those
are all relevant for every supported release, even the one behaviour change
regarding the channel selection. This behaviour change is IMHO providing
our users with the best experience possible from a support and consistency
point of view. Those two aspects are worth elaborating on as they have
serious consequences so let's compare the experience between regular
(non-minimal where LXD is seeded) and minimal installs (using
lxd-installer) of 20.04. For that, let's rewind to say, May 2020, shortly
after 20.04's release.

User A: installs 20.04 minimal and gets LXD via lxd-installer. Back then it
would have been pulling from `latest/stable` which came with LXD 4.1
(non-LTS) at that time [1]. Since `latest/stable` is a moving target, that
user would now be running LXD 6.1 (still non-LTS). Furthermore, they'd be
unable to use ZFS with 20.04's 5.4 kernel as support for such an old ZFS
version was dropped after LXD 5.21, the latest LXD LTS version. To make
this situation even worse, this user could not go back to LXD 4.0 as
downgrades are not supported.

User B: installs 20.04 non-minimal and gets LXD automatically as it's
seeded. That user would now be running LXD 4.0.10 as the seeded snap uses
the `4.0/stable/ubuntu-20.04` channel. This is an LTS version of LXD which
is supported for as long as 20.04 is and ZFS (and all storage drivers for
that matter) works as well as when initially installed.

Similarly, a user depending on lxd-installer would also have had trouble
with redeployments as `latest/stable` has always been a moving target. As
such, having this change backported would mean that on 20.04, you'd always
get LXD 4.0 from `4.0/stable/ubuntu-20.04` no matter which image type you
picked and whenever you did your deployment.

I also re-reviewed the changes made to lxd-agent-loader and I still believe
all are relevant for every supported release. lxd-agent-loader saw much
less changes and most of them are not very user-visible so I guess doing
piecemeal SRUs would be a viable option, but I would probably just SRU [2]
as the rest is nice to have:

* fixing links to docs
* fixing links sources
* improving the reliability of the service startup which rarely doesn't
start
* speeding up the service startup a little
* etc

But those nice to haves would be hard to justify if they each require
individual SRUs.

Hopefully, those additional arguments and examples will provide additional
justifications for this SRU exception request.

--
Simon

Next time, we'll have a baseline to review against then. If this becomes
> annoyingly repetitive then we could consider some kind of standing
> agreement to skip some parts of the review at that point.
>
> None of this precludes us having some documentation that explains this
> plan, of course.
>
> Does this work for you?
>
> Robie
>
> [1]
> https://canonical-sru-docs.readthedocs-hosted.com/en/latest/reference/requirements/#other-safe-cases
> [2]
> https://git.launchpad.net/ubuntu/+source/lxd-installer/diff/lxd-installer-service?id=b9a2b3c3461eaf73976c069432782329465f579d

[0]
https://documentation.ubuntu.com/lxd/en/latest/explanation/security/#supported-versions
[1] https://github.com/canonical/lxd/releases/tag/lxd-4.1
[2]
https://git.launchpad.net/ubuntu/+source/lxd-agent-loader/commit/?id=4ae4eee713e0bc1ec4cb8d00f19bd4d3161fa127
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-release/attachments/20241025/6eb87f38/attachment.html>