VirtualBox Postmortem: Second Large User-Facing Regression in the last 30 days
Steve Langasek
steve.langasek at ubuntu.com
Fri Feb 2 04:31:41 UTC 2024
It's been a couple of weeks since this message was sent. It was discussed
in the Technical Board meeting Tuesday that it should be the SRU Team that
responds.
In an SRU Team meeting today, it was agreed that some measure of response is
in order.
The SRU Team does not intend to make a point-by-point response to this
email. This email presents itself as containing a complete finding of fact,
draws conclusions, and demands remedies without ever consulting one of the
major parties involved (the SRU team) for input and is therefore does not
present a sound basis for a collaborative retrospective exercise.
Nevertheless, we agree that a review of this incident is required, as this
was a significant regression in the updates pocket that should not have been
allowed to happen.
The SRU Team will be taking the following actions:
- Conducting an internal retrospective regarding the fact that a kernel
update was released that regressed buildability of a dkms package, which
the existing kernel tests are intended to guard against. The results of
this retrospective will be posted to the ubuntu-release and
technical-board mailing lists.
- Following through on the request for an SRU exception for virtualbox.
Chris Halse Rogers will be leading this effort on behalf of the SRU Team.
On Wed, Jan 17, 2024 at 09:17:32PM +0000, Simon Quigley wrote:
> Dear Ubuntu Technical Board and Ubuntu Stable Release Updates Team,
>
> (I am writing to both teams in lieu of a proper escalation procedure
> within the Stable Release Updates Team for problems such as this. I
> understand it is a current action item for the Technical Board to work
> with the Stable Release Updates Team on a documented process for team
> membership. For the sake of the issue at hand, I will be ignoring this
> discussion and debate entirely.)
>
> VirtualBox is an incredibly popular piece of cross-platform
> virtualization software, with a wide-ranging user base. On Ubuntu,
> VirtualBox is used as an easy alternative to Red Hat's Virtual Machine
> Manager, and could very well be used in servers with a GUI. While this
> is in Multiverse, there is a large demand for VirtualBox.
>
> Gianfranco Costamanga effectively maintains VirtualBox in Debian and
> Ubuntu. He is an outstanding contributor with deep technical knowledge
> that surpasses mine at unexpected times. Gianfranco helps with
> transitions often, pinging in the appropriate channels to get eyes on
> issues from the appropriate teams. Even as my own knowledge continues to
> grow, when I am completely out of my depth, Gianfranco usually has the
> answer. I trust him wholeheartedly.
>
> A situation occurred in which the latest HWE kernel was released.
> Unfortunately, this caused a showstopping regression in VirtualBox,
> which has a set of DKMS modules. Here is the timeline of events:
> - 20151104: Martin Pitt ACKs a VirtualBox SRU on a single-case basis,
> as a response to Gianfranco's MRE request:
> https://lists.ubuntu.com/archives/technical-board/2015-November/002177.html
> - 20230420: Gianfranco files a bug for tracking of a new update to
> VirtualBox:
> https://bugs.launchpad.net/ubuntu/mantic/+source/virtualbox-hwe/+bug/2017101
> - 20230421: Andreas Hasenack follows up on the bug asking for more
> information, Gianfranco follows up afterwards but receives no response:
> https://bugs.launchpad.net/ubuntu/mantic/+source/virtualbox-hwe/+bug/2017101/comments/4
> - 20230915 10:14 UTC: Robie agrees this should receive a followup on
> the mailing list:
> https://irclogs.ubuntu.com/2023/09/15/%23ubuntu-release.html#t10:14
> - 20230915 10:23 UTC: Gianfranco follows up on this request, and asks
> for a general discussion on the issue. No response is received on the
> mailing list:
> https://lists.ubuntu.com/archives/ubuntu-release/2023-September/005787.html
> - 20230916: Gianfranco indicates that these uploads are now in the
> queue:
> https://bugs.launchpad.net/ubuntu/mantic/+source/virtualbox-hwe/+bug/2017101/comments/8
> - 2023/09/25/#ubuntu-release [10:19] <LocutusOfBorg> hello, ping to
> check virtualbox SRU
> - 2023/09/28/#ubuntu-release [16:00] <LocutusOfBorg> can anybody
> please have a look at virtualbox SRU?
> - 20230928 16:22 UTC: Gianfranco indicates that he updated the
> exception page given feedback from Robie Basak:
> https://irclogs.ubuntu.com/2023/09/28/%23ubuntu-release.html#t16:22
> - 2023/11/06/#ubuntu-release [11:16] <LocutusOfBorg> hello SRU team,
> any news w.r.t. virtualbox SRU?
> - 2023/11/07/#ubuntu-release [15:27] <LocutusOfBorg> tjaalton, hello,
> do you think we can do something w.r.t virtualbox sru?
> - 2023/11/17/#ubuntu-release [09:02] <LocutusOfBorg> also virtualbox
> is waiting there since months
> - 20231125:
> https://bugs.launchpad.net/ubuntu/+source/virtualbox/+bug/2044598 is
> filed regarding incompatibilities between the Linux 6.5 HWE kernel and
> virtualbox as in the archive.
> - 20231127: The Kernel Team uploads the initial 6.5 HWE kernel to
> Jammy:
> https://launchpad.net/ubuntu/+source/linux-meta-hwe-6.5/6.5.0.14.14~22.04.6
> - 2023/12/11/#ubuntu-release [13:04] <LocutusOfBorg> also ubuntu SRU,
> is it normal to have virtualbox in unapproved queue since april?
> - 2023/12/11/#ubuntu-release [13:04] <LocutusOfBorg>
> https://bugs.launchpad.net/ubuntu/mantic/+source/virtualbox-hwe/+bug/2017101
> - 20231213 10:02:32 PM US Central: The 6.5 HWE kernel is promoted to
> Main:
> https://launchpad.net/ubuntu/+source/linux-meta-hwe-6.5/+publishinghistory
> - 20240110 04:00:23 AM US Central: 6.5 HWE kernel lands in -updates
> and starts phasing.
> - 20240110 06:40:28 AM US Central: 6.5 HWE kernel lands in -security
> which sets phasing to 100%.
> - 20240111 3:18 AM US Central: Graham Inggs pings Gianfranco in a
> private channel asking if he knows anything about virtualbox-dkms
> failures in Jammy. Gianfranco follows up with deep frustration within a
> minute. He indicates that he stopped pinging because nobody would listen
> to him.
> - 20240111 21:00 UTC: I caught up on the backlog and realized what is
> going on. Pinged both the SRU Team and the Release Team (the latter of
> which simply for visibility):
> https://irclogs.ubuntu.com/2024/01/11/%23ubuntu-release.html#t21:00
> - 20240111 23:18 UTC to 20240112 01:16 UTC: Brian Murray reviews the
> SRUs in the queue.
> - 20240112 00:59 UTC: Aaron Rainbolt begins testing the package.
> - 20240112 18:48 UTC: I emphasize the gravity of the situation in
> #ubuntu-release and ask for Brian to carefully weigh the options:
> https://irclogs.ubuntu.com/2024/01/12/%23ubuntu-release.html#t18:48
> - 20240112 6:30 PM US Central: After several meetings, I look outside
> to see that the Wisconsin blizzard has my car almost fully covered. I
> had about 5-10 minutes to leave before I would have been stuck at the
> cowork space for the night. Aaron Rainbolt begins his usual sabbatical,
> which occurs from Friday at sunset to Saturday at sunset.
> - 20240112-20240115: I ping the Lubuntu Team asking for QA hands to
> give some extra help if they could. Nobody else besides me and Aaron
> ended up testing this.
> - 20240114 17:18 UTC: Aaron Rainbolt pings the SRU Team asking for
> acceptance of the package into -updates. Timo Aaltonen indicates that
> it's still a Sunday, and nobody will be around to accept it. I respond
> indicating that it was, in fact, Monday already for one SRU Team member,
> Chris Halse Rogers, who ignored my ping asking for a second set of eyes.
> - 20240115 12:20:39 PM US Central: The VirtualBox update was published
> to -updates, going through the normal phasing process, unlike the kernel
> update. Fortunately, the phasing has not been halted. Unfortunately, as
> of the time of writing, 20% of users are still facing this issue.
> - 20240115 20:28 UTC: Ubuntu Weekly Newsletter 822 was published, with
> two of the five top posts on Ask Ubuntu for the week being about this
> issue:
> https://lists.ubuntu.com/archives/ubuntu-news/2024-January/000898.html
>
> Before I get into the various issues here, allow me to thank the people
> who have been involved with this. Gianfranco, thank you for your efforts
> in trying to get this addressed. Aaron, thank you for taking the large
> amount of time to test the majority of the packages in question. Brian,
> thank you for promptly acting on my loud ping, and addressing it the
> best you can (I don't expect you to work weekends.) Robie, thank you for
> reviewing and guiding Gianfranco on the MRE process. And Martin, on the
> off chance you are reading this, thank you for your work on this in 2015.
>
> Here are the rather large issues I am seeing with how this was handled:
> - The Ubuntu Stable Release Updates Team needs to at least acknowledge
> proposals when given to them, especially by prominent community members.
> Robie took a great first step, but that mailing list post is from
> September, and no followup of any kind on the mailing list was received.
> - Gianfranco Costamanga is a contributor that is more valuable to
> Ubuntu than anyone could put into words. Ignoring his pings for four
> months only de-motivates him. I am glad he is happy about this being
> addressed, but what does this tell Gianfranco about future issues on a
> similar scale? The SRU Team as a whole owes him an apology.
> - Exact figures are not public, but VirtualBox is very widely used.
> Ubuntu's offering of VirtualBox needs to work, and user-facing
> regressions in an LTS release, let alone any stable release, are
> completely unacceptable. This can not happen again. All of us, as
> Ubuntu, owe an apology to our users.
> - In the Ask Ubuntu articles linked in UWN, the answers for both were
> "just install VirtualBox from a third-party repository." This may not
> seem like a large issue at first, but think about it. If I was a
> malicious attacker, this would have been the perfect opportunity to spin
> up a repository with a dirty VirtualBox. Users just want to be
> unblocked, most of them do not spend time worrying about what exactly is
> in this third-party repository that they're enabling (and it's a great
> argument for snaps in a general sense.) We are VERY lucky this did not
> happen.
> - Canonical has customers that rely on Ubuntu to be secure and
> relatively bug-free. I do not work for Canonical at this point in time,
> and I am completely unfamiliar with current customer deployments. That
> being said, it would not surprise me if a paying Canonical customer uses
> VirtualBox, and was angry about this regression. This not only hurt our
> image as Ubuntu, this hurt Canonical as a company.
>
> It really does pain me to write this email. I am not doing this for the
> sake of starting an argument, my genuine intent is to lay out all of the
> facts and discuss it, so we can move forward together and better. Nobody
> should take this as a personal attack, this is not how I mean it. That
> being said, this is the *second* large, user-facing regression in the
> last 30 days. The Desktop Team committed to providing a postmortem on
> the Mutter issue, where video playback was broken on all stable 23.10
> installs, but they have not delivered on that commitment. What is going
> on here?
>
> I have one, simple question for the Technical Board and the Stable
> Release Updates Team: how do we ensure this never happens again?
>
> --
> Simon Quigley
> simon at tsimonq2.net
> tsimonq2 on LiberaChat and OFTC
> @tsimonq2:ubuntu.com on Matrix
> 5C7A BEA2 0F86 3045 9CC8
> C8B5 E27F 2CF8 458C 2FA4
> --
> technical-board mailing list
> technical-board at lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/technical-board
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
slangasek at ubuntu.com vorlon at debian.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/technical-board/attachments/20240201/87a61018/attachment.sig>
More information about the technical-board
mailing list