Validation of keyring changes [was: Enhancing cross-distro collaboration via foreign archive keyring] availability

[Robie Basak]

On Fri, Sep 13, 2024 at 05:20:34PM +0200, Luca Boccassi wrote:
> I think we need to be careful to avoid making our lives unnecessarily
> hard without good reasons.

I agree with this principle. However, I think that given that you're
asking to make changes to a trust root for the purposes of maintaining
that trust root in the archive itself, it's reasonable to spend, at a
minimum, the effort needed to arrive at a documented scheme that
describes how that trust is established. So I hope that you don't
consider this converation here to be unnecessary.

>                            If certain workflows are good enough for
> example to import new kernel or compiler sources, or for
> ubuntu-keyring updates, then they also must be good enough for this,
> which has a much narrower and less impactful use case. That said, an
> agreed and documented robust-enough workflow is a good idea.

I think these are false equivalences. ubuntu-keyring updates are almost
never done, and validating them against the actual keys used by Ubuntu
infrastructure is trivial for Ubuntu developers to do. Further, for
trust purposes, the updates were provided by an employee of the same
entity that provides Ubuntu's infrastructure (Canonical), so there can
be some level of inherent trust that any replacement keys provided did
indeed belong to the same entity. Validation of the individual
employees' identities can be done trivially on our own infrastructure
that we must fully trust already for our archive publication
(Launchpad). But in the general case that you're requesting here, that
trust connection isn't present. This isn't a showstopper, but it does
mean that we need to establish an alternate means of validating the
trust chain.

Further, I don't know what specific validation was actually done in the
rare cases that ubuntu-keyring was changed in a stable release in the
past, but any shortcomings there are something we should fix in the wake
of supply chain attacks such as what happened in xz-utils. You're
proposing to do something new in ways that haven't been done before, so
now is the time to decide what level of validation is appropriate for
the security landscape today. What happened with ubuntu-keyring for a
couple of one-off events in the past cannot be considered to be a
precedent for the rather different thing you are proposing to do now.

In the common case for stable updates in Ubuntu, we review the actual
code changes themselves, so authenticating their origin isn't critical
for supply chain security. In contrast, for keyring changes, validating
the upstream origin is absolutely critical since key changes cannot be
checked by mere inspection.

One of the (many) aspects that saved us is the delay between Debian sid
being updated and Ubuntu stable releases getting it. But here, you're
proposing to take that delay away, updating Ubuntu stable releases as
soon as the change is signed and pushed to Debian Salsa. So this
situation is different and standards applied elsewhere do not transfer
because the security implications are different.

> So, a good enough workflow that should suffice for all intent and
> purposes in my view would be: instruct to never upload from a random
> tarball attached to a launchpad bug or an email. Always and only
> upload from a tag laid on the ubuntu/<release> branch on Salsa, after
> checking that the tag is signed.

I don't think we can expect sponsors to follow this kind of rule. Since
we usually expect them to use careful code inspection only, it's
difficult to create a place that they would automatically for special
rules such that we'd be confident that they'd follow them. That's not to
say that we shouldn't define and document this somewhere, but rather
that our process shouldn't depend on it.

Instead, I think we should use the SRU review process to ensure this
kind of checking is done. I'm less keen on using autopkgtests (again, to
rely upon; having them is fine) because in our pipeline packages would
still be published to -proposed and therefore available "officially"
before that validation has happened. This would need the tooling to be
able to be run outside the autopkgtest environment.

I think we need more specifics though.

We would check that the tag is signed, but by whom? Who decides the set
of signers that is acceptable? How do we manage changes to that set?

If that's all managed at the Salsa end, then that's worse security than
what Debian have today, because that's not necessarily even restricted
to those who can upload to the Debian archive!

I can sense potential frustration in anyone reading this. But
unfortunately this sort of validation provides no security value unless
these things are specified.

# How to automatically perform the validation

Could we place the set of acceptable signers for the git tag signature
in the source package itself? Then have a script in the source tree that
the sponsor and SRU reviewer can both run that validates against Salsa
as you suggest? The special case documention would then specify that
expectation and the SRU reviewer would merely have to run the script.
They already expect an explanation for any other changes, so that would
maintain the integrity of this check.

This is necessary because in our workflow we upload before the
autopkgtests run.

This way, when reviewing we would normally expect that script and the
acceptable signature list not to change, and it would be very clearly
evident if the proposed update requires them to change.

>                                  The current trustworthiness of git
> tags is good enough for everything else including the kernel, so it's
> good enough for this too, and there are no real attacks that can be
> done against git tags that need to worry anyone at this stage. Only
> members of the Salsa RPM Packaging Team have permission to push tags
> there. I am not the only person in the team, there are a couple of
> other people, so also the single-person-bottleneck issue is solved.

There's still the question of the entire trust chain failing if a single
person's personal keys are compromised.

We generally accept this in development release, but normally all
changes to stable releases in SRUs are reviewed by at least two people.
Is it possible to require the tag to have been signed by two appropriate
people who have independently checked that keys are correct? How is that
checking to be done and what commitment is the team signing tags on
Salsa making to do such checking?

Robie
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-devel/attachments/20240918/643049a8/attachment.sig>