Enhancing cross-distro collaboration via foreign archive keyring availability

[Luca Boccassi]

On Tue, 10 Sept 2024 at 19:12, Robie Basak <robie.basak at ubuntu.com> wrote:
>
> Hi,
>
> On Wed, Sep 04, 2024 at 10:51:30AM +0100, Luca Boccassi wrote:
> > For developers like myself who work across distributions, it is very
> > valuable to have a secure, simple and transparent way to bootstrap one
> > distro from another without relying on external trust anchors that
> > have to be verified manually.
>
> I agree that it makes sense to place a trust anchor in the Ubuntu
> archive, assuming that there's someone who commits to maintaining it (as
> you've said you will).
>
> What I don't understand is why all of this seems far more complicated
> than would seem necessary to me. I think this burdens patch pilots and
> the SRU team with unnecessary work for little benefit. Why isn't the
> architecture simpler? I go into that into detail below. Sorry the text
> is long, but there's no way to present a counterargument for something
> that seems unnecessarily complicated without diving into the apparent
> unnecessary complications :-/
>
> It seems burdensome to me to include a large set of keys and update them
> (presumably) more frequently, when (IMHO) only a single anchor is needed
> which would presumably need updating less frequently. It's not just the
> person maintaining the package, but also somebody else from a privileged
> team is going to have to review changes, whether we conclude that such
> changes should go into -updates or -backports.
>
> Further, how is such a reviewer going to validate the changes? Doing so
> is is crucially important. If this isn't done, then there's a path to
> invalidating all of the trust that is supposed to be achieved by
> including these keys in the first place.
>
> If consensus is to include any form of these packages, I think it's
> essential to also have consensus on what level of validation is
> acceptable, and then have that well documented and tooling provided that
> implements it. This is so important for keyring-type packages,
> especially where validation isn't obvious because they are coming from
> outside the distribution, that I think it's unreasonable to expect the
> SRU team to review these uploads without this having been thought about
> first. If there's one thing that needs to come out of me asking for
> consensus over the general approach first, over piecemeal updates, it's
> this.
>
> But if all we're doing is taking the keys from other places and updating
> them in Ubuntu, validated by some process that ultimately relies on some
> set of people to assert that the keys are correct, then what are we
> achieving anyway? Can this not just be automated then, and tooling be
> provided in the archive instead, so users can just do that directly when
> they need? Then there would be much reduced burden on maintainence,
> including for the relevant privileged review teams.
>
> We do gain from there being a root of trust in the archive, but that
> only need be a single root of trust to bootstrap trust in live, external
> sources. This is what I mean above when I say that IMHO only a single
> anchor is needed.
>
> Otherwise we're going to get a proliferation of external keyrings that
> need to be maintained, seemingly unbounded. I don't think this
> architecture scales.
>
> Here's an alternative, much simpler architecture. Someone could start an
> "upstream" project that exists for the purpose of maintaining these
> keyrings cross-distro. Let's call that a "cross-distro keyring". The
> most trivial way to implement this would be to use HTTPS, with trust
> rooted via ca-certificates. When a user needs the latest key for
> something external, the tooling would simply fetch the latest keys,
> validating it on the fly. Obvious improvements would be certificate
> pinning, signing of data at rest, etc, but these are implementation
> details that are orthogonal to my point. It would still all work with
> only a single root of trust needed in Ubuntu's archive, and just one
> upstream maintainer rather than a bunch of maintainers across a bunch of
> distributions.
>
> The one difference I can see is that it would be necessary to download
> the actual final signing keys, rather than have them available in eg. a
> local mirror. However, if you're bootstrapping something outside of
> Ubuntu, then you have to download that anyway, by definition. Having to
> grab the latest copy of the cross-distro keyring at the same time
> doesn't seem unreasonable. Once done, validation would still be able to
> use the distribution-shipped trust anchor.
>
> If the single root of trust needs updating, then sure. We might even
> need to do it urgently in case of a key revocation. But at least it
> scales, because one can expect it to be infrequent.
>
> > Given all of this, the costs appear minor, especially compared to
> > other updates that are part of point releases. Is there perhaps some
> > angle or detail that I am missing here? I appreciate Robie
> > double-checking that this is the right way to do it.
>
> Compared to regular updates that are part of point releases, this seems
> to be a) new (to us at least, in our stable releases); and b) affects
> roots of trust. I think that warrants special consideration. It would be
> reputationally damaging to Ubuntu if we were to allow changes to these
> keyrings without having thought about how we're validating them, and the
> update path were compromised.
>
> >                                                      I hope I was able
> > to explain that the architecture is the one commonly used and would
> > expect adding a different Ubuntu-only alternative would likely be a
> > costly approach, for little benefit.
>
> I'm not proposing a different Ubuntu-only alternative. I'm proposing
> changes that I think would be improvements applicable to the whole
> ecosystem, as a bar of quality to make this acceptable to Ubuntu to take
> in stable releases rather than de-facto from Debian from autosyncs in
> the development release. Unless I'm missing some use case or nuance that
> necessitates the current architecture that hasn't been pointed out to me
> yet.
>
> >                                      Therefore I wanted to ask if we
> > can agree on this kind of sporadic updates being ok? If it is helpful
> > we could even draft something to be added to
> > https://wiki.ubuntu.com/StableReleaseUpdates#Documentation_for_Special_Cases
> > so anyone involved in the future can refer to it more easily.
>
> Thank you for raising this here, and indeed my main concern is just that
> we're all agreed on the way forward. The appropriate balance is a
> subjective decision. I'd like that to incorporate deliberation on my
> points above, but ultimately I don't have a problem if the decision
> doesn't follow my opinion.
>
> The "Documentation for Special Cases" would be the right place to
> document this, indeed - but only if it's concluded that the SRU process
> is the appropriate one to use here, rather than Backports. I think
> whether -security, -updates or -backports is used is another decision we
> need to make.
>
> I think my question on validation is very important though - whatever we
> do, we should define what privileged review teams are supposed to do to
> validate keyring changes, make sure that we consider that acceptable
> from a trust and security perspective, and do what we can to ensure that
> it's not a burden for the privileged review teams. For example, if I
> could run a command that performs such validation when reviewing an SRU,
> and gives me a simple yes/no answer, then that would be ideal.
>
> In Ubuntu, we de-facto trust Debian to validate regular changes to
> packages, but only for the development release. We don't have any agreed
> or specific arrangement for stable releases.
>
> For new upstreams releases, I prefer to see uscan cryptographically
> validate orig tarballs. For software supply chain reasons, it's on my
> list of things to drive that I'd like to move towards this being
> mandated in Ubuntu as a condition of taking new upstream releases into
> Ubuntu stable releases without individual change review.
>
> I'm not sure that will work here though? The obvious thing would be to
> say that the identical keyring must already be in sid, but if that is
> compromised then Ubuntu will always lose over Debian because a
> compromise to the package in sid would hit Ubuntu users first in
> practice, so maybe that's not good enough.

Hi,

Thank you for sharing your views.

Regarding the alternative proposal, unfortunately there are several
show-stoppers: it essentially boils down to downloading stuff from the
internet at build time, which is exactly what we want to avoid. This
is how the Arch keyring used to work - you needed to build and ship
custom software, and run it at build time, and it would fetch and
prepare keys on the fly. It was an absolute nightmare, it constantly
broke, and made it impossible to support stable updates, and I am very
glad that was changed and now it's just a set of keys like everything
else. Breaking reproducible builds, making it impossible to have
offline builds, making it impossible to take advantage of well known
and well implemented package caching and mirroring, it requires having
to trust the internet CAs, and these are all deal breakers I'm afraid.
If it was acceptable to just download the latest things from the
internet, we'd already do it, but it's exactly what we want to avoid,
and use packaged content instead, so it would be a net downgrade.

Regarding your concerns, I am afraid it is not clear to me why the
current package would be "complicated". In my view on the contrary it
is dead simple, as trivial as it could possibly be. It's two packages
now, but I believe once Arch moves away from contributor keys to
distribution keys it would just use distribution-gpg-keys too, so it
would be down to a single package to maintain and validate. There is
no code being built, no scripts being shipped, no maintainer scripts
running on install/upgrade/removal, no build dependencies, no runtime
dependencies. It's a mere and pedestrian set of ascii text files, that
are fixed and never change - new ones are added, sometimes old ones
are removed. It needs an update twice a year when the new Fedora
release is being prepared. I don't see how it could possibly get any
simpler? There are literally zero moving parts. There is no chance
whatsoever of an update breaking other packages or use cases, barring
revocations (that AFAIK have not happened so far, so it's fair to say
they are rare). It's truly the simplest, barest scheme that could ever
be. And it matches what Ubuntu does with ubuntu-keyring, as far as I
can see.

Regarding the request for validation, could you please clarify exactly
what kind of validation you would like to see? I'm sure we can set up
some autopkgtest for it, which means it would all be automated, and
require no manual work. When ubuntu-keyring 2020.02.11.4 was uploaded
to focal-updates, 2018.09.18.1~18.04.2 to bionic-updates, and
2012.05.19.1 to xenial-updates, what reviews/tests were performed that
made those SRUs acceptable? In the end, as far as I can see, given
it's just a fixed set of text files, the only thing that really
matters is that the source is trusted - and it comes from git, so if
the content matches the upstream tag, which is set by a trusted
source, then that should be all that is necessary? What else would be
needed in your view? This is maintained using git-buildpackage and
merging the upstream tag directly, as it can be easily verified, so it
should be handled already? Is there anything missing from the existing
workflow that you would like to see implemented? Note that manually
signing tarballs with gpg is on the way out, and for good reasons: it
requires manual touches on a developer's machine, and cannot work
fully automated on a trusted, isolated system, and adds content that
is not in git, and as we've seen with xz it's trivially easy to add
content to a release tarball when it doesn't match the git archive.
For example, in systemd we are never going back to manually sign
tarballs, instead we sign git tags, and let Github generate the
tarballs, and the content can be trivially verified to match the
signed tag.