Bug Triage - Friday 4th December

Fri Dec 4 23:33:06 UTC 2020

Hi everyone,

Firstly, I deeply apologise for causing the regression.

Even with three separate people testing the test packages and the packages in
-proposed, the failure still went unnoticed. I should have considered
the impacts
of changing the default behaviour of adcli a little more deeply than treating it
like a normal SRU.

Here are the facts:

The failure is limited to adcli, version 0.8.2-1ubuntu1 on Bionic. At the time
of writing, it is still in the archive. To archive admins, this needs
to be pulled.

adcli versions 0.9.0-1ubuntu0.20.04.1 in Focal, 0.9.0-1ubuntu1.2 in Groovy and
0.9.0-1ubuntu2 in Hirsute are not affected.

sssd 1.16.1-1ubuntu1.7 in Bionic, and 2.2.3-3ubuntu0.1 in Focal are
not affected.

Bug Reports:

There are two launchpad bugs open:

LP #1906627 "adcli fails, can't contact LDAP server"
https://bugs.launchpad.net/ubuntu/+source/adcli/+bug/1906627

LP #1906673 "Realm join hangs"
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1906673

Customer Cases:

SF 00298839 "Ubuntu Client Not Joining the Nasdaq AD Domain"
https://canonical.my.salesforce.com/5004K000003u9EW

SF 00299039 "Regression Issue due to
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1906673"
https://canonical.my.salesforce.com/5004K000003uAkL

Root Cause:

The recent SRU in LP #1868703 "Support "ad_use_ldaps" flag for new AD
requirements (ADV190023)"
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1868703

introduced two changes for adcli on Bionic. The first, was to change from
GSS-API to GSS-SPNEGO, and the second was to implement support for the flag
--use-ldaps.

I built a upstream master of adcli, and it still fails on Ubuntu. This indicates
that the failure is not actually in the adcli package. adcli does not implement
GSS-SPNEGO, it is linked in from the libsasl2-modules-gssapi-mit package,
which is a part of cyrus-sasl2.

I built the source of cyrus-sasl2 2.1.27+dfsg-2 from Focal on Bionic, and it
works with the problematic adcli package.

The root cause is that the implementation of GSS-SPNEGO in cyrus-sasl2 on
Bionic is broken, and has never worked.

There is more details about commits which the cyrus-sasl2 package in Bionic is
missing in comment #5 in LP #1906627.

https://bugs.launchpad.net/ubuntu/+source/adcli/+bug/1906627/comments/5

Steps taken yesterday:

I added regression-update to LP #1906627, and I pinged ubuntu-archive in
#ubuntu-release with these details, but they seem to have been lost in the
noise.

Located root cause to cryus-sasl2 on Bionic.

Next steps:

We don't need to revert any changes for adcli or sssd on Focal onward.

We don't need to revert any changes on sssd on Bionic.

We need to push a new adcli into Bionic with the recent patches reverted.

We need to fix the GSS-SPNEGO implementation in cyrus-sasl2 in Bionic.

We need to re-release all the SRUs from LP #1868703 after some very thorough
testing and validation.

Again, I am deeply sorry for causing this regression. I will fix it, starting
with getting adcli removed from the Bionic archive.

Thanks,
Matthew

On Fri, Dec 4, 2020 at 10:40 PM Lukasz Zemczak
<lukasz.zemczak at canonical.com> wrote:
>
> Hey!
>
> I prefer broken upgrades to get pulled anyway. Besides, packages are
> updated by unattended-upgrades in up-to 24 hours, so some users might
> have not gotten it yet. And there's also those not using
> undattended-upgrades. Let me demote it back to -proposed from -updates
> as well.
>
> On Fri, 4 Dec 2020 at 10:00, Christian Ehrhardt
> <christian.ehrhardt at canonical.com> wrote:
> >
> > On Fri, Dec 4, 2020 at 9:49 AM Lukasz Zemczak
> > <lukasz.zemczak at canonical.com> wrote:
> > >
> > > Hey Christian!
> > >
> > > This sounds bad indeed, let's see what Matthew has to say. In the
> > > meantime I have backed it out from both bionic-security and
> > > focal-security.
> >
> > Thank you
> >
> > > Should we also consider dropping it from -updates?
> >
> > Well, compared to other cases in this case we don't even yet have a
> > "ok this is a mess, but this is how you can resolve it afterwards to
> > work again".
> > Therefore I think pulling it from -updates as well makes sense until
> > Matthew had time to look at it in detail and give all-clear (or not).
> >
> > P.S.: you slightly raced vorlon who had a different assessment
> >   [09:30] <vorlon> cpaelzer: well, by this point almost everyone will
> > have picked it up from security via unattended-upgrades so there's not
> > much point
> > But having it pulled for now is on the safe-side and we can re-instate
> > it at any time once we know more.
> >
> > > Cheers,
> > >
> > > On Fri, 4 Dec 2020 at 09:01, Christian Ehrhardt
> > > <christian.ehrhardt at canonical.com> wrote:
> > > >
> > > > I was looking at 16 recently touched bugs. Of these a few needed a comment or
> > > > task update but not a lot of work. Worth to mention are two of them.
> > > >
> > > > First we've had "one more" kind of conflicting mysql packages from
> > > > third party breaking install/upgrade of the one provided by Ubuntu. I
> > > > dupped it onto bug 1771630 which is our single place to unite all
> > > > those.
> > > >
> > > >
> > > > A recent sssd update (driven by SEG) seems to have regressed users
> > > > that now end in a hang.
> > > > I've pinged on [1], subscribed Matthew (and our Team) on [2]. I've
> > > > marked it regression-update and also pinged Matthew him via Chat.
> > > > Furthermore I've set him on CC on this mail.
> > > > @Matthew - once you've done your initial assessment would you mind
> > > > replying here with the next steps on this case please?
> > > > I've marked it prio high, if other triagers see more such reports
> > > > please mark it even critical then (in that case it is less likely to
> > > > be just one odd special setup)
> > > > The release is 21h ago, I'll ping ubuntu-archive (also on CC) if we
> > > > should - for now until clarified by Matthew - remove it from
> > > > -security.
> > > >
> > > >
> > > > [1]: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1868703/comments/86
> > > > [2]: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1906673
> > > >
> > > >
> > > > --
> > > > Christian Ehrhardt
> > > > Staff Engineer, Ubuntu Server
> > > > Canonical Ltd
> > > >
> > > > --
> > > > ubuntu-archive mailing list
> > > > ubuntu-archive at lists.ubuntu.com
> > > > https://lists.ubuntu.com/mailman/listinfo/ubuntu-archive
> > >
> > >
> > >
> > > --
> > > Łukasz 'sil2100' Zemczak
> > >  Foundations Team
> > >  lukasz.zemczak at canonical.com
> > >  www.canonical.com
> >
> >
> >
> > --
> > Christian Ehrhardt
> > Staff Engineer, Ubuntu Server
> > Canonical Ltd
>
>
>
> --
> Łukasz 'sil2100' Zemczak
>  Foundations Team
>  lukasz.zemczak at canonical.com
>  www.canonical.com