Bug Triage - Friday 4th December

[Matthew Ruffell]

Status update:

- There is a new build of adcli, version 0.8.2-1ubuntu2 which reverts the
  patches introduced in the previous build, on the -unapproved queue in
  -proposed. This is likely to be released to fix anyone using the faulty
  0.8.2-1ubuntu1 package.
- As mentioned in previous messages, I have determined the root cause of the
  failure to be an incompatible implementation of GSS-SPNEGO in cyrus-sasl2,
  and I have created a debdiff which fixes the problem [1].
- I have added a SRU template for cyrus-sasl2 in [2], and asked for the changes
  to be sponsored and placed into -proposed.

This regression will be resolved when either the cyrus-sasl2 fixes have made
their way to -updates, likely in a week's time, or when the adcli package with
the reverted patches is released.

Once the fixed cyrus-sasl2 is released, we will re-perform verification on the
changes to adcli and sssd in LP #1868703, and hopefully go for release again.

Again, I apologise for the regression, and things are on their way to being
fixed.

Thanks,
Matthew

[1] https://bugs.launchpad.net/ubuntu/+source/adcli/+bug/1906627/+attachment/5441530/+files/lp1906627_cyrus_sasl2_bionic.debdiff
[2] https://bugs.launchpad.net/ubuntu/+source/adcli/+bug/1906627

On Sat, Dec 5, 2020 at 3:32 PM Matthew Ruffell
<matthew.ruffell at canonical.com> wrote:
>
> Status update:
>
> - all recent releases of sssd and adcli have been pulled from -updates and
>   -security, and placed back into -proposed.
>
> - I made a debdiff to revert the problematic patches for adcli in Bionic,
>   Lukasz has built it in
> https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4336/+packages
>
> - Currently waiting for adcli - 0.8.2-1ubuntu2 to be bin-synced from the above
>   ppa to bionic-proposed for testing.
>
> - We need to release adcli - 0.8.2-1ubuntu2 to -updates and -security after.
>
> - I have written to customers and confirmed the regression to be limited to
>   adcli on Bionic, and given them instructions to dowongrade to the version in
>   the -release pocket.
>
> Again, I am sorry for causing the regression. On Monday I will begin fixing up
> cyrus-sasl2 on Bionic to have a working GSS-SPNEGO implementation.
>
> Thanks,
> Matthew
>
> On Sat, Dec 5, 2020 at 12:33 PM Matthew Ruffell
> <matthew.ruffell at canonical.com> wrote:
> >
> > Hi everyone,
> >
> > Firstly, I deeply apologise for causing the regression.
> >
> > Even with three separate people testing the test packages and the packages in
> > -proposed, the failure still went unnoticed. I should have considered
> > the impacts
> > of changing the default behaviour of adcli a little more deeply than treating it
> > like a normal SRU.
> >
> > Here are the facts:
> >
> > The failure is limited to adcli, version 0.8.2-1ubuntu1 on Bionic. At the time
> > of writing, it is still in the archive. To archive admins, this needs
> > to be pulled.
> >
> > adcli versions 0.9.0-1ubuntu0.20.04.1 in Focal, 0.9.0-1ubuntu1.2 in Groovy and
> > 0.9.0-1ubuntu2 in Hirsute are not affected.
> >
> > sssd 1.16.1-1ubuntu1.7 in Bionic, and 2.2.3-3ubuntu0.1 in Focal are
> > not affected.
> >
> > Bug Reports:
> >
> > There are two launchpad bugs open:
> >
> > LP #1906627 "adcli fails, can't contact LDAP server"
> > https://bugs.launchpad.net/ubuntu/+source/adcli/+bug/1906627
> >
> > LP #1906673 "Realm join hangs"
> > https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1906673
> >
> > Customer Cases:
> >
> > SF 00298839 "Ubuntu Client Not Joining the Nasdaq AD Domain"
> > https://canonical.my.salesforce.com/5004K000003u9EW
> >
> > SF 00299039 "Regression Issue due to
> > https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1906673"
> > https://canonical.my.salesforce.com/5004K000003uAkL
> >
> > Root Cause:
> >
> > The recent SRU in LP #1868703 "Support "ad_use_ldaps" flag for new AD
> > requirements (ADV190023)"
> > https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1868703
> >
> > introduced two changes for adcli on Bionic. The first, was to change from
> > GSS-API to GSS-SPNEGO, and the second was to implement support for the flag
> > --use-ldaps.
> >
> > I built a upstream master of adcli, and it still fails on Ubuntu. This indicates
> > that the failure is not actually in the adcli package. adcli does not implement
> > GSS-SPNEGO, it is linked in from the libsasl2-modules-gssapi-mit package,
> > which is a part of cyrus-sasl2.
> >
> > I built the source of cyrus-sasl2 2.1.27+dfsg-2 from Focal on Bionic, and it
> > works with the problematic adcli package.
> >
> > The root cause is that the implementation of GSS-SPNEGO in cyrus-sasl2 on
> > Bionic is broken, and has never worked.
> >
> > There is more details about commits which the cyrus-sasl2 package in Bionic is
> > missing in comment #5 in LP #1906627.
> >
> > https://bugs.launchpad.net/ubuntu/+source/adcli/+bug/1906627/comments/5
> >
> > Steps taken yesterday:
> >
> > I added regression-update to LP #1906627, and I pinged ubuntu-archive in
> > #ubuntu-release with these details, but they seem to have been lost in the
> > noise.
> >
> > Located root cause to cryus-sasl2 on Bionic.
> >
> > Next steps:
> >
> > We don't need to revert any changes for adcli or sssd on Focal onward.
> >
> > We don't need to revert any changes on sssd on Bionic.
> >
> > We need to push a new adcli into Bionic with the recent patches reverted.
> >
> > We need to fix the GSS-SPNEGO implementation in cyrus-sasl2 in Bionic.
> >
> > We need to re-release all the SRUs from LP #1868703 after some very thorough
> > testing and validation.
> >
> > Again, I am deeply sorry for causing this regression. I will fix it, starting
> > with getting adcli removed from the Bionic archive.
> >
> > Thanks,
> > Matthew
> >
> > On Fri, Dec 4, 2020 at 10:40 PM Lukasz Zemczak
> > <lukasz.zemczak at canonical.com> wrote:
> > >
> > > Hey!
> > >
> > > I prefer broken upgrades to get pulled anyway. Besides, packages are
> > > updated by unattended-upgrades in up-to 24 hours, so some users might
> > > have not gotten it yet. And there's also those not using
> > > undattended-upgrades. Let me demote it back to -proposed from -updates
> > > as well.
> > >
> > > On Fri, 4 Dec 2020 at 10:00, Christian Ehrhardt
> > > <christian.ehrhardt at canonical.com> wrote:
> > > >
> > > > On Fri, Dec 4, 2020 at 9:49 AM Lukasz Zemczak
> > > > <lukasz.zemczak at canonical.com> wrote:
> > > > >
> > > > > Hey Christian!
> > > > >
> > > > > This sounds bad indeed, let's see what Matthew has to say. In the
> > > > > meantime I have backed it out from both bionic-security and
> > > > > focal-security.
> > > >
> > > > Thank you
> > > >
> > > > > Should we also consider dropping it from -updates?
> > > >
> > > > Well, compared to other cases in this case we don't even yet have a
> > > > "ok this is a mess, but this is how you can resolve it afterwards to
> > > > work again".
> > > > Therefore I think pulling it from -updates as well makes sense until
> > > > Matthew had time to look at it in detail and give all-clear (or not).
> > > >
> > > > P.S.: you slightly raced vorlon who had a different assessment
> > > >   [09:30] <vorlon> cpaelzer: well, by this point almost everyone will
> > > > have picked it up from security via unattended-upgrades so there's not
> > > > much point
> > > > But having it pulled for now is on the safe-side and we can re-instate
> > > > it at any time once we know more.
> > > >
> > > > > Cheers,
> > > > >
> > > > > On Fri, 4 Dec 2020 at 09:01, Christian Ehrhardt
> > > > > <christian.ehrhardt at canonical.com> wrote:
> > > > > >
> > > > > > I was looking at 16 recently touched bugs. Of these a few needed a comment or
> > > > > > task update but not a lot of work. Worth to mention are two of them.
> > > > > >
> > > > > > First we've had "one more" kind of conflicting mysql packages from
> > > > > > third party breaking install/upgrade of the one provided by Ubuntu. I
> > > > > > dupped it onto bug 1771630 which is our single place to unite all
> > > > > > those.
> > > > > >
> > > > > >
> > > > > > A recent sssd update (driven by SEG) seems to have regressed users
> > > > > > that now end in a hang.
> > > > > > I've pinged on [1], subscribed Matthew (and our Team) on [2]. I've
> > > > > > marked it regression-update and also pinged Matthew him via Chat.
> > > > > > Furthermore I've set him on CC on this mail.
> > > > > > @Matthew - once you've done your initial assessment would you mind
> > > > > > replying here with the next steps on this case please?
> > > > > > I've marked it prio high, if other triagers see more such reports
> > > > > > please mark it even critical then (in that case it is less likely to
> > > > > > be just one odd special setup)
> > > > > > The release is 21h ago, I'll ping ubuntu-archive (also on CC) if we
> > > > > > should - for now until clarified by Matthew - remove it from
> > > > > > -security.
> > > > > >
> > > > > >
> > > > > > [1]: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1868703/comments/86
> > > > > > [2]: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1906673
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Christian Ehrhardt
> > > > > > Staff Engineer, Ubuntu Server
> > > > > > Canonical Ltd
> > > > > >
> > > > > > --
> > > > > > ubuntu-archive mailing list
> > > > > > ubuntu-archive at lists.ubuntu.com
> > > > > > https://lists.ubuntu.com/mailman/listinfo/ubuntu-archive
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Łukasz 'sil2100' Zemczak
> > > > >  Foundations Team
> > > > >  lukasz.zemczak at canonical.com
> > > > >  www.canonical.com
> > > >
> > > >
> > > >
> > > > --
> > > > Christian Ehrhardt
> > > > Staff Engineer, Ubuntu Server
> > > > Canonical Ltd
> > >
> > >
> > >
> > > --
> > > Łukasz 'sil2100' Zemczak
> > >  Foundations Team
> > >  lukasz.zemczak at canonical.com
> > >  www.canonical.com