Bug Triage - Friday 4th December

Sat Dec 5 02:32:29 UTC 2020

Status update:

- all recent releases of sssd and adcli have been pulled from -updates and
  -security, and placed back into -proposed.

- I made a debdiff to revert the problematic patches for adcli in Bionic,
  Lukasz has built it in
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4336/+packages

- Currently waiting for adcli - 0.8.2-1ubuntu2 to be bin-synced from the above
  ppa to bionic-proposed for testing.

- We need to release adcli - 0.8.2-1ubuntu2 to -updates and -security after.

- I have written to customers and confirmed the regression to be limited to
  adcli on Bionic, and given them instructions to dowongrade to the version in
  the -release pocket.

Again, I am sorry for causing the regression. On Monday I will begin fixing up
cyrus-sasl2 on Bionic to have a working GSS-SPNEGO implementation.

Thanks,
Matthew

On Sat, Dec 5, 2020 at 12:33 PM Matthew Ruffell
<matthew.ruffell at canonical.com> wrote:
>
> Hi everyone,
>
> Firstly, I deeply apologise for causing the regression.
>
> Even with three separate people testing the test packages and the packages in
> -proposed, the failure still went unnoticed. I should have considered
> the impacts
> of changing the default behaviour of adcli a little more deeply than treating it
> like a normal SRU.
>
> Here are the facts:
>
> The failure is limited to adcli, version 0.8.2-1ubuntu1 on Bionic. At the time
> of writing, it is still in the archive. To archive admins, this needs
> to be pulled.
>
> adcli versions 0.9.0-1ubuntu0.20.04.1 in Focal, 0.9.0-1ubuntu1.2 in Groovy and
> 0.9.0-1ubuntu2 in Hirsute are not affected.
>
> sssd 1.16.1-1ubuntu1.7 in Bionic, and 2.2.3-3ubuntu0.1 in Focal are
> not affected.
>
> Bug Reports:
>
> There are two launchpad bugs open:
>
> LP #1906627 "adcli fails, can't contact LDAP server"
> https://bugs.launchpad.net/ubuntu/+source/adcli/+bug/1906627
>
> LP #1906673 "Realm join hangs"
> https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1906673
>
> Customer Cases:
>
> SF 00298839 "Ubuntu Client Not Joining the Nasdaq AD Domain"
> https://canonical.my.salesforce.com/5004K000003u9EW
>
> SF 00299039 "Regression Issue due to
> https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1906673"
> https://canonical.my.salesforce.com/5004K000003uAkL
>
> Root Cause:
>
> The recent SRU in LP #1868703 "Support "ad_use_ldaps" flag for new AD
> requirements (ADV190023)"
> https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1868703
>
> introduced two changes for adcli on Bionic. The first, was to change from
> GSS-API to GSS-SPNEGO, and the second was to implement support for the flag
> --use-ldaps.
>
> I built a upstream master of adcli, and it still fails on Ubuntu. This indicates
> that the failure is not actually in the adcli package. adcli does not implement
> GSS-SPNEGO, it is linked in from the libsasl2-modules-gssapi-mit package,
> which is a part of cyrus-sasl2.
>
> I built the source of cyrus-sasl2 2.1.27+dfsg-2 from Focal on Bionic, and it
> works with the problematic adcli package.
>
> The root cause is that the implementation of GSS-SPNEGO in cyrus-sasl2 on
> Bionic is broken, and has never worked.
>
> There is more details about commits which the cyrus-sasl2 package in Bionic is
> missing in comment #5 in LP #1906627.
>
> https://bugs.launchpad.net/ubuntu/+source/adcli/+bug/1906627/comments/5
>
> Steps taken yesterday:
>
> I added regression-update to LP #1906627, and I pinged ubuntu-archive in
> #ubuntu-release with these details, but they seem to have been lost in the
> noise.
>
> Located root cause to cryus-sasl2 on Bionic.
>
> Next steps:
>
> We don't need to revert any changes for adcli or sssd on Focal onward.
>
> We don't need to revert any changes on sssd on Bionic.
>
> We need to push a new adcli into Bionic with the recent patches reverted.
>
> We need to fix the GSS-SPNEGO implementation in cyrus-sasl2 in Bionic.
>
> We need to re-release all the SRUs from LP #1868703 after some very thorough
> testing and validation.
>
> Again, I am deeply sorry for causing this regression. I will fix it, starting
> with getting adcli removed from the Bionic archive.
>
> Thanks,
> Matthew
>
> On Fri, Dec 4, 2020 at 10:40 PM Lukasz Zemczak
> <lukasz.zemczak at canonical.com> wrote:
> >
> > Hey!
> >
> > I prefer broken upgrades to get pulled anyway. Besides, packages are
> > updated by unattended-upgrades in up-to 24 hours, so some users might
> > have not gotten it yet. And there's also those not using
> > undattended-upgrades. Let me demote it back to -proposed from -updates
> > as well.
> >
> > On Fri, 4 Dec 2020 at 10:00, Christian Ehrhardt
> > <christian.ehrhardt at canonical.com> wrote:
> > >
> > > On Fri, Dec 4, 2020 at 9:49 AM Lukasz Zemczak
> > > <lukasz.zemczak at canonical.com> wrote:
> > > >
> > > > Hey Christian!
> > > >
> > > > This sounds bad indeed, let's see what Matthew has to say. In the
> > > > meantime I have backed it out from both bionic-security and
> > > > focal-security.
> > >
> > > Thank you
> > >
> > > > Should we also consider dropping it from -updates?
> > >
> > > Well, compared to other cases in this case we don't even yet have a
> > > "ok this is a mess, but this is how you can resolve it afterwards to
> > > work again".
> > > Therefore I think pulling it from -updates as well makes sense until
> > > Matthew had time to look at it in detail and give all-clear (or not).
> > >
> > > P.S.: you slightly raced vorlon who had a different assessment
> > >   [09:30] <vorlon> cpaelzer: well, by this point almost everyone will
> > > have picked it up from security via unattended-upgrades so there's not
> > > much point
> > > But having it pulled for now is on the safe-side and we can re-instate
> > > it at any time once we know more.
> > >
> > > > Cheers,
> > > >
> > > > On Fri, 4 Dec 2020 at 09:01, Christian Ehrhardt
> > > > <christian.ehrhardt at canonical.com> wrote:
> > > > >
> > > > > I was looking at 16 recently touched bugs. Of these a few needed a comment or
> > > > > task update but not a lot of work. Worth to mention are two of them.
> > > > >
> > > > > First we've had "one more" kind of conflicting mysql packages from
> > > > > third party breaking install/upgrade of the one provided by Ubuntu. I
> > > > > dupped it onto bug 1771630 which is our single place to unite all
> > > > > those.
> > > > >
> > > > >
> > > > > A recent sssd update (driven by SEG) seems to have regressed users
> > > > > that now end in a hang.
> > > > > I've pinged on [1], subscribed Matthew (and our Team) on [2]. I've
> > > > > marked it regression-update and also pinged Matthew him via Chat.
> > > > > Furthermore I've set him on CC on this mail.
> > > > > @Matthew - once you've done your initial assessment would you mind
> > > > > replying here with the next steps on this case please?
> > > > > I've marked it prio high, if other triagers see more such reports
> > > > > please mark it even critical then (in that case it is less likely to
> > > > > be just one odd special setup)
> > > > > The release is 21h ago, I'll ping ubuntu-archive (also on CC) if we
> > > > > should - for now until clarified by Matthew - remove it from
> > > > > -security.
> > > > >
> > > > >
> > > > > [1]: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1868703/comments/86
> > > > > [2]: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1906673
> > > > >
> > > > >
> > > > > --
> > > > > Christian Ehrhardt
> > > > > Staff Engineer, Ubuntu Server
> > > > > Canonical Ltd
> > > > >
> > > > > --
> > > > > ubuntu-archive mailing list
> > > > > ubuntu-archive at lists.ubuntu.com
> > > > > https://lists.ubuntu.com/mailman/listinfo/ubuntu-archive
> > > >
> > > >
> > > >
> > > > --
> > > > Łukasz 'sil2100' Zemczak
> > > >  Foundations Team
> > > >  lukasz.zemczak at canonical.com
> > > >  www.canonical.com
> > >
> > >
> > >
> > > --
> > > Christian Ehrhardt
> > > Staff Engineer, Ubuntu Server
> > > Canonical Ltd
> >
> >
> >
> > --
> > Łukasz 'sil2100' Zemczak
> >  Foundations Team
> >  lukasz.zemczak at canonical.com
> >  www.canonical.com