+1 maintenance report (dogtag-pki vs 389-ds-base)

[Sergio Durigan Junior]

On Friday, January 22 2021, Brian Murray wrote:

> On Thu, Jan 21, 2021 at 03:25:02PM -0500, Sergio Durigan Junior wrote:
>> On Thursday, January 21 2021, Timo Aaltonen wrote:
>> 
>> > On 21.1.2021 18.59, Lukas Märdian wrote:
>> >> NO - dogtag-pki vs ['389-ds-base/1.4.4.9-1build2',
>> >> 'net-snmp/5.9+dfsg-3ubuntu1']
>> >>
>> >> So I had a closer look into the dogtag-pki failure on s390x. I could
>> >> easily reproduce the problem inside a s390x LXD container, but
>> >> wasn't able to isolate the root cause... After quite some
>> >> investigation I was able to produce a debug trace of the problem,
>> >> and to me it looks like the issue is actually not inside this
>> >> package, but rather inside the LDAP server (i.e. 389-ds-base), as
>> >> the debug log shows an "SEVERE: Unable to modify o=pki-tomcat-CA:
>> >> netscape.ldap.LDAPException: error result (1); Operations error",
>> >> i.e. "Internal Server Error"
>> >> (https://docs.oracle.com/cd/E19957-01/816-5618-10/netscape/ldap/LDAPException.html#OPERATION_ERROR
>> >> <https://docs.oracle.com/cd/E19957-01/816-5618-10/netscape/ldap/LDAPException.html#OPERATION_ERROR>). I
>> >> do not really understand why the LDAP server would behave
>> >> differently on s390x than on all the other architectures, but I
>> >> guess this is for another time...
>> >>
>> >> Debug log: https://paste.ubuntu.com/p/vx9JB6VTjF/
>> >> <https://paste.ubuntu.com/p/vx9JB6VTjF/>
>> >
>> > This seems to go back to at least 389-ds-base 1.4.4.4, probably
>> > longer. Would be useful to know where it regressed.
>> >
>> > As to why it only happens on s390x my guess would be that it's related
>> > to endianness (it's big-endian). Upstream tests only on amd64 (LE).
>> 
>> I'm also very interested in solving this, because it's blocking net-snmp
>> and a bunch of other packages from migrating.
>> 
>> Last week I did some investigation and pretty much stopped at the same
>> point as Lukas did.  I wasn't able to pinpoint exactly what the root
>> cause is, but Timo's guess is a good starting point.
>> 
>> I fiddled a bit with autopkgtest.db and confirmed that the failures
>> started with 389-ds-base/1.4.4.4-1:
>> 
>>   sqlite> SELECT test.package, result.version, result.triggers,
>> result.exitcode FROM result INNER JOIN test ON test.id =
>> result.test_id where test.package = 'dogtag-pki' and result.triggers
>> LIKE '%389-ds-base%';
>
> I find your querying of the SQLite database pretty interesting. Is there
> any documentation about the structure of the database and queries that
> might be useful?

Hey Brian,

Not that I know of.  The database is pretty simple, though; you can
obtain the schema for the tables by doing:

  PRAGMA TABLE_info(test);
  PRAGMA TABLE_info(result);

>From there, you can start playing with queries.  I specifically like to
use the database at the end of a +1 maintenance week, because it's
pretty convenient to let you know what you did.

Cheers,

-- 
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0  EB2F 106D A1C8 C3CB BF14