[Bug 1931104] [NEW] Test of dogtag-pki is failing on s390x vs the nss v3.63 in impish-proposed

Christian Ehrhardt  1931104 at bugs.launchpad.net
Mon Jun 7 11:07:55 UTC 2021


Public bug reported:

The test of dogtag-pki is failing on the nss 3.63 that is in impish proposed.
Example:
https://autopkgtest.ubuntu.com/results/autopkgtest-impish/impish/s390x/d/dogtag-pki/20210516_212719_e6522@/log.gz

Bad:
Installing CA into /var/lib/pki/pki-tomcat.
Installation failed: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
ERROR: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  File "/usr/lib/python3/dist-packages/pki/server/pkispawn.py", line 575, in main
    scriptlet.spawn(deployer)
  File "/usr/lib/python3/dist-packages/pki/server/deployment/scriptlets/configuration.py", line 995, in spawn
    cert = deployer.setup_cert(client, tag)
  File "/usr/lib/python3/dist-packages/pki/server/deployment/__init__.py", line 355, in setup_cert
    return client.setupCert(request)
  File "/usr/lib/python3/dist-packages/pki/system.py", line 389, in setupCert
    response = self.connection.post(
  File "/usr/lib/python3/dist-packages/pki/client.py", line 55, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/pki/client.py", line 293, in post
    r = self.session.post(
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
>>>> CA spawn failed:

Good:
nstalling CA into /var/lib/pki/pki-tomcat.
Notice: Trust flag u is set automatically if the private key is present.
/usr/lib/python3/dist-packages/urllib3/connection.py:455: SubjectAltNameWarning: Certificate for i-dogtag has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/urllib3/urllib3/issues/497 for details.)
  warnings.warn(

    ==========================================================================
                                INSTALLATION SUMMARY
    ==========================================================================
...

The good test above was with:
ii  libnss3:s390x        2:3.61-1ubuntu2  s390x        Network Security Service libraries
ii  389-ds-base    1.4.4.11-2      s390x        389 Directory Server suite - server

Worth to know, the good case test still fails later on with:
IOException: SocketException cannot write on socket: Failed to write to socket: (-5938) Encountered end of file.
ERROR: CalledProcessError: Command '['pki', '-d', '/etc/pki/pki-tomcat/alias', '-f', '/etc/pki/pki-tomcat/password.conf', '-U', 'https://i-dogtag:8443', 'securitydomain-join', '--session', '4717921475119312283', '--type', 'TKS', '--hostname', 'i-dogtag', '--unsecure-port', '8080', '--secure-port', '8443', 'TKS i-dogtag 8443']' returned non-zero exit status 255.
  File "/usr/lib/python3/dist-packages/pki/server/pkispawn.py", line 575, in main
    scriptlet.spawn(deployer)
  File "/usr/lib/python3/dist-packages/pki/server/deployment/scriptlets/configuration.py", line 1038, in spawn
    subsystem.join_security_domain(
  File "/usr/lib/python3/dist-packages/pki/server/subsystem.py", line 1201, in join_security_domain
    subprocess.check_call(cmd)
  File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
Installation failed: Command failed: pki -d /etc/pki/pki-tomcat/alias -f /etc/pki/pki-tomcat/password.conf -U https://i-dogtag:8443 securitydomain-join --session 4717921475119312283 --type TKS --hostname i-dogtag --unsecure-port 8080 --secure-port 8443 TKS i-dogtag 8443
Please check pkispawn logs in /var/log/pki/pki-tks-spawn.20210607093926.log

Well one issue at a time ... the current install issue first.

Since it worked with the nss in -release I was upgrading this to the new nss.
ii  389-ds-base    1.4.4.11-2      s390x        389 Directory Server suite - server
ii  libnss3:s390x  2:3.63-1ubuntu1 s390x        Network Security Service libraries

With this the install fail is reprodicible.
So we can switch in/out bad case by up/downgrading libnss3.

Comparing those two cases until they reach the first successful install message
I've seen a crash:

  pki-tomcat[37160]: #
  pki-tomcat[37160]: # A fatal error has been detected by the Java Runtime Environment:
  pki-tomcat[37160]: #
  pki-tomcat[37160]: #  SIGSEGV (0xb) at pc=0x000003ff9ce9ec02, pid=37160, tid=37246
  pki-tomcat[37160]: #
  pki-tomcat[37160]: # JRE version: OpenJDK Runtime Environment (11.0.12+4) (build 11.0.12-ea+4-Ubuntu-0ubuntu2)
  pki-tomcat[37160]: # Java VM: OpenJDK 64-Bit Server VM (11.0.12-ea+4-Ubuntu-0ubuntu2, mixed mode, tiered, compressed oops, serial gc, linux-s390x)
  pki-tomcat[37160]: # Problematic frame:
  pki-tomcat[37160]: # C  [libnss3.so+0x11ec02]
  pki-tomcat[37160]: #
  pki-tomcat[37160]: # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /var/lib/pki/pki-tomcat/core.37160)
  pki-tomcat[37160]: #
  pki-tomcat[37160]: # An error report file with more information is saved as:
  pki-tomcat[37160]: # /var/lib/pki/pki-tomcat/hs_err_pid37160.log
  pki-tomcat[37160]: #
  pki-tomcat[37160]: # If you would like to submit a bug report, please visit:
  pki-tomcat[37160]: #   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
  pki-tomcat[37160]: # The crash happened outside the Java Virtual Machine in native code.
  pki-tomcat[37160]: # See problematic frame for where to report the bug.

A few extra runs had also shown:
   # Problematic frame:
   # C  [libnssutil3.so+0x1b60c]  PORT_FreeArena_Util+0xc

And while I could not get a core dump out as the config required to be changed
is written on the fly and then started I was able to find the code.
Obviously there has to be a lot of abstraction but plenty of recent changes
fixed double frees and dangling pointer values.
For example https://github.com/nss-dev/nss/commit/350807b3a70f60928ea3f2bc95fd1795aae9b753

This is all (this and more similar fixes) in 3.66 which is released and in Debian unstable.
It might be worth to re-merge that, throw it into a PPA and re-run the tests.

** Affects: nss (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: update-excuse

** Description changed:

- The test of dogtag-pki is failing on the nss 3.63 that is in impish
- proposed.
- 
+ The test of dogtag-pki is failing on the nss 3.63 that is in impish proposed.
+ Example:
+ https://autopkgtest.ubuntu.com/results/autopkgtest-impish/impish/s390x/d/dogtag-pki/20210516_212719_e6522@/log.gz
  
  Bad:
  Installing CA into /var/lib/pki/pki-tomcat.
  Installation failed: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  ERROR: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
-   File "/usr/lib/python3/dist-packages/pki/server/pkispawn.py", line 575, in main
-     scriptlet.spawn(deployer)
-   File "/usr/lib/python3/dist-packages/pki/server/deployment/scriptlets/configuration.py", line 995, in spawn
-     cert = deployer.setup_cert(client, tag)
-   File "/usr/lib/python3/dist-packages/pki/server/deployment/__init__.py", line 355, in setup_cert
-     return client.setupCert(request)
-   File "/usr/lib/python3/dist-packages/pki/system.py", line 389, in setupCert
-     response = self.connection.post(
-   File "/usr/lib/python3/dist-packages/pki/client.py", line 55, in wrapper
-     return func(self, *args, **kwargs)
-   File "/usr/lib/python3/dist-packages/pki/client.py", line 293, in post
-     r = self.session.post(
-   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590, in post
-     return self.request('POST', url, data=data, json=json, **kwargs)
-   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
-     resp = self.send(prep, **send_kwargs)
-   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
-     r = adapter.send(request, **kwargs)
-   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
-     raise ConnectionError(err, request=request)
+   File "/usr/lib/python3/dist-packages/pki/server/pkispawn.py", line 575, in main
+     scriptlet.spawn(deployer)
+   File "/usr/lib/python3/dist-packages/pki/server/deployment/scriptlets/configuration.py", line 995, in spawn
+     cert = deployer.setup_cert(client, tag)
+   File "/usr/lib/python3/dist-packages/pki/server/deployment/__init__.py", line 355, in setup_cert
+     return client.setupCert(request)
+   File "/usr/lib/python3/dist-packages/pki/system.py", line 389, in setupCert
+     response = self.connection.post(
+   File "/usr/lib/python3/dist-packages/pki/client.py", line 55, in wrapper
+     return func(self, *args, **kwargs)
+   File "/usr/lib/python3/dist-packages/pki/client.py", line 293, in post
+     r = self.session.post(
+   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590, in post
+     return self.request('POST', url, data=data, json=json, **kwargs)
+   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
+     resp = self.send(prep, **send_kwargs)
+   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
+     r = adapter.send(request, **kwargs)
+   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
+     raise ConnectionError(err, request=request)
  >>>> CA spawn failed:
  
  Good:
  nstalling CA into /var/lib/pki/pki-tomcat.
  Notice: Trust flag u is set automatically if the private key is present.
  /usr/lib/python3/dist-packages/urllib3/connection.py:455: SubjectAltNameWarning: Certificate for i-dogtag has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/urllib3/urllib3/issues/497 for details.)
-   warnings.warn(
+   warnings.warn(
  
-     ==========================================================================
-                                 INSTALLATION SUMMARY
-     ==========================================================================
+     ==========================================================================
+                                 INSTALLATION SUMMARY
+     ==========================================================================
  ...
- 
  
  The good test above was with:
  ii  libnss3:s390x        2:3.61-1ubuntu2  s390x        Network Security Service libraries
  ii  389-ds-base    1.4.4.11-2      s390x        389 Directory Server suite - server
  
- 
  Worth to know, the good case test still fails later on with:
  IOException: SocketException cannot write on socket: Failed to write to socket: (-5938) Encountered end of file.
  ERROR: CalledProcessError: Command '['pki', '-d', '/etc/pki/pki-tomcat/alias', '-f', '/etc/pki/pki-tomcat/password.conf', '-U', 'https://i-dogtag:8443', 'securitydomain-join', '--session', '4717921475119312283', '--type', 'TKS', '--hostname', 'i-dogtag', '--unsecure-port', '8080', '--secure-port', '8443', 'TKS i-dogtag 8443']' returned non-zero exit status 255.
-   File "/usr/lib/python3/dist-packages/pki/server/pkispawn.py", line 575, in main
-     scriptlet.spawn(deployer)
-   File "/usr/lib/python3/dist-packages/pki/server/deployment/scriptlets/configuration.py", line 1038, in spawn
-     subsystem.join_security_domain(
-   File "/usr/lib/python3/dist-packages/pki/server/subsystem.py", line 1201, in join_security_domain
-     subprocess.check_call(cmd)
-   File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
-     raise CalledProcessError(retcode, cmd)
+   File "/usr/lib/python3/dist-packages/pki/server/pkispawn.py", line 575, in main
+     scriptlet.spawn(deployer)
+   File "/usr/lib/python3/dist-packages/pki/server/deployment/scriptlets/configuration.py", line 1038, in spawn
+     subsystem.join_security_domain(
+   File "/usr/lib/python3/dist-packages/pki/server/subsystem.py", line 1201, in join_security_domain
+     subprocess.check_call(cmd)
+   File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
+     raise CalledProcessError(retcode, cmd)
  Installation failed: Command failed: pki -d /etc/pki/pki-tomcat/alias -f /etc/pki/pki-tomcat/password.conf -U https://i-dogtag:8443 securitydomain-join --session 4717921475119312283 --type TKS --hostname i-dogtag --unsecure-port 8080 --secure-port 8443 TKS i-dogtag 8443
  Please check pkispawn logs in /var/log/pki/pki-tks-spawn.20210607093926.log
- 
  
  Well one issue at a time ... the current install issue first.
  
  Since it worked with the nss in -release I was upgrading this to the new nss.
  ii  389-ds-base    1.4.4.11-2      s390x        389 Directory Server suite - server
  ii  libnss3:s390x  2:3.63-1ubuntu1 s390x        Network Security Service libraries
  
  With this the install fail is reprodicible.
  So we can switch in/out bad case by up/downgrading libnss3.
  
  Comparing those two cases until they reach the first successful install message
  I've seen a crash:
  
-   pki-tomcat[37160]: #
-   pki-tomcat[37160]: # A fatal error has been detected by the Java Runtime Environment:
-   pki-tomcat[37160]: #
-   pki-tomcat[37160]: #  SIGSEGV (0xb) at pc=0x000003ff9ce9ec02, pid=37160, tid=37246
-   pki-tomcat[37160]: #
-   pki-tomcat[37160]: # JRE version: OpenJDK Runtime Environment (11.0.12+4) (build 11.0.12-ea+4-Ubuntu-0ubuntu2)
-   pki-tomcat[37160]: # Java VM: OpenJDK 64-Bit Server VM (11.0.12-ea+4-Ubuntu-0ubuntu2, mixed mode, tiered, compressed oops, serial gc, linux-s390x)
-   pki-tomcat[37160]: # Problematic frame:
-   pki-tomcat[37160]: # C  [libnss3.so+0x11ec02]
-   pki-tomcat[37160]: #
-   pki-tomcat[37160]: # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /var/lib/pki/pki-tomcat/core.37160)
-   pki-tomcat[37160]: #
-   pki-tomcat[37160]: # An error report file with more information is saved as:
-   pki-tomcat[37160]: # /var/lib/pki/pki-tomcat/hs_err_pid37160.log
-   pki-tomcat[37160]: #
-   pki-tomcat[37160]: # If you would like to submit a bug report, please visit:
-   pki-tomcat[37160]: #   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
-   pki-tomcat[37160]: # The crash happened outside the Java Virtual Machine in native code.
-   pki-tomcat[37160]: # See problematic frame for where to report the bug.
+   pki-tomcat[37160]: #
+   pki-tomcat[37160]: # A fatal error has been detected by the Java Runtime Environment:
+   pki-tomcat[37160]: #
+   pki-tomcat[37160]: #  SIGSEGV (0xb) at pc=0x000003ff9ce9ec02, pid=37160, tid=37246
+   pki-tomcat[37160]: #
+   pki-tomcat[37160]: # JRE version: OpenJDK Runtime Environment (11.0.12+4) (build 11.0.12-ea+4-Ubuntu-0ubuntu2)
+   pki-tomcat[37160]: # Java VM: OpenJDK 64-Bit Server VM (11.0.12-ea+4-Ubuntu-0ubuntu2, mixed mode, tiered, compressed oops, serial gc, linux-s390x)
+   pki-tomcat[37160]: # Problematic frame:
+   pki-tomcat[37160]: # C  [libnss3.so+0x11ec02]
+   pki-tomcat[37160]: #
+   pki-tomcat[37160]: # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /var/lib/pki/pki-tomcat/core.37160)
+   pki-tomcat[37160]: #
+   pki-tomcat[37160]: # An error report file with more information is saved as:
+   pki-tomcat[37160]: # /var/lib/pki/pki-tomcat/hs_err_pid37160.log
+   pki-tomcat[37160]: #
+   pki-tomcat[37160]: # If you would like to submit a bug report, please visit:
+   pki-tomcat[37160]: #   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
+   pki-tomcat[37160]: # The crash happened outside the Java Virtual Machine in native code.
+   pki-tomcat[37160]: # See problematic frame for where to report the bug.
  
  A few extra runs had also shown:
-    # Problematic frame:
-    # C  [libnssutil3.so+0x1b60c]  PORT_FreeArena_Util+0xc
+    # Problematic frame:
+    # C  [libnssutil3.so+0x1b60c]  PORT_FreeArena_Util+0xc
  
  And while I could not get a core dump out as the config required to be changed
  is written on the fly and then started I was able to find the code.
  Obviously there has to be a lot of abstraction but plenty of recent changes
  fixed double frees and dangling pointer values.
  For example https://github.com/nss-dev/nss/commit/350807b3a70f60928ea3f2bc95fd1795aae9b753
  
  This is all (this and more similar fixes) in 3.66 which is released and in Debian unstable.
  It might be worth to re-merge that, throw it into a PPA and re-run the tests.

** Tags added: update-excuse

-- 
You received this bug notification because you are a member of Mozilla
Bugs, which is subscribed to nss in Ubuntu.
https://bugs.launchpad.net/bugs/1931104

Title:
  Test of dogtag-pki is failing on s390x vs the nss v3.63 in impish-
  proposed

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nss/+bug/1931104/+subscriptions



More information about the Ubuntu-mozillateam-bugs mailing list