[Bug 2129676] Re: QPDF tries to use MD5 in FIPS mode

[Dariusz Gadomski]

** Patch removed: "resolute.debdiff"
   https://bugs.launchpad.net/ubuntu/+source/qpdf/+bug/2129676/+attachment/5927498/+files/resolute.debdiff

** Patch added: "resolute.debdiff"
   https://bugs.launchpad.net/ubuntu/+source/qpdf/+bug/2129676/+attachment/5932306/+files/resolute.debdiff

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/2129676

Title:
  QPDF tries to use MD5 in FIPS mode

Status in Qpdf:
  New
Status in qpdf package in Ubuntu:
  New
Status in qpdf source package in Jammy:
  New
Status in qpdf source package in Noble:
  New
Status in qpdf source package in Plucky:
  New
Status in qpdf source package in Questing:
  New
Status in qpdf source package in Resolute:
  New

Bug description:
  [ Impact ]

   * After enabling fips (or fips-updates) libqpdf will fail to
  calculate MD5 for decoding the PDFs (which are used by cups-filters
  during the printing process).

  * Printing may become impossible after enabling FIPS.

  * For such situations gnutls provides LAX mode [1]:

  GNUTLS_FIPS140_LAX
  The library still uses the FIPS140-2 relevant algorithms but all forbidden by FIPS140-2 operations are allowed; this is useful when the application is aware of the followed security policy, and needs to utilize disallowed operations for other reasons (e.g., compatibility).

  [1] https://www.gnutls.org/manual/html_node/FIPS140_002d2-mode.html

  [ Test Plan ]

   * Enable FIPS:
  pro attach # to be able to enable FIPS mode
  pro enable fips-updates
  reboot # to boot the fips kernel

  * Try to run the filter directly:
  /usr/lib/cups/filter/pdftopdf 555 $USER title 1 "" /usr/share/cups/data/confidential.pdf

  * With affected qpdf version it aborts with the following messages:
  ERROR: cfFilterPDFToPDF: Exception: gnutls: MD5 error: An algorithm that is not enabled was negotiated.
  ERROR: pdftopdf filter function failed.

  [ Where problems could occur ]

   * The patch relaxes gnutls restrictions on algorithms in non-security related,
  limited contexts (GNUTLS_FIPS140_LAX mode). If there is an automated tooling or  auditting software they may detect that not only FIPS-approved algorithms are
  being used.

  However, in context of decoding PDFs it is not possible to get rid of
  MD5 entirely. It's usage however is limited to only non-security
  related taks.

  [ Other Info ]

  Original bug description:

  After enabling fips/fips-updates it is impossible to print anymore.

  Requesting a print results in the following message in the cups logs:
  `ERROR: cfFilterPDFToPDF: Exception: gnutls: MD5 error: An algorithm that is not enabled was negotiated.`

  I have came up with a small reproducer:
  ```
  pro attach # to be able to enable FIPS mode
  pro enable fips-updates
  reboot # to boot the fips kernel
  # with FIPS mode enabled
  /usr/lib/cups/filter/pdftopdf 555 $USER title 1 "" /usr/share/cups/data/confidential.pdf
  ```

  The output ends with:
  ERROR: cfFilterPDFToPDF: Exception: gnutls: MD5 error: An algorithm that is not enabled was negotiated.
  ERROR: pdftopdf filter function failed.

  I have tracked the problematic code to:
  QPDF::compute_data_key in libqpdf/QPDF_encryption.cc

  It unconditionally uses MD5 (that in turn asks gnutls for MD5) and in
  FIPS mode it fails as MD5 is not fips-approved.

  The bottomline is: it is not possible to print with fips-mode enabled.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qpdf/+bug/2129676/+subscriptions