[Bug 1904585] Re: opal-prd: Have a worker process handle page offlining (Fixes "PlatServices: dyndealloc memory_error() failed" is getting reported in error log (opal-prd))

[Patricia Domingues]

Vasant, thanks for the verification on this bug.
Could you also check the other opal-prd bug which also needs verification, please: 

```
"Ubuntu 20.04: opal-prd fails to start on 20.04"
https://bugs.launchpad.net/bugs/1905393
```

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1904585

Title:
  opal-prd: Have a worker process handle page offlining (Fixes
  "PlatServices: dyndealloc memory_error() failed" is getting reported
  in error log (opal-prd))

Status in The Ubuntu-power-systems project:
  Fix Committed
Status in skiboot package in Ubuntu:
  Fix Released
Status in skiboot source package in Xenial:
  Fix Committed
Status in skiboot source package in Bionic:
  Fix Committed
Status in skiboot source package in Focal:
  Fix Committed
Status in skiboot source package in Groovy:
  Fix Committed
Status in skiboot source package in Hirsute:
  Fix Released

Bug description:
  [Impact]

  This impacts the opal-prd userspace command from the skiboot package
  The memory_error() hservice interface expects the memory_error() call to
  just accept the offline request and return without actually offlining the
  memory. Currently we will attempt to offline the marked pages before
  returning to HBRT which can result in an excessively long time spent in the memory_error() hservice call which blocks HBRT from processing other
  errors.

  [Test Case]

  Unfortunately due to the specific hardware requirement I wasn't able
  to reproduce this problem and provide a test case for it. However I
  was able to build this package into a ppa and got the IBM team to
  confirm this problem was resolved for groovy focal, bionic, xenial see
  comment #4 and #6

  Another verification test will be done (as part of the SRU process)
  again by the IBM Power team.

  [What could go wrong]

  To avoid long delays (that may blocks HBRT from processing other
  errors) the memory offlining process is now separated in a dedicated
  worker process, that can now be handled in the background.

  If broken this can introduce further issues, like hangs in the worker process, not returning, and processes that pile up or in worst case memory pages that are not offlined, but reported otherwise.
  The latter one would be a significant memory management problem, that even may break the system over time entirely.

  But the code seem to have taken this into account with 'sigaction',
  the return-code/exit-status check and the reaping of the worker
  threads.

  The fix was prepared back in September and was upstream accepted,
  hence it's unlikely that regressions are in and in between it already
  landed in hirsute.

  On top a PPA with a patched skiboot package was created, shared and
  eventually successfully tested by IBM (the initial bug reporter).

  [Original Description]

  https://github.com/open-
  power/skiboot/commit/8cbd0de88d162e387f11569eee1bdecef8fad2e3

  opal-prd: Have a worker process handle page offlining

  The memory_error() hservice interface expects the memory_error() call to
  just accept the offline request and return without actually offlining the
  memory. Currently we will attempt to offline the marked pages before
  returning to HBRT which can result in an excessively long time spent in the
  memory_error() hservice call which blocks HBRT from processing other
  errors. Fix this by adding a worker process which performs the page
  offlining via the sysfs memory error interfaces.

  Reviewed-by: Vasant Hegde - hegdevasant at linux.vnet.ibm.com
  Signed-off-by: Oliver O'Halloran - oohall at gmail.com

  Thanks in advance for your support.

  Machine Type = Power8 and Power9 OPAL systems

  ---Steps to Reproduce---
  * Inject memory error (UE)
  * Verify that opal-prd doesn't return asynchronously to the platform after requesting the memory offlining operation

  Userspace tool common name: opal-prd

  We need this fix for 16.04.x and 18.04.x LTS releases.

  Fix also is needed for 20.04 and 20.10.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-power-systems/+bug/1904585/+subscriptions