[Bug 1958488] Re: [nvidia][xorg] display hangs on boot LOGO due to race of gdm and nvidia driver probe

Wed Mar 16 00:51:25 UTC 2022

Hello jeremyszu, or anyone else affected,

Accepted ubuntu-drivers-common into impish-proposed. The package will
build now and be available at
https://launchpad.net/ubuntu/+source/ubuntu-drivers-
common/1:0.9.2.4~0.21.10.1 in a few hours, and then in the -proposed
repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
impish to verification-done-impish. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-impish. In either case, without details of your testing we will
not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: ubuntu-drivers-common (Ubuntu Impish)
       Status: In Progress => Fix Committed

** Tags added: verification-needed-impish

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1958488

Title:
  [nvidia][xorg] display hangs on boot LOGO due to race of gdm and
  nvidia driver probe

Status in OEM Priority Project:
  Confirmed
Status in gdm3 package in Ubuntu:
  Incomplete
Status in ubuntu-drivers-common package in Ubuntu:
  Fix Released
Status in gdm3 source package in Focal:
  Incomplete
Status in ubuntu-drivers-common source package in Focal:
  Fix Committed
Status in gdm3 source package in Impish:
  Incomplete
Status in ubuntu-drivers-common source package in Impish:
  Fix Committed

Bug description:
  [Impact]

   * In Ubuntu 20.04 (either impish, jammy, upstream gdm) (which using Xorg with proprietary nvidia driver), in some cases, the nvidia driver will probe later than launching gdm.
   * If above race condition happens in iGPU + nvidia cases and monitor connects to dGPU, which will cause gdm starts with wayland as opposed to Xorg. Which may lead the monitor stuck in black-screen or boot LOGO.

  [Test Plan]

   * The environment:
    1. A desktop or workstation which containing an iGPU.
    2. Plug a nvidia graphic card to the system and installing proprietary
    nvidia driver (470 in my case)
    3. Attach a monitor to dGPU and leave iGPU connect to nothing.
    (in my test environment, there is the other ethernet card and TBT4 cards)
   * Setup a cronjob,
     e.g. @reboot /home/u/test.sh
   * Have a test script in something like /home/u/test.sh as

  #!/bin/bash

  sleep 20
  count="$(cat /home/ubuntu/count)"
  count=$((count+1))
  echo $count | tee /home/ubuntu/count
  journalctl -b | grep -q -i wayland || sudo reboot

   * the system will probably stuck in black-screen or boot LOGO.
   * Before applying the fix, the fail rate is 6/24 (fail 6 time in 24 runs).
   * After applying the fix, it got pass within 1000+ reboot cycles.
   * Test PPA can be found here https://launchpad.net/~os369510/+archive/ubuntu/lp1958488

  [Fix]
   * The patch makes gpu-manager to probe nvidia (if needed) first and waiting for the /run/u-d-c-nvidia-drm-was-loaded be touched by 71-u-d-c-gpu-detection.rules.
   * Also, the gdm is using 61-gdm.rules to configure the gdm mode by checking the nvidia driver presents or not.
   * gpu-manager is before display-manager. Thus, gpu-manager will wait for nvidia uevent be processed and then continue to work. When gdm be launched, the targeted nvidia uevent has been processed already. (71-u-d-c-gpu-detection.rules is later than 61-gdm.rules）

  [Where problems could occur]
   * there is not potential regression from my mind but it will lead the boot time be longer.
   * In my test cycles, it leads extra 0~1000ms in boot time. Usually, 0~200ms. Worst case, over 1 s in 8xx runs (of 1000).
   * I think the stability is important than performance in this case.

  [Other Info]
   * For non-ubuntu-desktop (which doesn't have gpu-manager), which using gdm will meet this issue still. The other potential fix (from either gdm or logind) is under discussion in https://gitlab.gnome.org/GNOME/gdm/-/issues/763#note_1385786.
   * u-d-c upstream fix: https://github.com/tseliot/ubuntu-drivers-common/pull/67

  ---

  Test environment/steps:
  1. A desktop or workstation which containing an iGPU.
  2. Plug a nvidia graphic card to the system and installing proprietary nvidia driver (470 in my case)
  3. Attach a monitor to dGPU and leave iGPU connect to nothing.
  (in my test environment, there is the other ethernet card and TBT4 cards)
  4. Reboot system.

  Based on:
  $ cat /lib/udev/rules.d/61-gdm.rules
  # disable Wayland on Hi1710 chipsets
  ATTR{vendor}=="0x19e5", ATTR{device}=="0x1711", RUN+="/usr/lib/gdm3/gdm-disable-wayland"
  # disable Wayland when using the proprietary nvidia driver
  DRIVER=="nvidia", RUN+="/usr/lib/gdm3/gdm-disable-wayland"

  It will disable wayland by default if proprietary nvidia driver load.
  But in some race condition cases, the nvidia probe is later than gnome launches. (The fail rate is 6/24.)
  Thus, ubuntu-gdm has a fix for Bug#1794280 to add "ExecStartPre=@libexecdir@/gdm-wait-for-drm".

  The gdm-wait-for-drm is intend to make sure all drm udev devices
  enumerated before launching gdm.

  It rely on at least one "master-of-seat" graphic card for gdm but it's not rigorous enough.
  Since most of graphic cards are own "master-of-seat"[1].

  In my case, it detects the iGPU is probed but dGPU.
  However, the display is attached to dGPU.

  We need to make sure the targeted gpu (connecting to monitor) is probe
  before launching gdm.

  debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1004131
  upstream bug: https://gitlab.gnome.org/GNOME/gdm/-/issues/763

  [1] https://www.freedesktop.org/wiki/Software/systemd/multiseat/
  /lib/udev/rules.d/71-seat.rules
  /lib/udev/rules.d/71-nvidia.rules

To manage notifications about this bug go to:
https://bugs.launchpad.net/oem-priority/+bug/1958488/+subscriptions