[Bug 1863919] [NEW] [regression] lingering pvscan during boot

Simon Déziel 1863919 at bugs.launchpad.net
Wed Feb 19 16:37:47 UTC 2020 

Public bug reported:

Since lvm2 was updated to 2.02.176-4.1ubuntu3.18.04.2 on Bionic (LP:
#1854981), we notice that some of our machines have lingering pvscan
processes apparently running from the initramfs's root that
persist/never finish/exit.

On the affected servers, this is visible as there are 2 instances of
systemd-udevd, one in the init.scope and another in system.slice:

$ systemd-cgls | cat
Control group /:
-.slice
├─user.slice
│ ├─user-501.slice
│ │ ├─session-7.scope
│ │ │ ├─12324 sshd: foo [priv]
│ │ │ ├─12353 sshd: foo at pts/2
│ │ │ ├─12354 script --quiet --return --command /bin/bash -l /dev/null
│ │ │ ├─12356 /bin/bash -l
│ │ │ ├─12375 sudo -i
│ │ │ └─12385 -bash
│ │ └─user at 501.service
│ │   └─init.scope
│ │     ├─12326 /lib/systemd/systemd --user
│ │     └─12327 (sd-pam)
│ └─user-500.slice
│   ├─user at 500.service
│   │ └─init.scope
│   │   ├─12185 /lib/systemd/systemd --user
│   │   └─12186 (sd-pam)
│   └─session-3.scope
│     ├─12170 sshd: sdeziel [priv]
│     ├─12254 sshd: sdeziel at pts/0
│     ├─12255 script --quiet --return --command /bin/bash -l /dev/null
│     ├─12257 /bin/bash -l
│     ├─14409 systemd-cgls
│     └─14410 cat
├─init.scope
│ ├─   1 /sbin/init kaslr nosplash
│ ├─1451 /lib/systemd/systemd-udevd --daemon --resolve-names=never
│ ├─1466 /sbin/lvm pvscan --cache --activate ay --major 8 --minor 3
│ ├─1470 /sbin/lvm pvscan --cache --activate ay --major 8 --minor 2
│ ├─1481 /sbin/lvm pvscan --cache --activate ay --major 253 --minor 1
│ └─1551 /sbin/lvm pvscan --cache --activate ay --major 253 --minor 1
└─system.slice
  ├─irqbalance.service
  │ └─2796 /usr/sbin/irqbalance --foreground
  ├─systemd-networkd.service
  │ └─2833 /lib/systemd/systemd-networkd
  ├─systemd-udevd.service
  │ └─1627 /lib/systemd/systemd-udevd
  ├─cron.service
  │ ├─ 2786 /usr/sbin/cron -f
  │ └─12308 /usr/bin/python /usr/sbin/ganeti-noded -b 172.20.30.212
  ├─system-serial\x2dgetty.slice
  │ └─serial-getty at ttyS0.service
  │   └─12112 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220
  ├─systemd-journald.service
  │ └─1601 /lib/systemd/systemd-journald
  ├─ssh.service
  │ └─12115 /usr/sbin/sshd -D
  ├─rsyslog.service
  │ └─2781 /usr/sbin/rsyslogd -n
  ├─nagios-nrpe-server.service
  │ └─12108 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f
  ├─lvm2-lvmetad.service
  │ └─1630 /sbin/lvmetad -f
  ├─systemd-resolved.service
  │ └─3280 /lib/systemd/systemd-resolved
  ├─dbus.service
  │ └─2762 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidf...
  ├─systemd-timesyncd.service
  │ └─2491 /lib/systemd/systemd-timesyncd
  ├─system-getty.slice
  │ └─getty at tty1.service
  │   └─12113 /sbin/agetty -o -p -- \u --noclear tty1 linux
  ├─systemd-logind.service
  │ └─2797 /lib/systemd/systemd-logind
  └─ganeti.service
    ├─ 3892 /usr/sbin/ganeti-confd
    ├─12173 /usr/sbin/ganeti-kvmd
    └─12276 /usr/sbin/ganeti-mond

The systemd-udevd in init.scope and the many /sbin/lvm pvscan it
launched during boot are left there forever.

This trips our monitoring for binaries using deleted binaries/libs and
is also visible like that:

# pgrep systemd-udevd
1451
1627

# ls -l /proc/1451/exe
lrwxrwxrwx 1 root root 0 Feb 19 10:35 /proc/1451/exe -> '/lib/systemd/systemd-udevd (deleted)'

# ls -l /proc/1627/exe
lrwxrwxrwx 1 root root 0 Feb 19 10:35 /proc/1627/exe -> /lib/systemd/systemd-udevd


This can be compared with another server *NOT AFFECTED*, where the init.scope is cleaner (/sbin/init only) and a single systemd-udevd process:

# systemd-cgls | cat
Control group /:
-.slice
├─user.slice
│ └─user-0.slice
│   ├─session-313.scope
│   │ ├─90332 sshd: root at pts/0
│   │ ├─90406 script --quiet --return --command /bin/bash -l /dev/null
│   │ ├─90410 /bin/bash -l
│   │ ├─90436 systemd-cgls
│   │ └─90437 cat
│   └─user at 0.service
│     └─init.scope
│       ├─90334 /lib/systemd/systemd --user
│       └─90335 (sd-pam)
├─init.scope
│ └─1 /sbin/init kaslr nosplash
└─system.slice
...
  ├─systemd-udevd.service
  │ └─2622 /lib/systemd/systemd-udevd
...
  ├─lvm2-lvmetad.service
  │ └─2621 /sbin/lvmetad -f
...


Additional information:

# lsb_release -rd
Description:	Ubuntu 18.04.4 LTS
Release:	18.04
# apt-cache policy lvm2 systemd
lvm2:
  Installed: 2.02.176-4.1ubuntu3.18.04.2
  Candidate: 2.02.176-4.1ubuntu3.18.04.2
  Version table:
 *** 2.02.176-4.1ubuntu3.18.04.2 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     2.02.176-4.1ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
systemd:
  Installed: 237-3ubuntu10.39
  Candidate: 237-3ubuntu10.39
  Version table:
 *** 237-3ubuntu10.39 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     237-3ubuntu10.38 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     237-3ubuntu10 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

** Affects: lvm2 (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: regression-update

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to lvm2 in Ubuntu.
https://bugs.launchpad.net/bugs/1863919

Title:
  [regression] lingering pvscan during boot

Status in lvm2 package in Ubuntu:
  New

Bug description:
  Since lvm2 was updated to 2.02.176-4.1ubuntu3.18.04.2 on Bionic (LP:
  #1854981), we notice that some of our machines have lingering pvscan
  processes apparently running from the initramfs's root that
  persist/never finish/exit.

  On the affected servers, this is visible as there are 2 instances of
  systemd-udevd, one in the init.scope and another in system.slice:

  $ systemd-cgls | cat
  Control group /:
  -.slice
  ├─user.slice
  │ ├─user-501.slice
  │ │ ├─session-7.scope
  │ │ │ ├─12324 sshd: foo [priv]
  │ │ │ ├─12353 sshd: foo at pts/2
  │ │ │ ├─12354 script --quiet --return --command /bin/bash -l /dev/null
  │ │ │ ├─12356 /bin/bash -l
  │ │ │ ├─12375 sudo -i
  │ │ │ └─12385 -bash
  │ │ └─user at 501.service
  │ │   └─init.scope
  │ │     ├─12326 /lib/systemd/systemd --user
  │ │     └─12327 (sd-pam)
  │ └─user-500.slice
  │   ├─user at 500.service
  │   │ └─init.scope
  │   │   ├─12185 /lib/systemd/systemd --user
  │   │   └─12186 (sd-pam)
  │   └─session-3.scope
  │     ├─12170 sshd: sdeziel [priv]
  │     ├─12254 sshd: sdeziel at pts/0
  │     ├─12255 script --quiet --return --command /bin/bash -l /dev/null
  │     ├─12257 /bin/bash -l
  │     ├─14409 systemd-cgls
  │     └─14410 cat
  ├─init.scope
  │ ├─   1 /sbin/init kaslr nosplash
  │ ├─1451 /lib/systemd/systemd-udevd --daemon --resolve-names=never
  │ ├─1466 /sbin/lvm pvscan --cache --activate ay --major 8 --minor 3
  │ ├─1470 /sbin/lvm pvscan --cache --activate ay --major 8 --minor 2
  │ ├─1481 /sbin/lvm pvscan --cache --activate ay --major 253 --minor 1
  │ └─1551 /sbin/lvm pvscan --cache --activate ay --major 253 --minor 1
  └─system.slice
    ├─irqbalance.service
    │ └─2796 /usr/sbin/irqbalance --foreground
    ├─systemd-networkd.service
    │ └─2833 /lib/systemd/systemd-networkd
    ├─systemd-udevd.service
    │ └─1627 /lib/systemd/systemd-udevd
    ├─cron.service
    │ ├─ 2786 /usr/sbin/cron -f
    │ └─12308 /usr/bin/python /usr/sbin/ganeti-noded -b 172.20.30.212
    ├─system-serial\x2dgetty.slice
    │ └─serial-getty at ttyS0.service
    │   └─12112 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220
    ├─systemd-journald.service
    │ └─1601 /lib/systemd/systemd-journald
    ├─ssh.service
    │ └─12115 /usr/sbin/sshd -D
    ├─rsyslog.service
    │ └─2781 /usr/sbin/rsyslogd -n
    ├─nagios-nrpe-server.service
    │ └─12108 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f
    ├─lvm2-lvmetad.service
    │ └─1630 /sbin/lvmetad -f
    ├─systemd-resolved.service
    │ └─3280 /lib/systemd/systemd-resolved
    ├─dbus.service
    │ └─2762 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidf...
    ├─systemd-timesyncd.service
    │ └─2491 /lib/systemd/systemd-timesyncd
    ├─system-getty.slice
    │ └─getty at tty1.service
    │   └─12113 /sbin/agetty -o -p -- \u --noclear tty1 linux
    ├─systemd-logind.service
    │ └─2797 /lib/systemd/systemd-logind
    └─ganeti.service
      ├─ 3892 /usr/sbin/ganeti-confd
      ├─12173 /usr/sbin/ganeti-kvmd
      └─12276 /usr/sbin/ganeti-mond

  The systemd-udevd in init.scope and the many /sbin/lvm pvscan it
  launched during boot are left there forever.

  This trips our monitoring for binaries using deleted binaries/libs and
  is also visible like that:

  # pgrep systemd-udevd
  1451
  1627

  # ls -l /proc/1451/exe
  lrwxrwxrwx 1 root root 0 Feb 19 10:35 /proc/1451/exe -> '/lib/systemd/systemd-udevd (deleted)'

  # ls -l /proc/1627/exe
  lrwxrwxrwx 1 root root 0 Feb 19 10:35 /proc/1627/exe -> /lib/systemd/systemd-udevd

  
  This can be compared with another server *NOT AFFECTED*, where the init.scope is cleaner (/sbin/init only) and a single systemd-udevd process:

  # systemd-cgls | cat
  Control group /:
  -.slice
  ├─user.slice
  │ └─user-0.slice
  │   ├─session-313.scope
  │   │ ├─90332 sshd: root at pts/0
  │   │ ├─90406 script --quiet --return --command /bin/bash -l /dev/null
  │   │ ├─90410 /bin/bash -l
  │   │ ├─90436 systemd-cgls
  │   │ └─90437 cat
  │   └─user at 0.service
  │     └─init.scope
  │       ├─90334 /lib/systemd/systemd --user
  │       └─90335 (sd-pam)
  ├─init.scope
  │ └─1 /sbin/init kaslr nosplash
  └─system.slice
  ...
    ├─systemd-udevd.service
    │ └─2622 /lib/systemd/systemd-udevd
  ...
    ├─lvm2-lvmetad.service
    │ └─2621 /sbin/lvmetad -f
  ...

  
  Additional information:

  # lsb_release -rd
  Description:	Ubuntu 18.04.4 LTS
  Release:	18.04
  # apt-cache policy lvm2 systemd
  lvm2:
    Installed: 2.02.176-4.1ubuntu3.18.04.2
    Candidate: 2.02.176-4.1ubuntu3.18.04.2
    Version table:
   *** 2.02.176-4.1ubuntu3.18.04.2 500
          500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
          100 /var/lib/dpkg/status
       2.02.176-4.1ubuntu3 500
          500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
  systemd:
    Installed: 237-3ubuntu10.39
    Candidate: 237-3ubuntu10.39
    Version table:
   *** 237-3ubuntu10.39 500
          500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
          100 /var/lib/dpkg/status
       237-3ubuntu10.38 500
          500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
       237-3ubuntu10 500
          500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/lvm2/+bug/1863919/+subscriptions

More information about the foundations-bugs mailing list