[Bug 1934147] Re: systemd leaks abandoned session scopes

[Dan Streetman]

ubuntu at lp1934147-b:~$ dpkg -l systemd|grep systemd
ii  systemd        237-3ubuntu10.51 amd64        system and service manager
ubuntu at lp1934147-b:~$ loginctl list-sessions
   SESSION        UID USER             SEAT             TTY             
         2       1000 ubuntu                            ttyS0           

1 sessions listed.
ubuntu at lp1934147-b:~$ for i in {1..100}; do sudo -b -i -u ubuntu ssh localhost -- sleep 1; done; for i in {1..20}; do echo 'Reloading...'; sudo systemctl daemon-reload; done
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
ubuntu at lp1934147-b:~$ loginctl list-sessions
   SESSION        UID USER             SEAT             TTY             
        60       1000 ubuntu                                            
       103       1000 ubuntu                                            
       105       1000 ubuntu                                            
        80       1000 ubuntu                                            
       104       1000 ubuntu                                            
         2       1000 ubuntu                            ttyS0           
       100       1000 ubuntu                                            
        63       1000 ubuntu                                            
        83       1000 ubuntu                                            
        74       1000 ubuntu                                            
        95       1000 ubuntu                                            
        64       1000 ubuntu                                            
        76       1000 ubuntu                                            
        59       1000 ubuntu                                            

14 sessions listed.

ubuntu at lp1934147-b:~$ dpkg -l systemd|grep systemd
ii  systemd        237-3ubuntu10.52 amd64        system and service manager
ubuntu at lp1934147-b:~$ loginctl list-sessions
   SESSION        UID USER             SEAT             TTY             
         2       1000 ubuntu                            ttyS0           

1 sessions listed.
ubuntu at lp1934147-b:~$ for i in {1..100}; do sudo -b -i -u ubuntu ssh localhost -- sleep 1; done; for i in {1..20}; do echo 'Reloading...'; sudo systemctl daemon-reload; done
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
ubuntu at lp1934147-b:~$ loginctl list-sessions
   SESSION        UID USER             SEAT             TTY             
         2       1000 ubuntu                            ttyS0           

1 sessions listed.


** Tags removed: verification-needed verification-needed-bionic verification-needed-focal verification-needed-hirsute
** Tags added: verification-done verification-done-bionic verification-done-focal verification-done-hirsute

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1934147

Title:
  systemd leaks abandoned session scopes

Status in snapd:
  New
Status in systemd:
  New
Status in systemd package in Ubuntu:
  Fix Committed
Status in systemd source package in Bionic:
  Fix Committed
Status in systemd source package in Focal:
  Fix Committed
Status in systemd source package in Groovy:
  Won't Fix
Status in systemd source package in Hirsute:
  Fix Committed
Status in systemd source package in Impish:
  Fix Committed

Bug description:
  [impact]

  systemd may leak sessions, leaving empty cgroups around as well as
  abandoned session scopes.

  [test case]

  on a system where the user has a ssh key that allows noninteractive
  login to localhost, and also has noninteractive sudo, run:

  $ for i in {1..100}; do sudo -b -i -u ubuntu ssh localhost -- sleep 1;
  done; for i in {1..20}; do echo 'Reloading...'; sudo systemctl daemon-
  reload; done

  check the sessions to see there have been leaked sessions:

  $ loginctl list-sessions

  SESSION  UID USER   SEAT TTY
        1 1000 ubuntu      ttyS0
      350 1000 ubuntu      
      351 1000 ubuntu      
      360 1000 ubuntu      
  ...

  to verify the sessions were leaked, clear them out with:

  $ echo '' | sudo tee
  /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-*.scope/cgroup.events

  that should result in all the leaked sessions being cleaned up.

  [regression potential]

  issues during systemd pid1 reexec/reload, or issues while cleaning up
  sessions, including leaking sessions/cgroups

  [scope]

  this is needed for all releases

  upstream bug linked above, and upstream PR:
  https://github.com/systemd/systemd/pull/20199

  [original description]

  On a system that is monitored via telegraf I found many abandoned
  systemd session which I believe are created by a potential race where
  systemd is reloading unit files and at the same time a user is
  connecting to the system via ssh or is executing the su command.

  The simple reproducer

  $ for i in {1..100}; do sleep 0.2; ssh localhost sudo systemctl
  daemon-reload & ssh localhost sleep 1 & done

  Wait > 1 second

  $ jobs -p | xargs --verbose --no-run-if-empty kill -KILL

  To clean out STOPPED jobs and

  $ systemctl status --all 2> /dev/null | grep --before-context 3
  abandoned

  will produce something similar to

             │ ├─  175 su - ubuntu
             │ ├─  178 -su
             │ ├─62375 systemctl status --all
             │ └─62376 grep --color=auto --before-context 3 abandoned
  --
  ● session-273.scope - Session 273 of user ubuntu
     Loaded: loaded (/run/systemd/transient/session-273.scope; transient)
  Transient: yes
     Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
  --
  ● session-274.scope - Session 274 of user ubuntu
     Loaded: loaded (/run/systemd/transient/session-274.scope; transient)
  Transient: yes
     Active: active (abandoned) since Wed 2021-06-30 13:32:03 UTC; 4min 7s ago
  --
  ● session-30.scope - Session 30 of user ubuntu
     Loaded: loaded (/run/systemd/transient/session-30.scope; transient)
  Transient: yes
     Active: active (abandoned) since Wed 2021-06-30 10:05:56 UTC; 3h 30min ago
  --
  ● session-302.scope - Session 302 of user ubuntu
     Loaded: loaded (/run/systemd/transient/session-302.scope; transient)
  Transient: yes
     Active: active (abandoned) since Wed 2021-06-30 13:32:04 UTC; 4min 6s ago
  --
             │ ├─  175 su - ubuntu
             │ ├─  178 -su
             │ ├─62375 systemctl status --all
             │ └─62376 grep --color=auto --before-context 3 abandoned

  The system in question is running Bionic, systemd-237-3ubuntu10.48

To manage notifications about this bug go to:
https://bugs.launchpad.net/snapd/+bug/1934147/+subscriptions