[Bug 1872021] Re: commissioning fails due to hung tasks setting up ipmitool

[Rafael David Tinoco]

Summary...

For Ubuntu Bionic, dpkg triggers for systemd (237-3ubuntu10.39) might
have caused systemd to hang:

[  363.776878]  wait_for_completion+0xba/0x140
[  363.776890]  __flush_work+0x15b/0x210
[  363.776901]  flush_delayed_work+0x41/0x50
[  363.776908]  fsnotify_wait_marks_destroyed+0x15/0x20
[  363.776912]  fsnotify_destroy_group+0x48/0xd0
[  363.776917]  inotify_release+0x1e/0x50
[  363.776923]  __fput+0xea/0x220
[  363.776929]  ____fput+0xe/0x10
[  363.776935]  task_work_run+0x9d/0xc0
[  363.776942]  exit_to_usermode_loop+0xc0/0xd0
[  363.776947]  do_syscall_64+0x121/0x130
[  363.776954]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2

and

[  364.050206]  wait_for_completion+0xba/0x140
[  364.050238]  __synchronize_srcu.part.13+0x85/0xb0
[  364.050248]  synchronize_srcu+0x66/0xe0
[  364.050256]  fsnotify_mark_destroy_workfn+0x7b/0xe0
[  364.050262]  process_one_work+0x1de/0x420
[  364.050267]  worker_thread+0x228/0x410
[  364.050272]  kthread+0x121/0x140

and

[  364.326985]  wait_for_completion+0xba/0x140
[  364.326988]  __synchronize_srcu.part.13+0x85/0xb0
[  364.326993]  synchronize_srcu+0x66/0xe0
[  364.326995]  ? synchronize_srcu+0x66/0xe0
[  364.326996]  fsnotify_connector_destroy_workfn+0x4a/0x80
[  364.326998]  process_one_work+0x1de/0x420
[  364.326999]  worker_thread+0x253/0x410
[  364.327001]  kthread+0x121/0x140

All stack traces seem to come from "fsnotify" subsystem and waiting on
delayed work (completion) for fsnotify marks destruction after a
inotify_release() was called. Completion did not happen for the past 2
minutes. Without a kernel dump it is hard to tell if completion was
still ok - due to kthread being overloaded doing scheduled work and/or
the marks group destruction - or there was a dead lock for the
completion due to a kernel bug.

If this is reproducible, I think that having a kernel dump would help
identifying the issue. I'm letting the kernel team to handle this and
marking all other issues as dealt per previous comments.


** No longer affects: ipmitool (Ubuntu)

** Changed in: maas
       Status: New => Invalid

** Summary changed:

- commissioning fails due to hung tasks setting up ipmitool
+ [Ubuntu][Bionic] systemd caused kernel to hang on fsnotify wait-on-completion

** Also affects: linux (Ubuntu)
   Importance: Undecided
       Status: New

** No longer affects: linux (Ubuntu)

** Project changed: linux => linux (Ubuntu)

** Also affects: linux (Ubuntu Bionic)
   Importance: Undecided
       Status: New

** Also affects: linux (Ubuntu Focal)
   Importance: Undecided
       Status: New

** Also affects: linux (Ubuntu Eoan)
   Importance: Undecided
       Status: New

** Changed in: linux (Ubuntu Bionic)
       Status: New => Triaged

** Changed in: linux (Ubuntu Bionic)
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Ubuntu
Server, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1872021

Title:
  [Ubuntu][Bionic] systemd caused kernel to hang on fsnotify wait-on-
  completion

To manage notifications about this bug go to:
https://bugs.launchpad.net/maas/+bug/1872021/+subscriptions