[Bug 1872118] [NEW] [SRU] DHCP Cluster crashes after a few hours

[Launchpad Bug Tracker]

You have been subscribed to a public bug by Ubuntu Foundations Team Bug Bot (crichton):

[Description]

isc-dhcp-server uses libisc-export (coming from bind9-libs package) for handling the socket event(s) when configured in peer mode (master/secondary). It's possible that a sequence of messages dispatched by the master that requires acknowledgment from its peers holds a socket
in a pending to send state, a timer or a subsequent write request can be scheduled into this socket and the !sock->pending_send assertion
will be raised when trying to write again at the time data hasn't been flushed entirely and the pending_send flag hasn't been reset to 0 state.

If this race condition happens, the following stacktrace will be
hit:

(gdb) info threads
  Id Target Id Frame
* 1 Thread 0x7fb4ddecb700 (LWP 3170) __GI_raise (sig=sig at entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
  2 Thread 0x7fb4dd6ca700 (LWP 3171) __lll_lock_wait (futex=futex at entry=0x7fb4de6d2028, private=0) at lowlevellock.c:52
  3 Thread 0x7fb4de6cc700 (LWP 3169) futex_wake (private=<optimized out>, processes_to_wake=1, futex_word=<optimized out>) at ../sysdeps/nptl/futex-internal.h:364
  4 Thread 0x7fb4de74f740 (LWP 3148) futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fb4de6cd0d0) at ../sysdeps/nptl/futex-internal.h:183

(gdb) frame 2
#2 0x00007fb4dec85985 in isc_assertion_failed (file=file at entry=0x7fb4decd8878 "../../../../lib/isc/unix/socket.c", line=line at entry=3361, type=type at entry=isc_assertiontype_insist,
    cond=cond at entry=0x7fb4decda033 "!sock->pending_send") at ../../../lib/isc/assertions.c:52
(gdb) bt
#1 0x00007fb4deaa7859 in __GI_abort () at abort.c:79
#2 0x00007fb4dec85985 in isc_assertion_failed (file=file at entry=0x7fb4decd8878 "../../../../lib/isc/unix/socket.c", line=line at entry=3361, type=type at entry=isc_assertiontype_insist,
    cond=cond at entry=0x7fb4decda033 "!sock->pending_send") at ../../../lib/isc/assertions.c:52
#3 0x00007fb4decc17e1 in dispatch_send (sock=0x7fb4de6d4990) at ../../../../lib/isc/unix/socket.c:4041
#4 process_fd (writeable=<optimized out>, readable=<optimized out>, fd=11, manager=0x7fb4de6d0010) at ../../../../lib/isc/unix/socket.c:4054
#5 process_fds (writefds=<optimized out>, readfds=0x7fb4de6d1090, maxfd=13, manager=0x7fb4de6d0010) at ../../../../lib/isc/unix/socket.c:4211
#6 watcher (uap=0x7fb4de6d0010) at ../../../../lib/isc/unix/socket.c:4397
#7 0x00007fb4dea68609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8 0x00007fb4deba4103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

(gdb) frame 3
#3 0x00007fb4decc17e1 in dispatch_send (sock=0x7fb4de6d4990) at ../../../../lib/isc/unix/socket.c:4041
4041 in ../../../../lib/isc/unix/socket.c
(gdb) p sock->pending_send
$2 = 1

[TEST CASE]

1) Install isc-dhcp-server in 2 focal machine(s).
2) Configure peer/cluster mode as follows:
   Primary configuration: https://pastebin.ubuntu.com/p/XYj648MghK/
   Secondary configuration: https://pastebin.ubuntu.com/p/PYkcshZCWK/
2) Run dhcpd as follows in both machine(s)

# dhcpd -f -d -4 -cf /etc/dhcp/dhcpd.conf --no-pid ens4

3) Leave the cluster running for a long (2h) period until the crash/race
condition is reproduced.

[REGRESSION POTENTIAL]

* The fix will prevent the assertion to happen in the dispatch_send
path, later versions of isch-dhcp upstream lack this logic and entirely removed the existence of this flag. Therefore, removing the need for this
assertion at process_fd shouldn't be problematic.

** Affects: dhcp
     Importance: Unknown
         Status: New

** Affects: bind9-libs (Ubuntu)
     Importance: Undecided
     Assignee: Jorge Niedbalski (niedbalski)
         Status: In Progress

** Affects: isc-dhcp (Ubuntu)
     Importance: Undecided
     Assignee: Jorge Niedbalski (niedbalski)
         Status: In Progress

** Affects: bind9-libs (Ubuntu Focal)
     Importance: Undecided
     Assignee: Jorge Niedbalski (niedbalski)
         Status: In Progress

** Affects: isc-dhcp (Ubuntu Focal)
     Importance: Undecided
     Assignee: Jorge Niedbalski (niedbalski)
         Status: In Progress

** Affects: bind9-libs (Ubuntu Groovy)
     Importance: Undecided
     Assignee: Jorge Niedbalski (niedbalski)
         Status: In Progress

** Affects: isc-dhcp (Ubuntu Groovy)
     Importance: Undecided
     Assignee: Jorge Niedbalski (niedbalski)
         Status: In Progress


** Tags: focal patch rls-ff-incoming
-- 
[SRU] DHCP Cluster crashes after a few hours
https://bugs.launchpad.net/bugs/1872118
You received this bug notification because you are a member of Ubuntu Sponsors Team, which is subscribed to the bug report.