Friday bug triage

[Bryce Harrington]

On Fri, Jan 07, 2022 at 08:34:57PM +0530, Utkarsh Gupta wrote:
> Hello,
> 
> On Fri, Jan 7, 2022 at 7:56 PM Utkarsh Gupta
> <utkarsh.gupta at canonical.com> wrote:
> >   LP: #1954854 - (Incomplete) [bind9] - bind
> >   9.16.1-ubuntu on ubuntu 20.04 randomly
> >   exits with [...]
> >
> > The bug reporter got back with some logs
> > and files collected via apport. Since the
> > bug is not easily reproducible, I am not quite
> > sure what's the best way forward. That said,
> > me/Paride are looking into it.
> 
> Here's an update: we couldn't find anything worthwhile from the
> upstream tracker or changelog or anything and neither of us have
> enough experience with crash files.

Here's a paint-by-numbers way to get stacktraces from .crash files:

0.  Enable debug symbols.  Various ways to do this, one way is to append
    ddebs to your apt sources:

    $ echo "deb http://ddebs.ubuntu.com focal main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
    $ echo "deb http://ddebs.ubuntu.com focal-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list

    $ sudo apt install ubuntu-dbgsym-keyring
    $ sudo apt update

1.  Install package with debug symbols

    $ sudo apt install bind9 bind9*-dbgsym

    Often there are also secondary libs or deps with more syms we'll
    need.  This examines the binary for other possible dbgsym's:

    $ sudo apt install debian-goodies
    $ sudo apt install $(find-dbgsym-packages /usr/sbin/named 2>/dev/null)

    The following NEW packages will be installed:
      libcap2-dbgsym libcom-err2-dbgsym libgcc-s1-dbgsym libicu67-dbgsym libjson-c5-dbgsym libkeyutils1-dbgsym
      libkrb5-dbg liblmdb0-dbgsym liblzma5-dbgsym libmaxminddb0-dbgsym libxml2-dbgsym zlib1g-dbgsym


2.  Download the .crash file to a tmp directory

    $ mkdir /tmp/bind9-crash
    $ cd /tmp/bind9-crash
    $ wget https://bugs.launchpad.net/ubuntu/+source/bind9/+bug/1954854/+attachment/5551855/+files/_usr_sbin_named.114.crash

3.  Register the package with the crash file

    $ cat <(echo "Package: bind9") _usr_sbin_named.114.crash > bind9_named.114.crash

    Apport-retrace complains if the package isn't specified, but apport
    hooks don't insert it, so you have to do it manually.  ¯\_(ツ)_/¯

4.  Doublecheck that your installed version matches the reporter's
    version exactly:

    $ apt-cache policy bind9 | grep '^  Installed'
      Installed: 1:9.16.1-0ubuntu2.9

    Unfortunately the .crash file doesn't include the version number,
    but the LP bug report will show it:

     Package: bind9 1:9.16.1-0ubuntu2.9

5.  Retrace symbols

    $ sudo apt-get install apport-retrace
    $ apport-retrace bind9_named.114.crash

    The above command inserted an empty 'separator: ' line into the crash
    file, which apport-unpack will choke on, so delete that line:

    $ sed -i '/^separator: *$/d' ./bind9_named.114.crash


6.  Unpack crash

    $ apport-unpack bind9_named.114.crash crash-114

    The 'crash-114/' subdirectory will now include a ThreadStacktrace file
    that hopefully(!) should have a usable backtrace.  This one looks
    great but the full stacktrace is rather TMI.

    $ wc -l crash-114/ThreadStacktrace 
    415 crash-114/ThreadStacktrace


7.  (Advanced) GDB

    $ apt-get source bind9
    $ cd bind9-9.16.1/

    $ apport-retrace --gdb /tmp/bind9-crash/bind9_named.114.crash

    This will give you an interactive gdb session on the reporter's
    coredump.  For example, with this particular crash, we can get a
    simple backtrace:

(gdb) bt 
#0  isc__nm_tcp_send (handle=0x7eff7522dbb0, region=0x7eff7d39a9b8, cb=0x7eff887675a0 <tcpdnssend_cb>, 
    cbarg=0x7eff7d39a9a8) at tcp.c:852
#1  0x00007eff88a2e707 in client_sendpkg (client=client at entry=0x7eff754c31b0, buffer=<optimized out>, 
    buffer=<optimized out>) at client.c:331
#2  0x00007eff88a2ffe9 in ns_client_send (client=client at entry=0x7eff754c31b0) at client.c:592
#3  0x00007eff88a3e9b0 in query_send (client=0x7eff754c31b0) at query.c:552
#4  0x00007eff88a469a7 in ns_query_done (qctx=qctx at entry=0x7eff85476850) at query.c:10914
#5  0x00007eff88a4dde6 in query_respond (qctx=0x7eff85476850) at query.c:7407
#6  query_prepresponse (qctx=qctx at entry=0x7eff85476850) at query.c:9906
#7  0x00007eff88a49936 in query_gotanswer (qctx=qctx at entry=0x7eff85476850, res=res at entry=0) at query.c:6823
#8  0x00007eff88a4f4c6 in query_resume (qctx=0x7eff85476850) at query.c:6121
#9  fetch_callback (task=<optimized out>, event=<optimized out>) at query.c:5703
#10 0x00007eff88770fa1 in dispatch (threadid=<optimized out>, manager=<optimized out>) at task.c:1152
#11 run (queuep=<optimized out>) at task.c:1344
#12 0x00007eff88239609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#13 0x00007eff8815a293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

(gdb) list tcp.c:852
file: "src/unix/tcp.c", line number: 852, symbol: "???"
847	src/unix/tcp.c: No such file or directory.
file: "tcp.c", line number: 852, symbol: "???"
847			 void *cbarg) {
848		isc_nmsocket_t *sock = handle->sock;
849		isc__netievent_tcpsend_t *ievent = NULL;
850		isc__nm_uvreq_t *uvreq = NULL;
851	
852		REQUIRE(sock->type == isc_nm_tcpsocket);
853	
854		uvreq = isc__nm_uvreq_get(sock->mgr, sock);
855		uvreq->uvbuf.base = (char *)region->base;
856		uvreq->uvbuf.len = region->length;
(gdb) 

(gdb) print sock
$1 = (isc_nmsocket_t *) 0x0
(gdb) print sock->mgr
Cannot access memory at address 0x10

The user supplied a second *.115.crash file, that you could use the
above process to unpack and examine and see how it compares.


Hope this helps!
Bryce