Friday bug triage

Tue Jan 18 18:47:04 UTC 2022

On Fri, Jan 7, 2022 at 5:10 PM Bryce Harrington
<bryce.harrington at canonical.com> wrote:
>
> On Fri, Jan 07, 2022 at 08:34:57PM +0530, Utkarsh Gupta wrote:
> > Hello,
> >
> > On Fri, Jan 7, 2022 at 7:56 PM Utkarsh Gupta
> > <utkarsh.gupta at canonical.com> wrote:
> > >   LP: #1954854 - (Incomplete) [bind9] - bind
> > >   9.16.1-ubuntu on ubuntu 20.04 randomly
> > >   exits with [...]
> > >
> > > The bug reporter got back with some logs
> > > and files collected via apport. Since the
> > > bug is not easily reproducible, I am not quite
> > > sure what's the best way forward. That said,
> > > me/Paride are looking into it.
> >
> > Here's an update: we couldn't find anything worthwhile from the
> > upstream tracker or changelog or anything and neither of us have
> > enough experience with crash files.
>
> Here's a paint-by-numbers way to get stacktraces from .crash files:
>
> 0.  Enable debug symbols.  Various ways to do this, one way is to append
>     ddebs to your apt sources:
>
>     $ echo "deb http://ddebs.ubuntu.com focal main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
>     $ echo "deb http://ddebs.ubuntu.com focal-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
>
>     $ sudo apt install ubuntu-dbgsym-keyring
>     $ sudo apt update
>
> 1.  Install package with debug symbols
>
>     $ sudo apt install bind9 bind9*-dbgsym

Note this (using ddebs repo) only works if the crash resulted with the
latest version of all associated packages; in a normal support
situation where at least some package versions (that are relevant to
what caused the crash, e.g. application package and dep libs) are not
the latest, the ddebs repo unfortunately isn't useful since it only
provides the dbgsyms for the latest package versions.

It's frequently more helpful to simply use pull-lp-ddebs to get the
specific dbgsyms for the package versions that generated the crash
file.

Of course, hopefully eventually someone (*ahem* sergiodj) will get an
Ubuntu debuginfod server set up, which would allow direct access to
the appropriate dbgsyms, but that's not available yet.

>
>     Often there are also secondary libs or deps with more syms we'll
>     need.  This examines the binary for other possible dbgsym's:
>
>     $ sudo apt install debian-goodies
>     $ sudo apt install $(find-dbgsym-packages /usr/sbin/named 2>/dev/null)
>
>     The following NEW packages will be installed:
>       libcap2-dbgsym libcom-err2-dbgsym libgcc-s1-dbgsym libicu67-dbgsym libjson-c5-dbgsym libkeyutils1-dbgsym
>       libkrb5-dbg liblmdb0-dbgsym liblzma5-dbgsym libmaxminddb0-dbgsym libxml2-dbgsym zlib1g-dbgsym
>
>
> 2.  Download the .crash file to a tmp directory
>
>     $ mkdir /tmp/bind9-crash
>     $ cd /tmp/bind9-crash
>     $ wget https://bugs.launchpad.net/ubuntu/+source/bind9/+bug/1954854/+attachment/5551855/+files/_usr_sbin_named.114.crash
>
> 3.  Register the package with the crash file
>
>     $ cat <(echo "Package: bind9") _usr_sbin_named.114.crash > bind9_named.114.crash
>
>     Apport-retrace complains if the package isn't specified, but apport
>     hooks don't insert it, so you have to do it manually.  ¯\_(ツ)_/¯
>
> 4.  Doublecheck that your installed version matches the reporter's
>     version exactly:
>
>     $ apt-cache policy bind9 | grep '^  Installed'
>       Installed: 1:9.16.1-0ubuntu2.9
>
>     Unfortunately the .crash file doesn't include the version number,
>     but the LP bug report will show it:
>
>      Package: bind9 1:9.16.1-0ubuntu2.9
>
> 5.  Retrace symbols
>
>     $ sudo apt-get install apport-retrace
>     $ apport-retrace bind9_named.114.crash
>
>     The above command inserted an empty 'separator: ' line into the crash
>     file, which apport-unpack will choke on, so delete that line:
>
>     $ sed -i '/^separator: *$/d' ./bind9_named.114.crash
>
>
> 6.  Unpack crash
>
>     $ apport-unpack bind9_named.114.crash crash-114
>
>     The 'crash-114/' subdirectory will now include a ThreadStacktrace file
>     that hopefully(!) should have a usable backtrace.  This one looks
>     great but the full stacktrace is rather TMI.
>
>     $ wc -l crash-114/ThreadStacktrace
>     415 crash-114/ThreadStacktrace
>
>
> 7.  (Advanced) GDB
>
>     $ apt-get source bind9
>     $ cd bind9-9.16.1/
>
>     $ apport-retrace --gdb /tmp/bind9-crash/bind9_named.114.crash
>
>     This will give you an interactive gdb session on the reporter's
>     coredump.  For example, with this particular crash, we can get a
>     simple backtrace:
>
> (gdb) bt
> #0  isc__nm_tcp_send (handle=0x7eff7522dbb0, region=0x7eff7d39a9b8, cb=0x7eff887675a0 <tcpdnssend_cb>,
>     cbarg=0x7eff7d39a9a8) at tcp.c:852
> #1  0x00007eff88a2e707 in client_sendpkg (client=client at entry=0x7eff754c31b0, buffer=<optimized out>,
>     buffer=<optimized out>) at client.c:331
> #2  0x00007eff88a2ffe9 in ns_client_send (client=client at entry=0x7eff754c31b0) at client.c:592
> #3  0x00007eff88a3e9b0 in query_send (client=0x7eff754c31b0) at query.c:552
> #4  0x00007eff88a469a7 in ns_query_done (qctx=qctx at entry=0x7eff85476850) at query.c:10914
> #5  0x00007eff88a4dde6 in query_respond (qctx=0x7eff85476850) at query.c:7407
> #6  query_prepresponse (qctx=qctx at entry=0x7eff85476850) at query.c:9906
> #7  0x00007eff88a49936 in query_gotanswer (qctx=qctx at entry=0x7eff85476850, res=res at entry=0) at query.c:6823
> #8  0x00007eff88a4f4c6 in query_resume (qctx=0x7eff85476850) at query.c:6121
> #9  fetch_callback (task=<optimized out>, event=<optimized out>) at query.c:5703
> #10 0x00007eff88770fa1 in dispatch (threadid=<optimized out>, manager=<optimized out>) at task.c:1152
> #11 run (queuep=<optimized out>) at task.c:1344
> #12 0x00007eff88239609 in start_thread (arg=<optimized out>) at pthread_create.c:477
> #13 0x00007eff8815a293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
>
> (gdb) list tcp.c:852
> file: "src/unix/tcp.c", line number: 852, symbol: "???"
> 847     src/unix/tcp.c: No such file or directory.
> file: "tcp.c", line number: 852, symbol: "???"
> 847                      void *cbarg) {
> 848             isc_nmsocket_t *sock = handle->sock;
> 849             isc__netievent_tcpsend_t *ievent = NULL;
> 850             isc__nm_uvreq_t *uvreq = NULL;
> 851
> 852             REQUIRE(sock->type == isc_nm_tcpsocket);
> 853
> 854             uvreq = isc__nm_uvreq_get(sock->mgr, sock);
> 855             uvreq->uvbuf.base = (char *)region->base;
> 856             uvreq->uvbuf.len = region->length;
> (gdb)
>
> (gdb) print sock
> $1 = (isc_nmsocket_t *) 0x0
> (gdb) print sock->mgr
> Cannot access memory at address 0x10
>
> The user supplied a second *.115.crash file, that you could use the
> above process to unpack and examine and see how it compares.
>
>
> Hope this helps!
> Bryce
>
> --
> ubuntu-server mailing list
> ubuntu-server at lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
> More info: https://wiki.ubuntu.com/ServerTeam