[Bug 2043114] Re: sshd segmentation fault on 20.04.6 (focal)

Wed Nov 15 19:59:35 UTC 2023

Thank you for providing more information.

Unfortunately I am still unable to reproduce the problem.  I tried using
a container and a VM, to no avail.

But I did open the coredump:

(gdb) bt
#0  _int_free (av=av at entry=0x7fcbaccd8b80 <main_arena>, p=p at entry=0x558afb81e0c0, have_lock=<optimized out>, have_lock at entry=1) at malloc.c:4341
#1  0x00007fcbacb84f22 in _int_realloc (av=av at entry=0x7fcbaccd8b80 <main_arena>, oldp=oldp at entry=0x558afb81e070, oldsize=oldsize at entry=8208, nb=80) at malloc.c:4644
#2  0x00007fcbacb86fb6 in __GI___libc_realloc (oldmem=0x558afb81e080, bytes=64) at malloc.c:3226
#3  0x00007fcbacb77748 in _IO_mem_finish (fp=0x558afb805e80, dummy=<optimized out>) at memstream.c:131
#4  0x00007fcbacb6de41 in _IO_new_fclose (fp=fp at entry=0x558afb805e80) at libioP.h:948
#5  0x00007fcbacc03ddb in __vsyslog_internal (pri=<optimized out>, fmt=0x558afa13ac80 "%.500s", ap=0x7ffd8170e5c0, mode_flags=2) at ../misc/syslog.c:237
#6  0x00007fcbacc04363 in __syslog_chk (pri=pri at entry=7, flag=flag at entry=1, fmt=fmt at entry=0x558afa13ac80 "%.500s") at ../misc/syslog.c:136
#7  0x0000558afa0f8b78 in syslog (__fmt=0x558afa13ac80 "%.500s", __pri=7) at /usr/include/x86_64-linux-gnu/bits/syslog.h:31
#8  do_log (level=level at entry=SYSLOG_LEVEL_DEBUG1, fmt=<optimized out>, args=args at entry=0x7ffd8170ef00) at ../../log.c:476
#9  0x0000558afa0f8ff8 in debug (fmt=<optimized out>) at ../../log.c:229
#10 0x0000558afa0ae3fe in server_accept_loop (config_s=0x7ffd8170f050, newsock=<synthetic pointer>, sock_out=<synthetic pointer>, sock_in=<synthetic pointer>) at ../../sshd.c:1338
#11 main (ac=<optimized out>, av=<optimized out>) at ../../sshd.c:2040

This stack trace is a bit intriguing.  It's realloc that is crashing,
way too deep into glibc.  It seems to point at some weird interaction
between your setup and the memory management involved in syslog.

I spent time trying to find upstream bugs to see if there was anything
remotely similar, but couldn't find anything either.  Can you provide
more details on the setup you're using to reproduce the problem?  For
example, are you using a VM, a container, bare metal?  How many (v)CPUs?
What about memory?  If it's a container/VM, what's the underlying host?

Also, since you can reproduce the issue pretty reliably, could you
perhaps check if the same crash happens on Jammy?

Thank you.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2043114

Title:
  sshd segmentation fault on 20.04.6 (focal)

Status in openssh package in Ubuntu:
  Confirmed

Bug description:
  We have a physical server running Ubuntu 20.04.6 LTS (amd64) and openssh-server 1:8.2p1-4ubuntu0.9. Sometimes sshd crashes with a segmentation fault on remote login with key authentication:
  [193107.651745] sshd[1229630]: segfault at 5557eba6a008 ip 00007f2326a2ca53 sp 00007ffcba40c510 error 4 in libc-2.31.so[7f23269b8000+178000]

  We’ve changed only the following values in the stock sshd_config file:

  LogLevel DEBUG
  PasswordAuthentication no
  MaxStartups 100:30:100

  The server is used for automated software testing, and sometimes our test suite might make a large amount of SSH connections in a short period of time, which seems to be correlated with the crashes. But at the same time, I have to note that the connection count was not near the MaxStartups limit, and we’ve had crashes before adding that setting.
  Since the backtrace shows the debug logging function in the stack, we’re currently experimenting with using `LogLevel INFO` to try and isolate the issue.

  I am attaching the backtrace. I could provide the full dump file,
  although I am hesitant due to the possibility of private keys or other
  sensitive information leaking.

  # apt-cache policy openssh-server
  openssh-server:
    Installed: 1:8.2p1-4ubuntu0.9
    Candidate: 1:8.2p1-4ubuntu0.9
    Version table:
   *** 1:8.2p1-4ubuntu0.9 500
          500 http://mirrors.storpool.com/ubuntu/archive focal-updates/main amd64 Packages
          500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
          100 /var/lib/dpkg/status
       1:8.2p1-4 500
          500 http://mirrors.storpool.com/ubuntu/archive focal/main amd64 Packages
  --- 
  ProblemType: Bug
  ApportVersion: 2.20.11-0ubuntu27.27
  Architecture: amd64
  CasperMD5CheckResult: skip
  DistroRelease: Ubuntu 20.04
  Package: openssh-server 1:8.2p1-4ubuntu0.9
  PackageArchitecture: amd64
  ProcVersionSignature: Ubuntu 5.4.0-128.144-generic 5.4.210
  Tags:  focal
  Uname: Linux 5.4.0-128-generic x86_64
  UpgradeStatus: Upgraded to focal on 2021-01-13 (1030 days ago)
  UserGroups: N/A
  _MarkForUpload: True

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2043114/+subscriptions