[Bug 1908473] Re: rsyslog-relp: imrelp module leaves sockets in CLOSE_WAIT state which leads to file descriptor leak

Wed Jan 13 18:34:58 UTC 2021

This bug was fixed in the package librelp - 1.9.0-1ubuntu1

---------------
librelp (1.9.0-1ubuntu1) hirsute; urgency=medium

  * Merge from Debian unstable. (LP: #1910307)
    Remaining changes:
    [ William Grant ]
    - d/p/shrink-receiver-abort-tests.sh: Reduce message count
      so tests pass on slow platforms like riscv64.
    Dropped changes:
    - d/p/python2.diff: No longer needed due to tests being
      ported to python3 in recent versions.
  * Fix file descriptor leak as sockets are stuck in CLOSE_WAIT
    due to not being closed properly due to memory leak. (LP: #1908473)

 -- Matthew Ruffell <matthew.ruffell at canonical.com>  Thu, 26 Nov 2020
21:52:43 +0000

** Changed in: librelp (Ubuntu Hirsute)
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1908473

Title:
  rsyslog-relp: imrelp module leaves sockets in CLOSE_WAIT state which
  leads to file descriptor leak

Status in librelp package in Ubuntu:
  Fix Released
Status in rsyslog package in Ubuntu:
  Fix Released
Status in librelp source package in Focal:
  In Progress
Status in rsyslog source package in Focal:
  Won't Fix
Status in librelp source package in Groovy:
  In Progress
Status in rsyslog source package in Groovy:
  Fix Released
Status in librelp source package in Hirsute:
  Fix Released
Status in rsyslog source package in Hirsute:
  Fix Released

Bug description:
  [Impact]

  In recent versions of rsyslog and librelp, the imrelp module leaks
  file descriptors due to a bug where it does not correctly close
  sockets, and instead, leaves them in the CLOSE_WAIT state.

  This causes rsyslogd on busy servers to eventually hit the limit of
  maximum open files allowed, which locks rsyslogd up until it is
  restarted.

  A workaround is to restart rsyslogd every month or so to manually
  close all of the open sockets.

  Only users of the imrelp module are affected, and not rsyslog users in
  general.

  [Testcase]

  Install the rsyslog-relp module like so:

  $ sudo apt install rsyslog rsyslog-relp

  Next, generate a working directory, and make a config file that loads
  the relp module.

  $ sudo mkdir /workdir
  $ cat << EOF >> ./spool.conf
  \$LocalHostName spool
  \$AbortOnUncleanConfig on
  \$PreserveFQDN on

  global(
      workDirectory="/workdir"
      maxMessageSize="256k"
  )

  main_queue(queue.type="Direct")
  module(load="imrelp")
  input(
      type="imrelp"
      name="imrelp"
      port="601"
      ruleset="spool"
      MaxDataSize="256k"
  )

  ruleset(name="spool" queue.type="direct") {
  }

  # Just so rsyslog doesn't whine that we do not have outputs
  ruleset(name="noop" queue.type="direct") {
      action(
          type="omfile"
          name="omfile"
          file="/workdir/spool.log"
      )
  }
  EOF

  Verify that the config is valid, then start a rsyslog server.

  $ sudo rsyslogd -f ./spool.conf -N9
  $ sudo rsyslogd -f ./spool.conf -i /workdir/rsyslogd.pid

  Fetch the rsyslogd PID and check for open files.

  $ RLOGPID=$(cat /workdir/rsyslogd.pid)
  $ sudo ls -l /proc/$RLOGPID/fd
  total 0
  lr-x------ 1 root root 64 Dec 17 01:22 0 -> /dev/urandom
  lrwx------ 1 root root 64 Dec 17 01:22 1 -> 'socket:[41228]'
  lrwx------ 1 root root 64 Dec 17 01:22 3 -> 'socket:[41222]'
  lrwx------ 1 root root 64 Dec 17 01:22 4 -> 'socket:[41223]'
  lrwx------ 1 root root 64 Dec 17 01:22 7 -> 'anon_inode:[eventpoll]'

  We have 3 sockets open by default. Next, use netcat to open 100
  connections:

  $ for i in {1..100} ; do nc -z 127.0.0.1 601 ; done

  Now check for open file descriptors, and there will be an extra 100 sockets
  in the list:

  $ sudo ls -l /proc/$RLOGPID/fd

  https://paste.ubuntu.com/p/f6NQVNbZcR/

  We can check the state of these sockets with:

  $ ss -t

  https://paste.ubuntu.com/p/7Ts2FbxJrg/

  The listening sockets will be in CLOSE-WAIT, and the netcat sockets
  will be in FIN-WAIT-2.

  $ ss -t | grep CLOSE-WAIT | wc -l
  100

  If you install the test package available in the following ppa:

  https://launchpad.net/~mruffell/+archive/ubuntu/sf299578-test

  When you open connections with netcat, these will be closed properly,
  and the file descriptor leak will be fixed.

  [Where problems could occur]

  If a regression were to occur, it would be limited to users of the
  imrelp module, which is a part of the rsyslogd-relp package, and
  depends on librelp.

  rsyslog-relp is not part of a default installation of rsyslog, and is
  opt in by changing a configuration file to enable imrelp.

  The changes to rsyslog implement a testcase which exercises the
  problematic code to ensure things are working as expected, and should
  run during autopkgtest time.

  [Other]

  Upstream bug list:

  https://github.com/rsyslog/rsyslog/issues/4350
  https://github.com/rsyslog/rsyslog/issues/4005
  https://github.com/rsyslog/librelp/issues/188
  https://github.com/rsyslog/librelp/pull/193

  The following commits fix the problem:

  rsyslogd
  ========

  commit baee0bd5420649329793746f0daf87c4f59fe6a6
  Author: Andre lorbach <alorbach at adiscon.com>
  Date:   Thu Apr 9 13:00:35 2020 +0200
  Subject: testbench: Add test for imrelp to check broken session handling.
  Link: https://github.com/rsyslog/rsyslog/commit/baee0bd5420649329793746f0daf87c4f59fe6a6

  librelp
  =======

  commit 7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0
  Author: Andre lorbach <alorbach at adiscon.com>
  Date:   Mon May 11 14:59:55 2020 +0200
  Subject: fix memory leak on session break.
  Link: https://github.com/rsyslog/librelp/commit/7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0

  commit 4a6ad8637c244fd3a1caeb9a93950826f58e956a
  Author: Andre lorbach <alorbach at adiscon.com>
  Date:   Wed Apr 8 15:55:32 2020 +0200
  Subject: replsess: fix double free of sendbuf in some cases.
  Link: https://github.com/rsyslog/librelp/commit/4a6ad8637c244fd3a1caeb9a93950826f58e956a

  commit 3797944fb62273fa1164acd3104f0894b337c4d0
  Author: Ognyan Kulev <ognyan at ognyankulev.com>
  Date:   Mon Jun 15 14:10:08 2020 +0300
  Subject: Fix FD leak when socket shutdown is one-sided
  Link: https://github.com/rsyslog/librelp/commit/3797944fb62273fa1164acd3104f0894b337c4d0

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/librelp/+bug/1908473/+subscriptions