[Bug 1908473] Re: rsyslog-relp: imrelp module leaves sockets in CLOSE_WAIT state which leads to file descriptor leak

Tue Jan 19 18:04:55 UTC 2021

Hello Matthew, or anyone else affected,

Accepted librelp into groovy-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/librelp/1.5.0-1ubuntu2.20.10.1 in a
few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
groovy to verification-done-groovy. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-groovy. In either case, without details of your testing we will
not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: librelp (Ubuntu Groovy)
       Status: In Progress => Fix Committed

** Tags added: verification-needed verification-needed-groovy

** Changed in: librelp (Ubuntu Focal)
       Status: In Progress => Fix Committed

** Tags added: verification-needed-focal

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1908473

Title:
  rsyslog-relp: imrelp module leaves sockets in CLOSE_WAIT state which
  leads to file descriptor leak

Status in librelp package in Ubuntu:
  Fix Released
Status in rsyslog package in Ubuntu:
  Fix Released
Status in librelp source package in Focal:
  Fix Committed
Status in rsyslog source package in Focal:
  Won't Fix
Status in librelp source package in Groovy:
  Fix Committed
Status in rsyslog source package in Groovy:
  Fix Released
Status in librelp source package in Hirsute:
  Fix Released
Status in rsyslog source package in Hirsute:
  Fix Released

Bug description:
  [Impact]

  In recent versions of rsyslog and librelp, the imrelp module leaks
  file descriptors due to a bug where it does not correctly close
  sockets, and instead, leaves them in the CLOSE_WAIT state.

  This causes rsyslogd on busy servers to eventually hit the limit of
  maximum open files allowed, which locks rsyslogd up until it is
  restarted.

  A workaround is to restart rsyslogd every month or so to manually
  close all of the open sockets.

  Only users of the imrelp module are affected, and not rsyslog users in
  general.

  [Testcase]

  Install the rsyslog-relp module like so:

  $ sudo apt install rsyslog rsyslog-relp

  Next, generate a working directory, and make a config file that loads
  the relp module.

  $ sudo mkdir /workdir
  $ cat << EOF >> ./spool.conf
  \$LocalHostName spool
  \$AbortOnUncleanConfig on
  \$PreserveFQDN on

  global(
      workDirectory="/workdir"
      maxMessageSize="256k"
  )

  main_queue(queue.type="Direct")
  module(load="imrelp")
  input(
      type="imrelp"
      name="imrelp"
      port="601"
      ruleset="spool"
      MaxDataSize="256k"
  )

  ruleset(name="spool" queue.type="direct") {
  }

  # Just so rsyslog doesn't whine that we do not have outputs
  ruleset(name="noop" queue.type="direct") {
      action(
          type="omfile"
          name="omfile"
          file="/workdir/spool.log"
      )
  }
  EOF

  Verify that the config is valid, then start a rsyslog server.

  $ sudo rsyslogd -f ./spool.conf -N9
  $ sudo rsyslogd -f ./spool.conf -i /workdir/rsyslogd.pid

  Fetch the rsyslogd PID and check for open files.

  $ RLOGPID=$(cat /workdir/rsyslogd.pid)
  $ sudo ls -l /proc/$RLOGPID/fd
  total 0
  lr-x------ 1 root root 64 Dec 17 01:22 0 -> /dev/urandom
  lrwx------ 1 root root 64 Dec 17 01:22 1 -> 'socket:[41228]'
  lrwx------ 1 root root 64 Dec 17 01:22 3 -> 'socket:[41222]'
  lrwx------ 1 root root 64 Dec 17 01:22 4 -> 'socket:[41223]'
  lrwx------ 1 root root 64 Dec 17 01:22 7 -> 'anon_inode:[eventpoll]'

  We have 3 sockets open by default. Next, use netcat to open 100
  connections:

  $ for i in {1..100} ; do nc -z 127.0.0.1 601 ; done

  Now check for open file descriptors, and there will be an extra 100 sockets
  in the list:

  $ sudo ls -l /proc/$RLOGPID/fd

  https://paste.ubuntu.com/p/f6NQVNbZcR/

  We can check the state of these sockets with:

  $ ss -t

  https://paste.ubuntu.com/p/7Ts2FbxJrg/

  The listening sockets will be in CLOSE-WAIT, and the netcat sockets
  will be in FIN-WAIT-2.

  $ ss -t | grep CLOSE-WAIT | wc -l
  100

  If you install the test package available in the following ppa:

  https://launchpad.net/~mruffell/+archive/ubuntu/sf299578-test

  When you open connections with netcat, these will be closed properly,
  and the file descriptor leak will be fixed.

  [Where problems could occur]

  If a regression were to occur, it would be limited to users of the
  imrelp module, which is a part of the rsyslogd-relp package, and
  depends on librelp.

  rsyslog-relp is not part of a default installation of rsyslog, and is
  opt in by changing a configuration file to enable imrelp.

  The changes to rsyslog implement a testcase which exercises the problematic code to ensure things are working as expected; this
  can be enabled manually on build, and has been verified to pass (#7).

  [Other]

  Upstream bug list:

  https://github.com/rsyslog/rsyslog/issues/4350
  https://github.com/rsyslog/rsyslog/issues/4005
  https://github.com/rsyslog/librelp/issues/188
  https://github.com/rsyslog/librelp/pull/193

  The following commits fix the problem:

  rsyslogd
  ========

  commit baee0bd5420649329793746f0daf87c4f59fe6a6
  Author: Andre lorbach <alorbach at adiscon.com>
  Date:   Thu Apr 9 13:00:35 2020 +0200
  Subject: testbench: Add test for imrelp to check broken session handling.
  Link: https://github.com/rsyslog/rsyslog/commit/baee0bd5420649329793746f0daf87c4f59fe6a6

  librelp
  =======

  commit 7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0
  Author: Andre lorbach <alorbach at adiscon.com>
  Date:   Mon May 11 14:59:55 2020 +0200
  Subject: fix memory leak on session break.
  Link: https://github.com/rsyslog/librelp/commit/7907c9c57f6ed94c8ce5a4e63c3c4e019f71cff0

  commit 4a6ad8637c244fd3a1caeb9a93950826f58e956a
  Author: Andre lorbach <alorbach at adiscon.com>
  Date:   Wed Apr 8 15:55:32 2020 +0200
  Subject: replsess: fix double free of sendbuf in some cases.
  Link: https://github.com/rsyslog/librelp/commit/4a6ad8637c244fd3a1caeb9a93950826f58e956a

  commit 3797944fb62273fa1164acd3104f0894b337c4d0
  Author: Ognyan Kulev <ognyan at ognyankulev.com>
  Date:   Mon Jun 15 14:10:08 2020 +0300
  Subject: Fix FD leak when socket shutdown is one-sided
  Link: https://github.com/rsyslog/librelp/commit/3797944fb62273fa1164acd3104f0894b337c4d0

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/librelp/+bug/1908473/+subscriptions