[Bug 1969775] Re: rbd-target-api crashes with `blacklist removal failed`

[Chris MacNaughton]

Attached is a debdiff adding support for the newer "blocklist" syntax

** Patch added: "lp1969775.debdiff"
   https://bugs.launchpad.net/ubuntu/+source/ceph-iscsi/+bug/1969775/+attachment/5597267/+files/lp1969775.debdiff

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph-iscsi in Ubuntu.
https://bugs.launchpad.net/bugs/1969775

Title:
  rbd-target-api crashes with `blacklist removal failed`

Status in ceph-iscsi package in Ubuntu:
  New

Bug description:
  [Impact]
   * ceph-iscsi on Focal talking to a Pacific or later Ceph cluster

   * rbd-target-api service fails to start if there is a blocklist
     entry for the unit.

   * When the rbd-target-api service starts it checks if any of the
     ip addresses on the machine it is running on are listed as
     blocked. If there are entries it tries to remove them. When it
     issues the block removal command it checks stdout from the
     removal command looking for the string `un-blacklisting`.
     However from Pacific onward a successful unblocking returns
     `un-blocklisting` instead (https://github.com/ceph/ceph/commit/dfd01d765304ed8783cef613930e65980d9aee23)

  [Test Plan]

   If an existing ceph-iscsi deployment is available then skip to
   step 3.

   1) Deploy the bundle below (tested with OpenStack provider).

  series: focal
  applications:
    ceph-iscsi:
      charm: cs:ceph-iscsi
      num_units: 2
    ceph-osd:
      charm: ch:ceph-osd
      num_units: 3
      storage:
        osd-devices: 'cinder,10G'
      options:
        osd-devices: '/dev/test-non-existent'
        source: yoga
      channel: latest/edge
    ceph-mon:
      charm: ch:ceph-mon
      num_units: 3
      options:
        monitor-count: '3'
        source: yoga
      channel: latest/edge
  relations:
    - - 'ceph-mon:client'
      - 'ceph-iscsi:ceph-client'
    - - 'ceph-osd:mon'
      - 'ceph-mon:osd'

   2) Connect to ceph-iscsi unit:

  juju ssh -m zaza-a1d88053ab85 ceph-iscsi/0

   3) Stop rbd-target-api via systemd to make test case clearer:

  sudo systemctl stop rbd-target-api

   4) Add 2 blocklist entries for this unit (due to another issue the
  ordering of the output from `osd blacklist ls` matters which can lead
  to the reproduction of this bug being intermittent. To avoid this add
  two entries which ensures there is always an entry for this node in
  the list of blocklist entries to be removed).

  sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist add $(hostname --all-ip-addresses | awk '{print $1}'):0/1
  sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist add $(hostname --all-ip-addresses | awk '{print $1}'):0/2
  sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist ls
    listed 2 entries
    172.20.0.135:0/2 2022-02-23T11:14:54.850352+0000
    172.20.0.135:0/1 2022-02-23T11:14:52.502592+0000

   5) Attempt to start service:

  sudo /usr/bin/python3 /usr/bin/rbd-target-api

  At this point the process should be running in the foreground but instead
  it will die. The log from the service will have an entry like:

  2022-04-21 12:35:21,695 CRITICAL [gateway.py:51:ceph_rm_blacklist()] -
  blacklist removal failed. Run 'ceph -n client.ceph-iscsi --conf
  /etc/ceph/iscsi/ceph.conf osd blacklist rm 172.20.0.156:0/1'

  [Where problems could occur]

   * Problems could occur with the service starting as this blocklist
  check is done at startup.

   * Blocklist entries could fail to be removed.

  
  This issue is very similar to Bug #1883112

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph-iscsi/+bug/1969775/+subscriptions