[Bug 1883112] Re: rbd-target-api crashes with python TypeError

Thu Jun 2 16:34:21 UTC 2022

The model used to verify the above looks like:

Model       Controller        Cloud/Region             Version  SLA          Timestamp
rbd-mirror  icey-serverstack  serverstack/serverstack  2.9.29   unsupported  16:34:08Z

App         Version  Status  Scale  Charm       Channel    Rev  Exposed  Message
ceph-iscsi           active      2  ceph-iscsi               1  no       Unit is ready
ceph-mon    16.2.7   active      3  ceph-mon    candidate   71  no       Unit is ready and clustered
ceph-osd    16.2.7   active      3  ceph-osd    candidate  511  no       Unit is ready (1 OSD)

Unit           Workload  Agent  Machine  Public address  Ports  Message
ceph-iscsi/0   active    idle   0        10.5.1.220             Unit is ready
ceph-iscsi/1*  active    idle   1        10.5.1.10              Unit is ready
ceph-mon/0     active    idle   2        10.5.1.205             Unit is ready and clustered
ceph-mon/1     active    idle   3        10.5.0.166             Unit is ready and clustered
ceph-mon/2*    active    idle   4        10.5.1.50              Unit is ready and clustered
ceph-osd/0*    active    idle   5        10.5.1.149             Unit is ready (1 OSD)
ceph-osd/1     active    idle   6        10.5.1.210             Unit is ready (1 OSD)
ceph-osd/2     active    idle   7        10.5.2.178             Unit is ready (1 OSD)

Machine  State    DNS         Inst id                               Series  AZ    Message
0        started  10.5.1.220  a48a6a91-8d18-41af-86fb-0fb12c75cc40  impish  nova  ACTIVE
1        started  10.5.1.10   1e786e43-119c-42eb-b95d-bdc5aa37d8e7  impish  nova  ACTIVE
2        started  10.5.1.205  c60362a7-fe77-4267-87e7-02bfcdf68ace  impish  nova  ACTIVE
3        started  10.5.0.166  bad1f0ac-896e-45d1-bc53-67caf48fecf7  impish  nova  ACTIVE
4        started  10.5.1.50   cd432ac9-3442-497f-83e0-5816d6c4be07  impish  nova  ACTIVE
5        started  10.5.1.149  220e20cc-11dd-4152-adb2-e2a7cf8b5caf  impish  nova  ACTIVE
6        started  10.5.1.210  05921fb6-1e97-40ca-9b89-a03efd5b8af3  impish  nova  ACTIVE
7        started  10.5.2.178  1d968bfc-97bf-491d-9778-76cc2fe2e1cc  impish  nova  ACTIVE

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph-iscsi in Ubuntu.
https://bugs.launchpad.net/bugs/1883112

Title:
  rbd-target-api crashes with python TypeError

Status in ceph-iscsi package in Ubuntu:
  Fix Released
Status in ceph-iscsi source package in Focal:
  Incomplete
Status in ceph-iscsi source package in Impish:
  Fix Committed

Bug description:
  [Impact]

   * rbd-target-api service fails to start if there is a blocklist
     entry for the unit making the service unavailable.

   * When the rbd-target-api service starts it checks if any of the
     ip addresses on the machine it is running on are listed as
     blocked. If there are entries it tries to remove them. In the
     process of removing the entries the code attempts to test whether
     a string is in the result of a subprocess.check_output call. This 
     would have worked in python2 but with python3 a byte like object
     is returned and check now throws a TypeError. This fix, taken from
     upstream, changes the code to remove the `in` check and replace it
     with a try/except

  [Test Plan]

   If an existing ceph-iscsi deployment is available then skip to
   step 3.

   1) Deploy the bundle below (tested with OpenStack providor).

  series: focal
  applications:
    ceph-iscsi:
      charm: cs:ceph-iscsi
      num_units: 2
    ceph-osd:
      charm: ch:ceph-osd
      num_units: 3
      storage:
        osd-devices: 'cinder,10G'
      options:
        osd-devices: '/dev/test-non-existent'
      channel: latest/edge
    ceph-mon:
      charm: ch:ceph-mon
      num_units: 3
      options:
        monitor-count: '3'
      channel: latest/edge
  relations:
    - - 'ceph-mon:client'
      - 'ceph-iscsi:ceph-client'
    - - 'ceph-osd:mon'
      - 'ceph-mon:osd'

   2) Connect to ceph-iscsi unit:

  juju ssh -m zaza-a1d88053ab85 ceph-iscsi/0

   3) Stop rbd-target-api via systemd to make test case clearer:

  sudo systemctl stop rbd-target-api

   4) Add 2 blocklist entries for this unit (due to another issue the
  ordering of the output from `osd blacklist ls` matters which can lead
  to the reproduction of this bug being intermittent. To avoid this add
  two entries which ensures there is always an entry for this node in
  the list of blocklist entries to be removed).

  sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist add $(hostname --all-ip-addresses | awk '{print $1}'):0/1
  sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist add $(hostname --all-ip-addresses | awk '{print $1}'):0/2
  sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist ls
    listed 2 entries
    172.20.0.135:0/2 2022-02-23T11:14:54.850352+0000
    172.20.0.135:0/1 2022-02-23T11:14:52.502592+0000

   5) Attempt to start service:

  sudo /usr/bin/python3 /usr/bin/rbd-target-api
  Traceback (most recent call last):
    File "/usr/bin/rbd-target-api", line 2952, in <module>
      main()
    File "/usr/bin/rbd-target-api", line 2862, in main
      osd_state_ok = ceph_gw.osd_blacklist_cleanup()
    File "/usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py", line 111, in osd_blacklist_cleanup
      rm_ok = self.ceph_rm_blacklist(blacklist_entry.split(' ')[0])
    File "/usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py", line 46, in ceph_rm_blacklist
      if ("un-blacklisting" in result) or ("isn't blacklisted" in result):
  TypeError: a bytes-like object is required, not 'str'

  [Where problems could occur]

   * Problems could occur with the service starting as this blocklist check is done at startup.

   * Blocklist entries could fail to be removed.

  Old bug description:

  $ lsb_release -rd
  Description:	Ubuntu 20.04 LTS
  Release:	20.04

  $ dpkg -S /usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py
  ceph-iscsi: /usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py

  $ apt-cache policy ceph-iscsi
  ceph-iscsi:
    Installed: 3.4-0ubuntu2
    Candidate: 3.4-0ubuntu2
    Version table:
   *** 3.4-0ubuntu2 500
          500 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 Packages
          500 http://de.archive.ubuntu.com/ubuntu focal/universe i386 Packages
          100 /var/lib/dpkg/status

  On second startup after a reboot, rbd-target-api crashes with a
  TypeError:

  Traceback (most recent call last):
    File "/usr/bin/rbd-target-api", line 2952, in <module>
      main()
    File "/usr/bin/rbd-target-api", line 2862, in main
      osd_state_ok = ceph_gw.osd_blacklist_cleanup()
    File "/usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py", line 110, in osd_blacklist_cleanup
      rm_ok = self.ceph_rm_blacklist(blacklist_entry.split(' ')[0])
    File "/usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py", line 46, in ceph_rm_blacklist
      if ("un-blacklisting" in result) or ("isn't blacklisted" in result):
  TypeError: a bytes-like object is required, not 'str'

  Upstream has a fix: https://github.com/ceph/ceph-
  iscsi/commit/e48dcb0d3099b27595b9f4433da8493f5edb9206#diff-f1381af4114a1e777ef5e8b7b7452a01

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph-iscsi/+bug/1883112/+subscriptions