[Bug 1883112] Re: rbd-target-api crashes with python TypeError
Robie Basak
1883112 at bugs.launchpad.net
Wed Mar 2 15:42:08 UTC 2022
The obvious minimal fix would be something like:
- if ("un-blacklisting" in result) or ("isn't blacklisted" in result):
+ if (b"un-blacklisting" in result) or (b"isn't blacklisted" in result):
...since in Ubuntu Focal and Impish we know that we're running on Python
3, so 2+3 compatibility isn't required.
Cherry-picking the upstream change is additionally making new
assumptions about the behaviour of ceph's CLI I think - that it returns
the correct exit status. Have you checked to ensure that the behaviour
of the version of ceph in Focal and Impish is as expected in both the
success and failure cases? If so then since presumably this code never
worked on Focal and Impish before, then the cherry-pick is fine. But if
you want to use the upstream (non-minimal) fix that makes this
additional assumption then think it needs to be confirmed before
accepting.
For reference, SRU policy says the following. In this case I think
either way is fine providing that assumptions are checked.
> In line with this, the requirements for stable updates are not
necessarily the same as those in the development release. When preparing
future releases, one of our goals is to construct the most elegant and
maintainable system possible, and this often involves fundamental
improvements to the system's architecture, rearranging packages to avoid
bundled copies of other software so that we only have to maintain it in
one place, and so on. However, once we have completed a release, the
priority is normally to minimise risk caused by changes not explicitly
required to fix qualifying bugs, and this tends to be well-correlated
with minimising the size of those changes. As such, the same bug may
need to be fixed in different ways in stable and development releases.
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph-iscsi in Ubuntu.
https://bugs.launchpad.net/bugs/1883112
Title:
rbd-target-api crashes with python TypeError
Status in ceph-iscsi package in Ubuntu:
Fix Released
Status in ceph-iscsi source package in Focal:
Triaged
Status in ceph-iscsi source package in Impish:
Triaged
Bug description:
[Impact]
* rbd-target-api service fails to start if there is a blocklist
entry for the unit making the service unavailable.
* When the rbd-target-api service starts it checks if any of the
ip addresses on the machine it is running on are listed as
blocked. If there are entries it tries to remove them. In the
process of removing the entries the code attempts to test whether
a string is in the result of a subprocess.check_output call. This
would have worked in python2 but with python3 a byte like object
is returned and check now throws a TypeError. This fix, taken from
upstream, changes the code to remove the `in` check and replace it
with a try/except
[Test Plan]
If an existing ceph-iscsi deployment is available then skip to
step 3.
1) Deploy the bundle below (tested with OpenStack providor).
series: focal
applications:
ceph-iscsi:
charm: cs:ceph-iscsi
num_units: 2
ceph-osd:
charm: ch:ceph-osd
num_units: 3
storage:
osd-devices: 'cinder,10G'
options:
osd-devices: '/dev/test-non-existent'
channel: latest/edge
ceph-mon:
charm: ch:ceph-mon
num_units: 3
options:
monitor-count: '3'
channel: latest/edge
relations:
- - 'ceph-mon:client'
- 'ceph-iscsi:ceph-client'
- - 'ceph-osd:mon'
- 'ceph-mon:osd'
2) Connect to ceph-iscsi unit:
juju ssh -m zaza-a1d88053ab85 ceph-iscsi/0
3) Stop rbd-target-api via systemd to make test case clearer:
sudo systemctl stop rbd-target-api
4) Add 2 blocklist entries for this unit (due to another issue the
ordering of the output from `osd blacklist ls` matters which can lead
to the reproduction of this bug being intermittent. To avoid this add
two entries which ensures there is always an entry for this node in
the list of blocklist entries to be removed).
sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist add $(hostname --all-ip-addresses | awk '{print $1}'):0/1
sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist add $(hostname --all-ip-addresses | awk '{print $1}'):0/2
sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist ls
listed 2 entries
172.20.0.135:0/2 2022-02-23T11:14:54.850352+0000
172.20.0.135:0/1 2022-02-23T11:14:52.502592+0000
5) Attempt to start service:
sudo /usr/bin/python3 /usr/bin/rbd-target-api
Traceback (most recent call last):
File "/usr/bin/rbd-target-api", line 2952, in <module>
main()
File "/usr/bin/rbd-target-api", line 2862, in main
osd_state_ok = ceph_gw.osd_blacklist_cleanup()
File "/usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py", line 111, in osd_blacklist_cleanup
rm_ok = self.ceph_rm_blacklist(blacklist_entry.split(' ')[0])
File "/usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py", line 46, in ceph_rm_blacklist
if ("un-blacklisting" in result) or ("isn't blacklisted" in result):
TypeError: a bytes-like object is required, not 'str'
[Where problems could occur]
* Problems could occur with the service starting as this blocklist check is done at startup.
* Blocklist entries could fail to be removed.
Old bug description:
$ lsb_release -rd
Description: Ubuntu 20.04 LTS
Release: 20.04
$ dpkg -S /usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py
ceph-iscsi: /usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py
$ apt-cache policy ceph-iscsi
ceph-iscsi:
Installed: 3.4-0ubuntu2
Candidate: 3.4-0ubuntu2
Version table:
*** 3.4-0ubuntu2 500
500 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 Packages
500 http://de.archive.ubuntu.com/ubuntu focal/universe i386 Packages
100 /var/lib/dpkg/status
On second startup after a reboot, rbd-target-api crashes with a
TypeError:
Traceback (most recent call last):
File "/usr/bin/rbd-target-api", line 2952, in <module>
main()
File "/usr/bin/rbd-target-api", line 2862, in main
osd_state_ok = ceph_gw.osd_blacklist_cleanup()
File "/usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py", line 110, in osd_blacklist_cleanup
rm_ok = self.ceph_rm_blacklist(blacklist_entry.split(' ')[0])
File "/usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py", line 46, in ceph_rm_blacklist
if ("un-blacklisting" in result) or ("isn't blacklisted" in result):
TypeError: a bytes-like object is required, not 'str'
Upstream has a fix: https://github.com/ceph/ceph-
iscsi/commit/e48dcb0d3099b27595b9f4433da8493f5edb9206#diff-f1381af4114a1e777ef5e8b7b7452a01
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph-iscsi/+bug/1883112/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list