[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
Dmitrii Shcherbakov
1828534 at bugs.launchpad.net
Thu Jul 4 16:44:15 UTC 2019
Managed to reproduce it (only on one unit though), this time with
debug=true - looks like it is a py2 - py3 issue:
designate/0 active idle 2/lxd/2 10.232.46.209 9001/tcp Unit is ready
hacluster-designate/0* active idle 10.232.46.209 Unit is ready and clustered
designate/1* blocked idle 3/lxd/1 10.232.46.208 9001/tcp Services not running that should be: designate-producer
hacluster-designate/1 active idle 10.232.46.208 Unit is ready and clustered
/var/log/designate/designate-producer.log
https://paste.ubuntu.com/p/pMbt2q6KTF/
2019-07-03 23:43:48.226 37919 DEBUG designate.service [-] Starting RPC server on topic 'producer' start /usr/lib/python3/dist-packages/designate/service.py:171
2019-07-03 23:43:48.252 37919 DEBUG designate.coordination [-] Starting partitioner start /usr/lib/python3/dist-packages/designate/coordination.py:213
2019-07-03 23:43:48.257 37919 ERROR oslo_service.service [-] Error starting thread.: TypeError: '<' not supported between instances of 'str' and 'bytes'
2019-07-03 23:43:48.257 37919 ERROR oslo_service.service Traceback (most recent call last):
2019-07-03 23:43:48.257 37919 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 794, in run_service
2019-07-03 23:43:48.257 37919 ERROR oslo_service.service service.start()
2019-07-03 23:43:48.257 37919 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/designate/producer/service.py", line 72, in start
2019-07-03 23:43:48.257 37919 ERROR oslo_service.service self._partitioner.start()
2019-07-03 23:43:48.257 37919 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/designate/coordination.py", line 223, in start
2019-07-03 23:43:48.257 37919 ERROR oslo_service.service self._my_partitions = self._update_partitions()[1]
2019-07-03 23:43:48.257 37919 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/designate/coordination.py", line 200, in _update_partitions
2019-07-03 23:43:48.257 37919 ERROR oslo_service.service members = sorted(list(self._get_members(self._group_id)))
2019-07-03 23:43:48.257 37919 ERROR oslo_service.service TypeError: '<' not supported between instances of 'str' and 'bytes'
systemctl status designate-producer
● designate-producer.service - OpenStack Designate DNSaaS producer
Loaded: loaded (/lib/systemd/system/designate-producer.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Wed 2019-07-03 23:43:49 UTC; 16h ago
Main PID: 37919 (code=exited, status=0/SUCCESS)
** Attachment added: "/var/log/ from designate/1 (broken)"
https://bugs.launchpad.net/charm-designate/+bug/1828534/+attachment/5275081/+files/var-log-designate-1.tar.gz
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to designate in Ubuntu.
https://bugs.launchpad.net/bugs/1828534
Title:
[19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not
running that should be: designate-producer"
Status in OpenStack Designate Charm:
Invalid
Status in designate package in Ubuntu:
New
Bug description:
Designate has to use Worker and Producer as of Rocky as the support
for zone manager and pool manager was up for removal in Rocky. This
was addressed in https://bugs.launchpad.net/charm-
designate/+bug/1773190
During a Queens to Rocky upgrade I ran into an issue with designate-
producer being down (see the details below).
I was able to successfully start designate-producer by hand. It looks
like there were several attempts to restart the service by systemd
itself ("Scheduled restart job, restart counter is at 5.") which
eventually failed when the failcount became higher than 5. Debugging
mode was disabled so I did not see anything in the producer log.
designate/0* blocked idle 2/lxd/2 10.232.46.153 9001/tcp Services not running that should be: designate-producer
hacluster-designate/0* active idle 10.232.46.153 Unit is ready and clustered
systemctl list-unit-files | grep designate-
designate-agent.service enabled
designate-api.service enabled
designate-central.service enabled
designate-mdns.service enabled
designate-producer.service enabled
designate-sink.service enabled
designate-worker.service enabled
ubuntu at juju-eeda89-2-lxd-2:~$ journalctl -u designate-producer.service
https://paste.ubuntu.com/p/WRXYvfynnd/
ubuntu at juju-eeda89-2-lxd-2:~$ pgrep -af designate
172877 bash /lib/systemd/system/jujud-unit-hacluster-designate-0/exec-start.sh
172878 bash /lib/systemd/system/jujud-unit-designate-0/exec-start.sh
172884 /var/lib/juju/tools/unit-designate-0/jujud unit --data-dir /var/lib/juju --unit-name designate/0 --debug
172886 /var/lib/juju/tools/unit-hacluster-designate-0/jujud unit --data-dir /var/lib/juju --unit-name hacluster-designate/0 --debug
392388 /usr/bin/python3.6 /usr/bin/designate-mdns --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-mdns.log
392399 /usr/bin/python3.6 /usr/bin/designate-agent --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-agent.log
392405 /usr/bin/python3.6 /usr/bin/designate-worker --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-worker.log
392411 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
392415 /usr/bin/python3.6 /usr/bin/designate-sink --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-sink.log
392423 /usr/bin/python3.6 /usr/bin/designate-api --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-api.log
392481 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
392482 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
392483 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
392484 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
----------
designate/0 unit log:
2019-05-09 00:20:38 DEBUG openstack-upgrade Setting up designate-producer (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:20:38 DEBUG openstack-upgrade Created symlink /etc/systemd/system/multi-user.target.wants/designate-producer.service → /lib/systemd/system/designate-producer.service.
# journalctl -u designate-producer
journalctl -u designate-producer | grep start
May 09 00:20:40 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:40 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 1.
May 09 00:20:41 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:41 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 2.
May 09 00:20:43 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:43 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 3.
May 09 00:20:44 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:44 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 4.
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 5.
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: Failed to start OpenStack Designate DNSaaS producer.
designate/0 unit log (continued):
2019-05-09 00:20:58 INFO juju-log Purging ['designate-pool-manager',
'designate-zone-manager', 'python-designate', 'python-memcache']
2019-05-09 00:20:59 DEBUG openstack-upgrade Removing designate-pool-manager (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:21:15 DEBUG openstack-upgrade Removing designate-zone-manager (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:21:31 DEBUG openstack-upgrade Removing python-designate (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:21:31 DEBUG openstack-upgrade update-alternatives: using
/usr/bin/python3-designate-producer to provide /usr/bin/designate-
producer (designate-producer) in auto mode
-------------------
grep producer /var/log/juju/unit-designate-0.log
2019-05-09 00:18:43 DEBUG openstack-upgrade update-alternatives: using /usr/bin/python2-designate-producer to provide /usr/bin/designate-producer (designate-producer) in auto mode
2019-05-09 00:19:58 INFO juju-log Installing ['designate-agent', 'designate-api', 'designate-central', 'designate-common', 'designate-mdns', 'designate-worker', 'designate-sink', 'designate-producer', 'bind9utils', 'python3-designate', 'python-apt', 'memcached', 'python3-memcache', 'haproxy', 'apache2'] with options: ['--option', 'Dpkg::Options::=--force-confnew', '--option', 'Dpkg::Options::=--force-confdef']
2019-05-09 00:19:58 DEBUG openstack-upgrade designate-producer designate-worker python3-amqp python3-anyjson
2019-05-09 00:20:09 DEBUG openstack-upgrade Get:112 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 designate-producer all 1:7.0.0-0ubuntu1~cloud0 [10.6 kB]
2019-05-09 00:20:20 DEBUG openstack-upgrade Selecting previously unselected package designate-producer.
2019-05-09 00:20:20 DEBUG openstack-upgrade Preparing to unpack .../105-designate-producer_1%3a7.0.0-0ubuntu1~cloud0_all.deb ...
2019-05-09 00:20:20 DEBUG openstack-upgrade Unpacking designate-producer (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:20:38 DEBUG openstack-upgrade Setting up designate-producer (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:20:38 DEBUG openstack-upgrade Created symlink /etc/systemd/system/multi-user.target.wants/designate-producer.service → /lib/systemd/system/designate-producer.service.
2019-05-09 00:21:31 DEBUG openstack-upgrade update-alternatives: using /usr/bin/python3-designate-producer to provide /usr/bin/designate-producer (designate-producer) in auto mode
To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list