[Bug 1768824] Re: [SRU] service_statuses table running full in Designate database
Brian Murray
brian at ubuntu.com
Wed Jul 10 20:07:32 UTC 2019
Hello Dmitry, or anyone else affected,
Accepted designate into bionic-proposed. The package will build now and
be available at
https://launchpad.net/ubuntu/+source/designate/1:6.0.1-0ubuntu1.1 in a
few hours, and then in the -proposed repository.
Please help us by testing this new package. See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed. Your feedback will aid us getting this
update out to other Ubuntu users.
If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested and change the tag from
verification-needed-bionic to verification-done-bionic. If it does not
fix the bug for you, please add a comment stating that, and change the
tag to verification-failed-bionic. In either case, without details of
your testing we will not be able to proceed.
Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
advance for helping!
N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.
** Changed in: designate (Ubuntu Bionic)
Status: Triaged => Fix Committed
** Tags added: verification-needed verification-needed-bionic
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1768824
Title:
[SRU] service_statuses table running full in Designate database
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive queens series:
Triaged
Status in Designate:
Fix Released
Status in designate package in Ubuntu:
Fix Released
Status in designate source package in Bionic:
Fix Committed
Bug description:
[Impact]
This patch is required to prevent pool-manager from creating unbounded amounts of status logs in the service_statuses table triggered by having > 1 log in there.
[Test Case]
* deploy openstack queens with designate
* mysql> select count(*) from service_statuses where service_name="pool_manager"; should return 1
* try to add an extra entry:
ts=date '+%Y-%m-%d %H:%M:%S'
svc_host=`mysql -B -h$host -u${service} -p$passwd ${service} -e 'select hostname from service_statuses where service_name="pool_manager";'| tail -n 1`
mysql -h$host -u${service} -p$passwd ${service} -e "insert into service_statuses values ('1234', '$ts', '$ts', 'pool_manager', '$svc_host', '$ts', 'UP', '{}', '{}');"
* this should fail since the hostname/servicename columns should now be a unique contraint
* can also check this with:
mysql -h$host -u${service} -p$passwd ${service} -e "select * from
INFORMATION_SCHEMA.TABLE_CONSTRAINTS where CONSTRAINT_TYPE='UNIQUE';"|
grep service_statuses
[Regression Potential]
if the table already has multiple records for pool_manager in the service_statuses table, it will be necessary to (manually) delete all but one record in order for the upgrade to succeed.
----------------------------------------------------------------------------
Hi,
The service_statuses table in Designate database is running full of
records in our deployment:
MariaDB [designate]> select count(*) from service_statuses;
+----------+
| count(*) |
+----------+
| 24474342 |
+----------+
1 row in set (7 min 19.09 sec)
We got millions of rows in just couple of month. The problem is that the same services running on the same hosts create new record (instead of updating existing) during status report to Designate.
This is how it looks in DB:
MariaDB [designate]> select * from service_statuses;
+----------------------------------+---------------------+---------------------+--------------+------------------------------------+---------------------+--------+-------+--------------+
| id | created_at | updated_at | service_name | hostname | heartbeated_at | status | stats | capabilities |
+----------------------------------+---------------------+---------------------+--------------+------------------------------------+---------------------+--------+-------+--------------+
| 0dde2b5f228549d5995cb0338841bd50 | 2018-05-02 12:06:03 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:06:03 | UP | {} | {} |
| 0e311d3000d8403d97066eba619490a3 | 2018-05-02 12:05:14 | NULL | api | designate-api-2042646259-6090v | 2018-05-02 12:05:13 | UP | {} | {} |
| 168448cd97cd428ea19318243570482c | 2018-05-02 12:05:48 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:05:48 | UP | {} | {} |
| 1685d7f80d8c4f75b052680e5e2f40ae | 2018-05-02 12:05:59 | NULL | api | designate-api-2042646259-6090v | 2018-05-02 12:05:58 | UP | {} | {} |
| 192275eb33854b4091b981b0c32d04f7 | 2018-05-02 12:05:41 | NULL | worker | designate-worker-3446544-7fzqx | 2018-05-02 12:05:35 | UP | {} | {} |
| 1e465011f21f47f096b54005675e8011 | 2018-05-02 12:05:25 | NULL | mdns | designate-mdns-4198843580-lw6s2 | 2018-05-02 12:05:25 | UP | {} | {} |
| 22e0ab87b3cd4228bc191e49923d13ba | 2018-05-02 12:05:58 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:05:58 | UP | {} | {} |
| 284d6b6bf53f47358fa14750d0c2a181 | 2018-05-02 12:05:18 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:05:18 | UP | {} | {} |
| 3617ac740b9945c09f65f4b8cf0a72b5 | 2018-05-02 12:05:25 | NULL | api | designate-api-2042646259-drzws | 2018-05-02 12:05:25 | UP | {} | {} |
| 3880157829f946398159f104b13a066b | 2018-05-02 12:06:00 | NULL | mdns | designate-mdns-4198843580-lw6s2 | 2018-05-02 12:06:00 | UP | {} | {} |
The respective part of the code is:
https://github.com/openstack/designate/blob/master/designate/central/service.py#L2879-L2898
It is not exactly clear to me why the id is needed to identify the
service. The 'hostname' and 'service_name' seem to be enough - it is
rather unlikely that there are two api services are running on the
same host..
So, I've removed the id part:
if service_status.obj_attr_is_set('id'):
criterion["id"] = service_status.id
and cleaned up the service_statuses table, now it looks fine:
MariaDB [designate]> select * from service_statuses;
+----------------------------------+---------------------+---------------------+--------------+-------------------------------------+---------------------+--------+-------+--------------+
| id | created_at | updated_at | service_name | hostname | heartbeated_at | status | stats | capabilities |
+----------------------------------+---------------------+---------------------+--------------+-------------------------------------+---------------------+--------+-------+--------------+
| 1e89d10b0ddb459f8a9522ebd62fa629 | 2018-05-02 13:03:21 | 2018-05-02 13:03:37 | worker | designate-worker-2851226431-9274c | 2018-05-02 13:03:36 | UP | {} | {} |
| 8d29d50270ee48999b78e90c1b88acf8 | 2018-05-02 13:03:23 | 2018-05-02 13:03:38 | api | designate-api-1699569558-9rvg6 | 2018-05-02 13:03:38 | UP | {} | {} |
| 9602ccf55d204c7b86277e21af3d739f | 2018-05-02 13:03:18 | 2018-05-02 13:03:38 | producer | designate-producer-3892623035-6vwt0 | 2018-05-02 13:03:38 | UP | {} | {} |
| cec705e082c542cdbfa83b9ed0c2b438 | 2018-05-02 13:03:19 | 2018-05-02 13:03:39 | api | designate-api-1699569558-r018m | 2018-05-02 13:03:39 | UP | {} | {} |
| ddd48f13b67448d2949ced39d76e2958 | 2018-05-02 13:03:20 | 2018-05-02 13:03:40 | central | designate-central-2632925199-wwj7j | 2018-05-02 13:03:40 | UP | {} | {} |
| ef0ae921a78d4ac090dc74a1dc686921 | 2018-05-02 13:03:21 | 2018-05-02 13:03:41 | mdns | designate-mdns-4162010419-pvclv | 2018-05-02 13:03:41 | UP | {} | {} |
+----------------------------------+---------------------+---------------------+--------------+-------------------------------------+---------------------+--------+-------+--------------+
6 rows in set (0.00 sec)
Our Designate version is Pike, but the code seems to be the same in
master branch.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1768824/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list