[Bug 1921453] Re: multi-zone replication doesn't work

[OpenStack Infra]

Reviewed:  https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/786030
Committed: https://opendev.org/openstack/charm-ceph-radosgw/commit/15d7a9d82729758e50efb48106f8cd1f1284210c
Submitter: "Zuul (22348)"
Branch:    master

commit 15d7a9d82729758e50efb48106f8cd1f1284210c
Author: James Page <james.page at ubuntu.com>
Date:   Tue Apr 13 10:43:39 2021 +0100

    Add otp pool to broker request
    
    Ceph RADOS gateway >= Mimic has an additional metadata pool (otp).
    
    Add this to the broker request to ensure that its created correctly
    by the ceph-mon application rather than being auto-created by the
    radosgw application
    
    Change-Id: I5e9b4e449bd1bc300225d223329bb62f3a381705
    Closes-Bug: 1921453


** Changed in: charm-ceph-radosgw
       Status: In Progress => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1921453

Title:
  multi-zone replication doesn't work

Status in OpenStack ceph-radosgw charm:
  Fix Committed
Status in ceph package in Ubuntu:
  Invalid

Bug description:
  I deploy the following bundle
  https://pastebin.ubuntu.com/p/qgt55P2Hv5/

  (tried both bionic-ussuri and focal-distro origins).
  There are two radosgw instances, each is backed by its own Ceph cluster.
  Each RGW pull info correctly, however the sync status on master:
  root at juju-66e0ee-1-lxd-0:~# radosgw-admin sync status
            realm 2ce9452b-1ec6-4507-9068-4ba929071c54 (replicated)
        zonegroup f68b626c-24b7-4659-9309-1a80e2986568 (us)
             zone c2d6f2e0-28c7-4ba7-8414-2e2147a579ba (us-east)
    metadata sync no sync (zone is master)
  2021-03-26T04:43:52.100+0000 7f084db56940  0 data sync zone:a2c19e5e ERROR: failed to fetch datalog info
        data sync source: a2c19e5e-5547-4adb-b3ad-88961858629b (us-west)
                          failed to retrieve sync info: (13) Permission denied

  
  On slave:
  root at juju-66e0ee-2-lxd-0:~# radosgw-admin sync status
            realm 2ce9452b-1ec6-4507-9068-4ba929071c54 (replicated)
        zonegroup f68b626c-24b7-4659-9309-1a80e2986568 (us)
             zone a2c19e5e-5547-4adb-b3ad-88961858629b (us-west)
    metadata sync preparing for full sync
                  full sync: 64/64 shards
                  full sync: 0 entries to sync
                  incremental sync: 0/64 shards
                  metadata is behind on 64 shards
                  behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63]
        data sync source: c2d6f2e0-28c7-4ba7-8414-2e2147a579ba (us-east)
                          syncing
                          full sync: 0/128 shards
                          incremental sync: 128/128 shards
                          data is caught up with source

  I see the system use on the master but don't see on the slave.
  The zones are configured correctly on both, and the slave can pull the realm successfully.

  One suspicious thing, on the slave there is a complaint in the log:
  2021-03-26T04:47:16.430+0000 7f21d7fe7700  5 req 4 0s :get_data_changes_log_info error reading user info, uid=TD10QYJHLSIIB6UCBZ5Z can't authenticate

  This is the access key of the system user used for replication, and it
  is correct (that's the one created on the master.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ceph-radosgw/+bug/1921453/+subscriptions