[Bug 1921453] Re: multi-zone replication doesn't work

[James Page]

It looks like the metadata sync is failing for some reason, resulting in
the secondary zone not have the right system user - which results in the
errors on the primary zone when it attempts to sync data back

** Also affects: ceph (Ubuntu)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1921453

Title:
  multi-zone replication doesn't work

Status in OpenStack ceph-radosgw charm:
  In Progress
Status in ceph package in Ubuntu:
  New

Bug description:
  I deploy the following bundle
  https://pastebin.ubuntu.com/p/qgt55P2Hv5/

  (tried both bionic-ussuri and focal-distro origins).
  There are two radosgw instances, each is backed by its own Ceph cluster.
  Each RGW pull info correctly, however the sync status on master:
  root at juju-66e0ee-1-lxd-0:~# radosgw-admin sync status
            realm 2ce9452b-1ec6-4507-9068-4ba929071c54 (replicated)
        zonegroup f68b626c-24b7-4659-9309-1a80e2986568 (us)
             zone c2d6f2e0-28c7-4ba7-8414-2e2147a579ba (us-east)
    metadata sync no sync (zone is master)
  2021-03-26T04:43:52.100+0000 7f084db56940  0 data sync zone:a2c19e5e ERROR: failed to fetch datalog info
        data sync source: a2c19e5e-5547-4adb-b3ad-88961858629b (us-west)
                          failed to retrieve sync info: (13) Permission denied

  
  On slave:
  root at juju-66e0ee-2-lxd-0:~# radosgw-admin sync status
            realm 2ce9452b-1ec6-4507-9068-4ba929071c54 (replicated)
        zonegroup f68b626c-24b7-4659-9309-1a80e2986568 (us)
             zone a2c19e5e-5547-4adb-b3ad-88961858629b (us-west)
    metadata sync preparing for full sync
                  full sync: 64/64 shards
                  full sync: 0 entries to sync
                  incremental sync: 0/64 shards
                  metadata is behind on 64 shards
                  behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63]
        data sync source: c2d6f2e0-28c7-4ba7-8414-2e2147a579ba (us-east)
                          syncing
                          full sync: 0/128 shards
                          incremental sync: 128/128 shards
                          data is caught up with source

  I see the system use on the master but don't see on the slave.
  The zones are configured correctly on both, and the slave can pull the realm successfully.

  One suspicious thing, on the slave there is a complaint in the log:
  2021-03-26T04:47:16.430+0000 7f21d7fe7700  5 req 4 0s :get_data_changes_log_info error reading user info, uid=TD10QYJHLSIIB6UCBZ5Z can't authenticate

  This is the access key of the system user used for replication, and it
  is correct (that's the one created on the master.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ceph-radosgw/+bug/1921453/+subscriptions