[Bug 2055325] [NEW] ceph-mon fails with hook failed: "mon-relation-changed"

Jeffrey Chang 2055325 at bugs.launchpad.net
Wed Feb 28 16:29:22 UTC 2024 

Public bug reported:

In SolQA testrun - https://solutions.qa.canonical.com/testruns/7aa425f2-ab53-4a4e-9de5-40439041d1cb
```
ceph-mon/0                            error     idle   0/lxd/0  10.246.166.119                hook failed: "mon-relation-changed"
```

debug logs
2024-02-28 08:49:42 DEBUG juju.worker.uniter agent.go:22 [AGENT-STATUS] executing: running mon-relation-changed hook for ceph-mon/2
2024-02-28 08:49:42 DEBUG juju.worker.uniter.runner runner.go:719 starting jujuc server  {unix @/var/lib/juju/agents/unit-ceph-mon-1/agent.socket <nil>}
2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: ops 2.9.0 up and running.
2024-02-28 08:49:42 WARNING unit.ceph-mon/1.juju-log server.go:325 mon:0: 0 containers are present in metadata.yaml and refresh_event was not specified. Defaulting to update_status. Metrics IP may not be set in a timely fashion.
2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Emitting Juju event mon_relation_changed.
2024-02-28 08:49:42 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Making dir /var/lib/charm/ceph-mon ceph:ceph 555
2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Writing file /var/lib/charm/ceph-mon/ceph.conf root:root 644
2024-02-28 08:49:42 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Making dir /var/run/ceph ceph:ceph 755
2024-02-28 08:49:42 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Making dir /var/lib/ceph/mon/ceph-juju-31f10f-1-lxd-0 ceph:ceph 755
2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.mon-relation-changed logger.go:60 creating /var/lib/ceph/tmp/juju-31f10f-1-lxd-0.mon.keyring
2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.mon-relation-changed logger.go:60 added entity mon. auth(key=AQCB895lE3iDKRAAWFYNdMO7MnctCwYQidpFUw==)
2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.mon-relation-changed logger.go:60 added 1 caps to entity mon.
2024-02-28 08:49:42 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon at juju-31f10f-1-lxd-0.service → /lib/systemd/system/ceph-mon at .service.
2024-02-28 08:49:42 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Waiting for quorum to be reached
2024-02-28 08:49:45 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Waiting for quorum to be reached
2024-02-28 08:49:48 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Waiting for quorum to be reached
2024-02-28 08:49:52 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Waiting for quorum to be reached
2024-02-28 08:49:55 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Making dir /var/lib/ceph/mgr/ceph-juju-31f10f-1-lxd-0 ceph:ceph 555
2024-02-28 08:50:00 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 Created symlink /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr at juju-31f10f-1-lxd-0.service → /lib/systemd/system/ceph-mgr at .service.
2024-02-28 08:50:00 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 Error ENOENT: key 'autotune' doesn't exist
2024-02-28 08:50:01 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Waiting for manager to be available
2024-02-28 08:50:06 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 set autotune
2024-02-28 08:50:08 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 module 'pg_autoscaler' is already enabled (always-on)
2024-02-28 08:50:08 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: mon cluster in quorum - providing fsid & keys
2024-02-28 08:50:08 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:08.697+0000 7fa885822640 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
2024-02-28 08:50:08 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:08.697+0000 7fa885822640 -1 AuthRegistry(0x7fa88005fd00) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
2024-02-28 08:50:08 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:08.981+0000 7fb0820e5640 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
2024-02-28 08:50:08 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:08.981+0000 7fb0820e5640 -1 AuthRegistry(0x7fb07c05fd00) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
2024-02-28 08:50:09 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 Error ENOENT: failed to find client.osd-upgrade in keyring
2024-02-28 08:50:09 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Creating new key for osd-upgrade
2024-02-28 08:50:09 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Calling check_output: ['sudo', '-u', 'ceph', 'ceph', '--name', 'mon.', '--keyring', '/var/lib/ceph/mon/ceph-juju-31f10f-1-lxd-0/keyring', 'auth', 'get-or-create', 'client.osd-upgrade', 'mon', 'allow command "config-key"; allow command "osd tree"; allow command "config-key list"; allow command "config-key put"; allow command "config-key get"; allow command "config-key exists"; allow command "osd out"; allow command "osd in"; allow command "osd rm"; allow command "auth del"']
2024-02-28 08:50:09 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:09.269+0000 7fc249f51640 -1 AuthRegistry(0x7fc24405fd00) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
2024-02-28 08:50:09 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:09.557+0000 7f17f0f1e640 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
2024-02-28 08:50:09 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:09.557+0000 7f17f0f1e640 -1 AuthRegistry(0x7f17ec05fd00) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
2024-02-28 08:50:09 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 Error ENOENT: failed to find client.osd-removal in keyring
2024-02-28 08:50:09 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Creating new key for osd-removal
2024-02-28 08:50:09 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Calling check_output: ['sudo', '-u', 'ceph', 'ceph', '--name', 'mon.', '--keyring', '/var/lib/ceph/mon/ceph-juju-31f10f-1-lxd-0/keyring', 'auth', 'get-or-create', 'client.osd-removal', 'mgr', 'allow *', 'mon', 'allow r; allow command "osd crush reweight"; allow command "osd purge"; allow command "osd destroy"']0000 7fe3ef8e9640 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
2024-02-28 08:50:09 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Creating new key for osd-removal

Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/7aa425f2-ab53-4a4e-9de5-40439041d1cb/index.html

** Affects: charm-ceph-mon
     Importance: Undecided
         Status: New

** Project changed: cloud-archive => charm-ceph-mon

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2055325

Title:
  ceph-mon fails with hook failed: "mon-relation-changed"

Status in Ceph Monitor Charm:
  New

Bug description:
  In SolQA testrun - https://solutions.qa.canonical.com/testruns/7aa425f2-ab53-4a4e-9de5-40439041d1cb
  ```
  ceph-mon/0                            error     idle   0/lxd/0  10.246.166.119                hook failed: "mon-relation-changed"
  ```

  debug logs
  2024-02-28 08:49:42 DEBUG juju.worker.uniter agent.go:22 [AGENT-STATUS] executing: running mon-relation-changed hook for ceph-mon/2
  2024-02-28 08:49:42 DEBUG juju.worker.uniter.runner runner.go:719 starting jujuc server  {unix @/var/lib/juju/agents/unit-ceph-mon-1/agent.socket <nil>}
  2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: ops 2.9.0 up and running.
  2024-02-28 08:49:42 WARNING unit.ceph-mon/1.juju-log server.go:325 mon:0: 0 containers are present in metadata.yaml and refresh_event was not specified. Defaulting to update_status. Metrics IP may not be set in a timely fashion.
  2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Emitting Juju event mon_relation_changed.
  2024-02-28 08:49:42 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Making dir /var/lib/charm/ceph-mon ceph:ceph 555
  2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Writing file /var/lib/charm/ceph-mon/ceph.conf root:root 644
  2024-02-28 08:49:42 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Making dir /var/run/ceph ceph:ceph 755
  2024-02-28 08:49:42 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Making dir /var/lib/ceph/mon/ceph-juju-31f10f-1-lxd-0 ceph:ceph 755
  2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.mon-relation-changed logger.go:60 creating /var/lib/ceph/tmp/juju-31f10f-1-lxd-0.mon.keyring
  2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.mon-relation-changed logger.go:60 added entity mon. auth(key=AQCB895lE3iDKRAAWFYNdMO7MnctCwYQidpFUw==)
  2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.mon-relation-changed logger.go:60 added 1 caps to entity mon.
  2024-02-28 08:49:42 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon at juju-31f10f-1-lxd-0.service → /lib/systemd/system/ceph-mon at .service.
  2024-02-28 08:49:42 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Waiting for quorum to be reached
  2024-02-28 08:49:45 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Waiting for quorum to be reached
  2024-02-28 08:49:48 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Waiting for quorum to be reached
  2024-02-28 08:49:52 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Waiting for quorum to be reached
  2024-02-28 08:49:55 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Making dir /var/lib/ceph/mgr/ceph-juju-31f10f-1-lxd-0 ceph:ceph 555
  2024-02-28 08:50:00 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 Created symlink /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr at juju-31f10f-1-lxd-0.service → /lib/systemd/system/ceph-mgr at .service.
  2024-02-28 08:50:00 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 Error ENOENT: key 'autotune' doesn't exist
  2024-02-28 08:50:01 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Waiting for manager to be available
  2024-02-28 08:50:06 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 set autotune
  2024-02-28 08:50:08 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 module 'pg_autoscaler' is already enabled (always-on)
  2024-02-28 08:50:08 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: mon cluster in quorum - providing fsid & keys
  2024-02-28 08:50:08 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:08.697+0000 7fa885822640 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
  2024-02-28 08:50:08 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:08.697+0000 7fa885822640 -1 AuthRegistry(0x7fa88005fd00) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
  2024-02-28 08:50:08 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:08.981+0000 7fb0820e5640 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
  2024-02-28 08:50:08 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:08.981+0000 7fb0820e5640 -1 AuthRegistry(0x7fb07c05fd00) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
  2024-02-28 08:50:09 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 Error ENOENT: failed to find client.osd-upgrade in keyring
  2024-02-28 08:50:09 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Creating new key for osd-upgrade
  2024-02-28 08:50:09 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Calling check_output: ['sudo', '-u', 'ceph', 'ceph', '--name', 'mon.', '--keyring', '/var/lib/ceph/mon/ceph-juju-31f10f-1-lxd-0/keyring', 'auth', 'get-or-create', 'client.osd-upgrade', 'mon', 'allow command "config-key"; allow command "osd tree"; allow command "config-key list"; allow command "config-key put"; allow command "config-key get"; allow command "config-key exists"; allow command "osd out"; allow command "osd in"; allow command "osd rm"; allow command "auth del"']
  2024-02-28 08:50:09 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:09.269+0000 7fc249f51640 -1 AuthRegistry(0x7fc24405fd00) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
  2024-02-28 08:50:09 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:09.557+0000 7f17f0f1e640 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
  2024-02-28 08:50:09 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:09.557+0000 7f17f0f1e640 -1 AuthRegistry(0x7f17ec05fd00) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
  2024-02-28 08:50:09 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 Error ENOENT: failed to find client.osd-removal in keyring
  2024-02-28 08:50:09 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Creating new key for osd-removal
  2024-02-28 08:50:09 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Calling check_output: ['sudo', '-u', 'ceph', 'ceph', '--name', 'mon.', '--keyring', '/var/lib/ceph/mon/ceph-juju-31f10f-1-lxd-0/keyring', 'auth', 'get-or-create', 'client.osd-removal', 'mgr', 'allow *', 'mon', 'allow r; allow command "osd crush reweight"; allow command "osd purge"; allow command "osd destroy"']0000 7fe3ef8e9640 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
  2024-02-28 08:50:09 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Creating new key for osd-removal

  Crashdumps and configs can be found here:
  https://oil-jenkins.canonical.com/artifacts/7aa425f2-ab53-4a4e-9de5-40439041d1cb/index.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ceph-mon/+bug/2055325/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list