[Bug 1026402] [NEW] mon cluster (no cephx) fails to start unless empty keyring files are created

[Paul Collins]

Public bug reported:

I'm running a 3-node test cluster on 12.04, without cephx
authentication. I started out running 0.47.2 (an impatiently-smashed-
together backport based on the upstream sources) and then upgraded to
0.48-1ubuntu1 (the packages from quantal rebuilt on precise). So my
situation may be a bit special.

When I upgraded from 0.47.2 to 0.48, I didn't notice that my first
monitor daemon hadn't restarted properly.  I rolled through the upgrade
and ended up with a system where "ceph -s" would hang, being unable to
find a monitor willing to accept responsibility for the cluster.  I
splashed around rather a lot turning on debug logging. The monitors
tended to get as far as

2012-07-17 02:38:52.254856 7f3c3b862780 -1 auth: error reading file: /srv/ceph/mon.leningradskaya/keyring: can't open /srv/ceph/mon.leningradskaya/keyring: (2) No such file or directory
2012-07-17 02:38:52.254874 7f3c3b862780 -1 mon.leningradskaya at -1(probing) e1 unable to load initial keyring /etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin
2012-07-17 02:38:53.006423 7f3c3b860700  1 -- 10.55.200.21:6789/0 >> :/0 pipe(0x7f3c2c0008c0 sd=17 pgs=0 cs=0 l=0).accept sd=17
2012-07-17 02:38:53.231137 7f3c386a1700  1 -- 10.55.200.21:6789/0 >> :/0 pipe(0x7f3c2c000f60 sd=18 pgs=0 cs=0 l=0).accept sd=18
2012-07-17 02:38:53.308857 7f3c3849f700  1 -- 10.55.200.21:6789/0 >> :/0 pipe(0x7f3c2c0015c0 sd=19 pgs=0 cs=0 l=0).accept sd=19
2012-07-17 02:38:53.668990 7f3c3829d700  1 -- 10.55.200.21:6789/0 >> :/0 pipe(0x7f3c2c001c20 sd=20 pgs=0 cs=0 l=0).accept sd=20

with lines like the last four streaming endlessly.  Eventually I tried
creating /srv/ceph/mon.leningradskaya/keyring and the monitor daemon
started right up. When I applied the same change to the rest of the
cluster, I was back in business. Here's a log snippet from a successful
0.48 monitor daemon startup:

2012-07-17 02:47:03.036077 7f5f2a66f780  2 auth: KeyRing::load: loaded key file /srv/ceph/mon.leningradskaya/keyring
2012-07-17 02:47:03.036283 7f5f2a66f780 10 mon.leningradskaya at -1(probing) e1 bootstrap
2012-07-17 02:47:03.036319 7f5f2a66f780 10 mon.leningradskaya at -1(probing) e1 unregister_cluster_logger - not registered
2012-07-17 02:47:03.036346 7f5f2a66f780 10 mon.leningradskaya at -1(probing) e1 cancel_probe_timeout (none scheduled)
2012-07-17 02:47:03.036383 7f5f2a66f780  0 mon.leningradskaya at -1(probing) e1  my rank is now 1 (was -1)

continuing to log more besides as the cluster came back up.

** Affects: ceph (Ubuntu)
     Importance: Undecided
         Status: New

** Affects: ceph (Ubuntu Quantal)
     Importance: Undecided
         Status: New


** Tags: canonistack

** Also affects: ceph (Ubuntu Quantal)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1026402

Title:
  mon cluster (no cephx) fails to start unless empty keyring files are
  created

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1026402/+subscriptions