[Maas-devel] ERROR Invalid SSH key

Thu Jun 7 16:52:20 UTC 2012

so our hypothesis atm is that maas isn't returning a populated userdata to
the machine on its first boot, and that's basically returning a 200 http
status response with empty content. if a user subsequently reboots the
machine the correct metadata is in place but cloud-init won't run the nesc
parts again as they were already run on first boot, this includes putting
keys into place and running user scripts which juju needs to initialize the
machine. Apparently destroy-environment followed by bootstrap works because
the correct metadata is returned subsequently. so it sounds like a timing
issue getting the metadata to the machine's 'first' first boot with a new
disk image. i wonder if in truth its the old metadata for the instance
that's being returned though on the subsequent bootstrap, since its
identical content and node uuids are stable for the machine
across re-imaging.

this is from a  machine that was rebooted subsequent to enlistement
cloud-init log
http://pastebin.ubuntu.com/1028750/

timestamps on various cloud-init files shows the
http://pastebin.ubuntu.com/1028826/

the hypothesis can be verified by getting the contents of
/var/lib/cloud/instance/user-data.txt from the first boot of a machine. the
user we where helping had rebooted the instance and the cloud-init data got
overwritten with the subsequently in place metadata.

cheers,

Kapil

On Thu, Jun 7, 2012 at 11:31 AM, Kapil Thangavelu <
kapil.thangavelu at canonical.com> wrote:

> whoops we had the wrong debug command there to check the output. no
> conclusions yet. we're still investigating debugging this.
>
>
> On Thu, Jun 7, 2012 at 11:27 AM, Kapil Thangavelu <
> kapil.thangavelu at canonical.com> wrote:
>
>> from some of the debugging we've (smoser, cheez0r) been doing on irc
>> (#maas), It appears that there is no instance user-data/metadata for
>> cloudinit to use for these instances, which means the instance comes up
>> effectively uninitialized.
>>
>> <cheez0r> so I ran "python
>> /usr/share/pyshared/cloudinit/DataSourceMAAS.py --config
>> /etc/cloud/cloud.cfg.d/90_dpkg_maas.cfg crawl
>> http://192.168.1.1/MAAS/metadata/
>> <cheez0r> "
>> <cheez0r> response was "== http://192.168.1.1/MAAS/metadata/2012-03-01 ==
>> <cheez0r> 2012-03-01
>> <cheez0r> latest
>> <cheez0r>
>> <cheez0r> == http//192.168.1.1/MAAS/metadata/latest ==
>> <cheez0r> 2012-03-01
>> <cheez0r> latest
>> <cheez0r>
>> <cheez0r> "
>>
>>
>>
>>  On Wed, Jun 6, 2012 at 8:16 PM, Kapil Thangavelu <
>> kapil.thangavelu at canonical.com> wrote:
>>
>>> Its sound like the key isn't being put into place. The key is pushed as
>>> seed/kickstart metadata to maas as part of cloudinit initialization of the
>>> machine (ie. pre juju). It would be nice to do two things. 1. Verify this
>>> metadata in maas. 2. Verify its execution on the new machine, that might
>>> want a separate user on the image being installed to allow for access to
>>> the machine outside of juju/maas.
>>>
>>> -kapil
>>>
>>> On Wed, Jun 6, 2012 at 5:04 PM, Francis J. Lacoste <
>>> francis.lacoste at canonical.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> This is a problem that has been reported a couple of time, and that I've
>>>> experienced myself. Usually, destroying the environment and doing it
>>>> again makes the error disappear.
>>>>
>>>> Is there anything the user could do to help us diagnose the root cause?
>>>>
>>>> Cheers
>>>>
>>>> -------- Original Message --------
>>>> Subject:        Re: [Maas-devel] ERROR Invalid SSH key
>>>> Date:   Wed, 6 Jun 2012 22:00:25 +0100
>>>> From:   Kevin Jackson <kevin at linuxservices.co.uk>
>>>> To:     Francis J. Lacoste <francis.lacoste at canonical.com>
>>>> CC:     Michael Hughes <itismike at gmail.com>,
>>>> maas-devel at lists.launchpad.net
>>>>
>>>>
>>>>
>>>> Hi all,
>>>> It had come to the time to re-visit MAAS to see if I could get past my
>>>> initial problem and given that I'm replying to this email over a month
>>>> later - I've not managed it.
>>>>
>>>> I'm following the instructions at
>>>> https://help.ubuntu.com/community/UbuntuCloudInfrastructure
>>>>
>>>> I've got just 2 nodes.  One is running MAAS and I've gone through the
>>>> process of installing another with Precise installed.  MAAS reports this
>>>> as Green and ready.
>>>>
>>>> I then continue the instructions:
>>>>
>>>> I log into the MAAS server as a normal user.
>>>> I create the environments.yaml file with the following contents:
>>>>
>>>> environments:
>>>>   maas:
>>>>       type: maas
>>>>       maas-server: 'http://172.16.0.250:80/MAAS'
>>>>       maas-oauth:
>>>> 'tcWxFpwbWqyeBFDd4P:HTCSqrsw7XQKBcvm8n:bp67u5TkSLu2wf2b7wUS2ckLjwELCZED'
>>>>       admin-secret: 'nothing'
>>>>       default-series: precise
>>>>
>>>> I then create an ssh key
>>>>
>>>> ssh-keygen
>>>>
>>>> then I do the juju bootstrap
>>>>
>>>> juju bootstrap
>>>>
>>>>
>>>> 2012-06-06 21:51:06,959 INFO Bootstrapping environment 'maas' (origin:
>>>> distro type: maas)...
>>>> 2012-06-06 21:51:09,598 INFO 'bootstrap' command finished successfully
>>>>
>>>> When running juju status I still get the error about the keys.
>>>>
>>>> juju status
>>>> 2012-06-06 21:52:15,600 INFO Connecting to environment...
>>>> 2012-06-06 21:52:16,081 ERROR Invalid SSH key
>>>>
>>>> Running with verbose
>>>>
>>>> 2012-06-06 21:52:24,341 DEBUG Initializing juju status runtime
>>>> 2012-06-06 21:52:24,349 INFO Connecting to environment...
>>>> 2012-06-06 21:52:24,434 DEBUG Connecting to environment using
>>>> node-080027dd1de4.local...
>>>> 2012-06-06 21:52:24,435 DEBUG Spawning SSH process with
>>>> remote_user="ubuntu" remote_host="node-080027dd1de4.local"
>>>> remote_port="2181" local_port="52568".
>>>> 2012-06-06 21:52:24,543 ERROR Invalid SSH key
>>>> 2012-06-06 21:52:24,942:4525(0x7fcdd688d700):ZOO_INFO at log_env@658:
>>>> Client environment:zookeeper.version=zookeeper C client 3.3.5
>>>> 2012-06-06 21:52:24,942:4525(0x7fcdd688d700):ZOO_INFO at log_env@662:
>>>> Client environment:host.name <http://host.name>=maas
>>>> 2012-06-06 21:52:24,943:4525(0x7fcdd688d700):ZOO_INFO at log_env@669:
>>>> Client environment:os.name <http://os.name>=Linux
>>>> 2012-06-06 21:52:24,943:4525(0x7fcdd688d700):ZOO_INFO at log_env@670:
>>>> Client environment:os.arch=3.2.0-24-generic
>>>> 2012-06-06 21:52:24,944:4525(0x7fcdd688d700):ZOO_INFO at log_env@671:
>>>> Client environment:os.version=#39-Ubuntu SMP Mon May 21 16:52:17 UTC
>>>> 2012
>>>> 2012-06-06 21:52:24,945:4525(0x7fcdd688d700):ZOO_INFO at log_env@679:
>>>> Client environment:user.name <http://user.name>=openstack
>>>> 2012-06-06 21:52:24,945:4525(0x7fcdd688d700):ZOO_INFO at log_env@687:
>>>> Client environment:user.home=/home/openstack
>>>> 2012-06-06 21:52:24,946:4525(0x7fcdd688d700):ZOO_INFO at log_env@699:
>>>> Client environment:user.dir=/home/openstack
>>>> 2012-06-06
>>>> 21:52:24,946:4525(0x7fcdd688d700):ZOO_INFO at zookeeper_init@727:
>>>> Initiating client connection, host=localhost:52568 sessionTimeout=10000
>>>> watcher=0x7fcdd48316b0 sessionId=0 sessionPasswd=<null>
>>>> context=0x30fdaf0 flags=0
>>>> 2012-06-06
>>>> 21:52:24,947:4525(0x7fcdd15cc700):ZOO_ERROR at handle_socket_error_msg@1579:
>>>> Socket
>>>> [127.0.0.1:52568 <http://127.0.0.1:52568>] zk retcode=-4,
>>>> errno=111(Connection refused): server refused to accept the client
>>>> 2012-06-06
>>>> 21:52:28,284:4525(0x7fcdd15cc700):ZOO_ERROR at handle_socket_error_msg@1579:
>>>> Socket
>>>> [127.0.0.1:52568 <http://127.0.0.1:52568>] zk retcode=-4,
>>>> errno=111(Connection refused): server refused to accept the client
>>>> 2012-06-06
>>>> 21:52:31,621:4525(0x7fcdd15cc700):ZOO_ERROR at handle_socket_error_msg@1579:
>>>> Socket
>>>> [127.0.0.1:52568 <http://127.0.0.1:52568>] zk retcode=-4,
>>>> errno=111(Connection refused): server refused to accept the client
>>>> 2012-06-06
>>>> 21:52:34,957:4525(0x7fcdd15cc700):ZOO_ERROR at handle_socket_error_msg@1579:
>>>> Socket
>>>> [127.0.0.1:52568 <http://127.0.0.1:52568>] zk retcode=-4,
>>>> errno=111(Connection refused): server refused to accept the client
>>>>
>>>> Any help appreciated!
>>>>
>>>> Cheers,
>>>>
>>>> Kev
>>>>
>>>> On 30 April 2012 03:09, Francis J. Lacoste
>>>> <francis.lacoste at canonical.com <mailto:francis.lacoste at canonical.com>>
>>>> wrote:
>>>>
>>>>    On 12-04-28 07:28 AM, Kevin Jackson wrote:
>>>>    > Hi yes, the reason for this post is that it's not working for me.
>>>>    > The adding key to MAAS was a last resort. Leading up to this I
>>>>    generated
>>>>    > my key then juju bootstrap. This succeeds but then issuing juju
>>>> status
>>>>    > gives this error.
>>>>    >
>>>>    > I've seen a few posts on just do this, then that and unfortunately
>>>> it
>>>>    > doesn't work for me.
>>>>    >
>>>>    > I'll create another account and go through the steps again.
>>>>    >
>>>>    > Else if there are ways to debug its appreciated.
>>>>    >
>>>>
>>>>    I've encountered that error from time to time, but never really got
>>>> into
>>>>    the guts of it. It looks like a Juju problem. Every time I've
>>>>    encountered it, I've done: juju destroy-environment and then the next
>>>>    bootstrap worked.
>>>>
>>>>    Cheers
>>>>
>>>>
>>>>    --
>>>>    Francis J. Lacoste
>>>>    francis.lacoste at canonical.com <mailto:francis.lacoste at canonical.com>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Kevin Jackson
>>>> @itarchitectkev
>>>>
>>>>
>>>> --
>>>> Juju mailing list
>>>> Juju at lists.ubuntu.com
>>>> Modify settings or unsubscribe at:
>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20120607/392995dc/attachment-0001.html>