so our hypothesis atm is that maas isn't returning a populated userdata to the machine on its first boot, and that's basically returning a 200 http status response with empty content. if a user subsequently reboots the machine the correct metadata is in place but cloud-init won't run the nesc parts again as they were already run on first boot, this includes putting keys into place and running user scripts which juju needs to initialize the machine. Apparently destroy-environment followed by bootstrap works because the correct metadata is returned subsequently. so it sounds like a timing issue getting the metadata to the machine's 'first' first boot with a new disk image. i wonder if in truth its the old metadata for the instance that's being returned though on the subsequent bootstrap, since its identical content and node uuids are stable for the machine across re-imaging.<div>
<br></div><div>this is from a machine that was rebooted subsequent to enlistement</div><div>cloud-init log</div><div><a href="http://pastebin.ubuntu.com/1028750/">http://pastebin.ubuntu.com/1028750/</a></div><div><br></div>
<div>timestamps on various cloud-init files shows the </div><div><a href="http://pastebin.ubuntu.com/1028826/">http://pastebin.ubuntu.com/1028826/</a><br><div><br></div><div>the hypothesis can be verified by getting the contents of /var/lib/cloud/instance/user-data.txt from the first boot of a machine. the user we where helping had rebooted the instance and the cloud-init data got overwritten with the subsequently in place metadata.</div>
<div><br></div><div>cheers,</div><div><br></div><div>Kapil</div><div><br></div><div><br></div><div><br><br><div class="gmail_quote">On Thu, Jun 7, 2012 at 11:31 AM, Kapil Thangavelu <span dir="ltr"><<a href="mailto:kapil.thangavelu@canonical.com" target="_blank">kapil.thangavelu@canonical.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>whoops we had the wrong debug command there to check the output. no conclusions yet. we're still investigating debugging this.</div>
<div class="HOEnZb"><div class="h5"><div><br><br><div class="gmail_quote">On Thu, Jun 7, 2012 at 11:27 AM, Kapil Thangavelu <span dir="ltr"><<a href="mailto:kapil.thangavelu@canonical.com" target="_blank">kapil.thangavelu@canonical.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">from some of the debugging we've (smoser, cheez0r) been doing on irc (#maas), It appears that there is no instance user-data/metadata for cloudinit to use for these instances, which means the instance comes up effectively uninitialized.<div>
<br></div><div><div><cheez0r> so I ran "python /usr/share/pyshared/cloudinit/DataSourceMAAS.py --config /etc/cloud/cloud.cfg.d/90_dpkg_maas.cfg crawl <a href="http://192.168.1.1/MAAS/metadata/" target="_blank">http://192.168.1.1/MAAS/metadata/</a></div>
<div><cheez0r> "</div><div><cheez0r> response was "== <a href="http://192.168.1.1/MAAS/metadata/2012-03-01" target="_blank">http://192.168.1.1/MAAS/metadata/2012-03-01</a> ==</div><div><cheez0r> 2012-03-01</div>
<div><cheez0r> latest</div><div><cheez0r> </div><div><cheez0r> == http//<a href="http://192.168.1.1/MAAS/metadata/latest" target="_blank">192.168.1.1/MAAS/metadata/latest</a> ==</div><div><cheez0r> 2012-03-01</div>
<div><cheez0r> latest</div><div><cheez0r> </div><div><cheez0r> "</div><div><div><div><br></div><div><br></div><div><font color="#222222" face="arial, sans-serif"><span style="white-space:nowrap"><br>
</span></font><div class="gmail_quote">
On Wed, Jun 6, 2012 at 8:16 PM, Kapil Thangavelu <span dir="ltr"><<a href="mailto:kapil.thangavelu@canonical.com" target="_blank">kapil.thangavelu@canonical.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Its sound like the key isn't being put into place. The key is pushed as seed/kickstart metadata to maas as part of cloudinit initialization of the machine (ie. pre juju). It would be nice to do two things. 1. Verify this metadata in maas. 2. Verify its execution on the new machine, that might want a separate user on the image being installed to allow for access to the machine outside of juju/maas. <div>
<br></div><div>-kapil</div><div><br><div class="gmail_quote"><div><div>On Wed, Jun 6, 2012 at 5:04 PM, Francis J. Lacoste <span dir="ltr"><<a href="mailto:francis.lacoste@canonical.com" target="_blank">francis.lacoste@canonical.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>Hi,<br>
<br>
This is a problem that has been reported a couple of time, and that I've<br>
experienced myself. Usually, destroying the environment and doing it<br>
again makes the error disappear.<br>
<br>
Is there anything the user could do to help us diagnose the root cause?<br>
<br>
Cheers<br>
<br>
-------- Original Message --------<br>
Subject: Re: [Maas-devel] ERROR Invalid SSH key<br>
Date: Wed, 6 Jun 2012 22:00:25 +0100<br>
From: Kevin Jackson <<a href="mailto:kevin@linuxservices.co.uk" target="_blank">kevin@linuxservices.co.uk</a>><br>
To: Francis J. Lacoste <<a href="mailto:francis.lacoste@canonical.com" target="_blank">francis.lacoste@canonical.com</a>><br>
CC: Michael Hughes <<a href="mailto:itismike@gmail.com" target="_blank">itismike@gmail.com</a>>, <a href="mailto:maas-devel@lists.launchpad.net" target="_blank">maas-devel@lists.launchpad.net</a><br>
<br>
<br>
<br>
Hi all,<br>
It had come to the time to re-visit MAAS to see if I could get past my<br>
initial problem and given that I'm replying to this email over a month<br>
later - I've not managed it.<br>
<br>
I'm following the instructions at<br>
<a href="https://help.ubuntu.com/community/UbuntuCloudInfrastructure" target="_blank">https://help.ubuntu.com/community/UbuntuCloudInfrastructure</a><br>
<br>
I've got just 2 nodes. One is running MAAS and I've gone through the<br>
process of installing another with Precise installed. MAAS reports this<br>
as Green and ready.<br>
<br>
I then continue the instructions:<br>
<br>
I log into the MAAS server as a normal user.<br>
I create the environments.yaml file with the following contents:<br>
<br>
environments:<br>
maas:<br>
type: maas<br>
maas-server: '<a href="http://172.16.0.250:80/MAAS" target="_blank">http://172.16.0.250:80/MAAS</a>'<br>
maas-oauth:<br>
'tcWxFpwbWqyeBFDd4P:HTCSqrsw7XQKBcvm8n:bp67u5TkSLu2wf2b7wUS2ckLjwELCZED'<br>
admin-secret: 'nothing'<br>
default-series: precise<br>
<br>
I then create an ssh key<br>
<br>
ssh-keygen<br>
<br>
then I do the juju bootstrap<br>
<br>
juju bootstrap<br>
<br>
<br>
2012-06-06 21:51:06,959 INFO Bootstrapping environment 'maas' (origin:<br>
distro type: maas)...<br>
2012-06-06 21:51:09,598 INFO 'bootstrap' command finished successfully<br>
<br>
When running juju status I still get the error about the keys.<br>
<br>
juju status<br>
2012-06-06 21:52:15,600 INFO Connecting to environment...<br>
2012-06-06 21:52:16,081 ERROR Invalid SSH key<br>
<br>
Running with verbose<br>
<br>
2012-06-06 21:52:24,341 DEBUG Initializing juju status runtime<br>
2012-06-06 21:52:24,349 INFO Connecting to environment...<br>
2012-06-06 21:52:24,434 DEBUG Connecting to environment using<br>
node-080027dd1de4.local...<br>
2012-06-06 21:52:24,435 DEBUG Spawning SSH process with<br>
remote_user="ubuntu" remote_host="node-080027dd1de4.local"<br>
remote_port="2181" local_port="52568".<br>
2012-06-06 21:52:24,543 ERROR Invalid SSH key<br>
2012-06-06 21:52:24,942:4525(0x7fcdd688d700):ZOO_INFO@log_env@658:<br>
Client environment:zookeeper.version=zookeeper C client 3.3.5<br>
2012-06-06 21:52:24,942:4525(0x7fcdd688d700):ZOO_INFO@log_env@662:<br>
Client environment:<a href="http://host.name" target="_blank">host.name</a> <<a href="http://host.name" target="_blank">http://host.name</a>>=maas<br>
2012-06-06 21:52:24,943:4525(0x7fcdd688d700):ZOO_INFO@log_env@669:<br>
Client environment:<a href="http://os.name" target="_blank">os.name</a> <<a href="http://os.name" target="_blank">http://os.name</a>>=Linux<br>
2012-06-06 21:52:24,943:4525(0x7fcdd688d700):ZOO_INFO@log_env@670:<br>
Client environment:os.arch=3.2.0-24-generic<br>
2012-06-06 21:52:24,944:4525(0x7fcdd688d700):ZOO_INFO@log_env@671:<br>
Client environment:os.version=#39-Ubuntu SMP Mon May 21 16:52:17 UTC 2012<br>
2012-06-06 21:52:24,945:4525(0x7fcdd688d700):ZOO_INFO@log_env@679:<br>
Client environment:<a href="http://user.name" target="_blank">user.name</a> <<a href="http://user.name" target="_blank">http://user.name</a>>=openstack<br>
2012-06-06 21:52:24,945:4525(0x7fcdd688d700):ZOO_INFO@log_env@687:<br>
Client environment:user.home=/home/openstack<br>
2012-06-06 21:52:24,946:4525(0x7fcdd688d700):ZOO_INFO@log_env@699:<br>
Client environment:user.dir=/home/openstack<br>
2012-06-06<br>
21:52:24,946:4525(0x7fcdd688d700):ZOO_INFO@zookeeper_init@727:<br>
Initiating client connection, host=localhost:52568 sessionTimeout=10000<br>
watcher=0x7fcdd48316b0 sessionId=0 sessionPasswd=<null><br>
context=0x30fdaf0 flags=0<br>
2012-06-06<br>
21:52:24,947:4525(0x7fcdd15cc700):ZOO_ERROR@handle_socket_error_msg@1579: Socket<br>
[<a href="http://127.0.0.1:52568" target="_blank">127.0.0.1:52568</a> <<a href="http://127.0.0.1:52568" target="_blank">http://127.0.0.1:52568</a>>] zk retcode=-4,<br>
errno=111(Connection refused): server refused to accept the client<br>
2012-06-06<br>
21:52:28,284:4525(0x7fcdd15cc700):ZOO_ERROR@handle_socket_error_msg@1579: Socket<br>
[<a href="http://127.0.0.1:52568" target="_blank">127.0.0.1:52568</a> <<a href="http://127.0.0.1:52568" target="_blank">http://127.0.0.1:52568</a>>] zk retcode=-4,<br>
errno=111(Connection refused): server refused to accept the client<br>
2012-06-06<br>
21:52:31,621:4525(0x7fcdd15cc700):ZOO_ERROR@handle_socket_error_msg@1579: Socket<br>
[<a href="http://127.0.0.1:52568" target="_blank">127.0.0.1:52568</a> <<a href="http://127.0.0.1:52568" target="_blank">http://127.0.0.1:52568</a>>] zk retcode=-4,<br>
errno=111(Connection refused): server refused to accept the client<br>
2012-06-06<br>
21:52:34,957:4525(0x7fcdd15cc700):ZOO_ERROR@handle_socket_error_msg@1579: Socket<br>
[<a href="http://127.0.0.1:52568" target="_blank">127.0.0.1:52568</a> <<a href="http://127.0.0.1:52568" target="_blank">http://127.0.0.1:52568</a>>] zk retcode=-4,<br>
errno=111(Connection refused): server refused to accept the client<br>
<br>
Any help appreciated!<br>
<br>
Cheers,<br>
<br>
Kev<br>
<br>
On 30 April 2012 03:09, Francis J. Lacoste<br>
<<a href="mailto:francis.lacoste@canonical.com" target="_blank">francis.lacoste@canonical.com</a> <mailto:<a href="mailto:francis.lacoste@canonical.com" target="_blank">francis.lacoste@canonical.com</a>>><br>
wrote:<br>
<br>
On 12-04-28 07:28 AM, Kevin Jackson wrote:<br>
> Hi yes, the reason for this post is that it's not working for me.<br>
> The adding key to MAAS was a last resort. Leading up to this I<br>
generated<br>
> my key then juju bootstrap. This succeeds but then issuing juju status<br>
> gives this error.<br>
><br>
> I've seen a few posts on just do this, then that and unfortunately it<br>
> doesn't work for me.<br>
><br>
> I'll create another account and go through the steps again.<br>
><br>
> Else if there are ways to debug its appreciated.<br>
><br>
<br>
I've encountered that error from time to time, but never really got into<br>
the guts of it. It looks like a Juju problem. Every time I've<br>
encountered it, I've done: juju destroy-environment and then the next<br>
bootstrap worked.<br>
<br>
Cheers<br>
<br>
<br>
--<br>
Francis J. Lacoste<br>
<a href="mailto:francis.lacoste@canonical.com" target="_blank">francis.lacoste@canonical.com</a> <mailto:<a href="mailto:francis.lacoste@canonical.com" target="_blank">francis.lacoste@canonical.com</a>><br>
<span><font color="#888888"><br>
<br>
<br>
<br>
--<br>
Kevin Jackson<br>
@itarchitectkev<br>
<br>
</font></span><br></div></div><span><font color="#888888">--<br>
Juju mailing list<br>
<a href="mailto:Juju@lists.ubuntu.com" target="_blank">Juju@lists.ubuntu.com</a><br>
Modify settings or unsubscribe at: <a href="https://lists.ubuntu.com/mailman/listinfo/juju" target="_blank">https://lists.ubuntu.com/mailman/listinfo/juju</a><br>
<br></font></span></blockquote></div><br></div>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>