juju hangs during bootstrapping

Sat Sep 30 11:55:17 UTC 2017

On Sat, Sep 30, 2017 at 5:55 AM <Dave.Chen at dell.com> wrote:

> Hi Narinder,
>
>
>
> Here is the log from the deployed node, actually, I can deploy Operation
> System successfully with MAAS or “juju bootstrap” but failed at some final
> steps, Our external network is exactly broken for some reason yesterday and
> won’t be recovered in short term, but I guess the network broken is
>  happened after I saw the below error message during bootstrapping.
>
> “Waiting for address
>
> Attempting to connect to 10.20.3.254:22”
>
>
>
> I can see apt-get update works from the log at the beginning, and the
> network is broken couple of hours after I saw those error message.
>
>
>
> Does the error message like below has any connection with connection
> refused error message? I am not quite sure.
>

No, they're not connected, but the failing curl command is what is
preventing bootstrap from proceeding. That's the server trying to download
the Juju agent.

>
>
> “0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
>
> Fetching Juju agent version 2.2.4 for amd64
>
> Attempt 1 to download tools from
> https://streams.canonical.com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz.
> ..
>
> curl: (6) Could not resolve host: streams.canonical.com
>
> “
>
>
>
> Anyway, I will try it again when our network back to normal, it would be
> great if you can see any other issue beside the network, thanks again for
> your help!
>
>
>
> Best Regards,
>
> Dave Chen
>
>
>
> *From:* Narinder Gupta [mailto:narinder.gupta at canonical.com]
> *Sent:* Friday, September 29, 2017 10:52 PM
> *To:* Chen2, Dave <Dave_Chen2 at Dell.com>
> *Cc:* juju <juju at lists.ubuntu.com>
> *Subject:* Re: juju hangs during bootstrapping
>
>
>
> Hi Dave,
>
> May I know which division of Dell you are working on? As i have setup
> Openstack deployed t Dell multiple time with MAAS and have not seen this
> issue so far.
>
>
>
> So please send me log /var/log/cloud-init-output.log which will let us
> know what is wrong. Also try sudo apt-get update on the bootstrap node to
> confirm you have external access.
>
>
>
> In MAAS you can always add the ssh keys to land into the installed nodes
> though.
>
>
>
>
> Thanks and Regards,
>
> Narinder Gupta (PMP)                   narinder.gupta at canonical.com
>
> Canonical, Ltd.                    narindergupta [irc.freenode.net]
>
> +1.281.736.5150 <(281)%20736-5150>                            narindergupta2007[skype]
>
>
>
> Ubuntu- Linux for human beings | www.ubuntu.com | www.canonical.com
>
>
>
> On Fri, Sep 29, 2017 at 9:42 AM, <Dave.Chen at dell.com> wrote:
>
> Hi All,
>
>
>
> I am trying to bootstrap a MAAS cloud based on juju’s official guide (
> https://jujucharms.com/docs/2.2/clouds-maas), everything seems correct
> but after the Operation System (Ubuntu 16.04 or Ubuntu14.0) has been
> installed, juju hangs when attempting to connect to the MAAS node, here is
> what I can see from the terminal,
>
>
>
> $ juju bootstrap maas-cloud
>
> Creating Juju controller "maas-cloud" on maas-cloud
>
> Looking for packaged Juju agent version 2.2.4 for amd64
>
> Launching controller instance(s) on maas-cloud...
>
> - cka68p (arch=amd64 mem=32G cores=12)
>
> Fetching Juju GUI 2.9.2
>
> Waiting for address
>
> Attempting to connect to 10.20.3.254:22 (JUJU hangs here!)
>
>
>
> And it’s pending here forever, so I tried it again with the debug mode,
>
> $ juju bootstrap --show-log --debug --bootstrap-series=trusty maas-cloud
> maas-cloud-controller
>
>
>
> I saw some detail information like below,
>
> Attempting to connect to 10.20.3.254:22
>
> 19:33:11 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22:
> Connection refused
>
> 19:33:16 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22:
> Connection refused
>
> 19:33:21 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22:
> Connection refused
>
> 19:33:56 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist
>
> 19:34:32 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist
>
> 19:35:08 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist
>
> 19:35:43 INFO  juju.cloudconfig userdatacfg_unix.go:410 Fetching agent:
> curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time
> %{time_total}s; size %{size_download} bytes; speed %{speed_download}
> bytes/s ' --retry 10 -o $bin/tools.tar.gz <[
> https://streams.canonical.com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz]>
>
>
>
> I have no idea what’s going wrong since I can telnet to the node and ssh
> to that node is also possible, I just need type “yes” then I can login to
> the node,
>
> $ ssh ubuntu at 10.20.3.254
>
> The authenticity of host ' 10.20.3.254 (10.20.3.254)' can't be established.
>
> ECDSA key fingerprint is
> SHA256:4FVm21s4dx7gc0/yDgz0+QAMGK4qWODoIqeoWtZg9RI.
>
> Are you sure you want to continue connecting (yes/no)?
>
>
>
> From the console of that node, I can find the controller’s public key has
> been injected to the node,
>
> -----BEGIN SSH HOST KEY KEYS-------
>
> …
>
> -----END SSH KEY FINGERPRINTS--------
>
> …
>
> Cloud-init v. 0.7.9 finished at … Datasource DataSourceMAAS [http://...:5240/MAAS/metadata/].
> Up 153.77 seconds.       (cloud-init hangs here!)
>
>
>
>
>
> I googled it and found someone said it is because “authorized-keys-path”
> is commented out in the “environments.yaml” [1], but the juju version I am
> using is “2.2.4-xenial-amd64”, the MAAS version is 2.2.2,
>
> Initially, I installed juju 1.25 and configured environments.yaml, but now
> I have uninstalled juju 1.25, removed all those file in $home/.juju/ and
> start it over again with juju 2.2.4.
>
> I really cannot figure out why it always hangs at this step, is there any
> cache persisted anywhere that masked the  “authorized-keys-path” even after
> the uninstallation of juju1.25? or there is any step I missed with juju
> 2.2.4?
>
>
>
> Where is user-data of cloud-init persisted on the filesystem? Any more
> detail logs I can refer to?
>
>
>
>
>
> I feel frustration after trying several days without any progress, pls
> help me out, many many thanks for any inputs!
>
>
>
>
>
> [1]
> https://serverfault.com/questions/588967/juju-bootstrap-fails-connection-refused-port-22
>
>
>
>
>
> Best Regards,
>
> Dave Chen
>
>
>
> Best Regards,
>
> Dave Chen
>
>
>
>
> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20170930/3cd876b2/attachment.html>