juju hangs during bootstrapping

Dave.Chen at Dell.com Dave.Chen at Dell.com
Wed Oct 11 01:57:09 UTC 2017


Thanks Narinder,  sorry for late response, just back from holiday, I think the issue has been solved by bootstrap with a config.yaml, where the config.yaml hold the proxy that is needed for external network access,

$ juju bootstrap maascloud maas-cloud-control --config=config.yaml
$ cat config.yaml
default-series: xenial
no-proxy: localhost
http-proxy: http://192.168.130.250:8000
https-proxy: http://192.168.130.250:8000
ftp-proxy: http://192.168.130.250:8000

I’m not clear why the proxy is needed since “192.168.130.250” is actually the IP address of MAAS server, anywhere, the bootstrap succeeded.

Best Regards,
Dave

From: Narinder Gupta [mailto:narinder.gupta at canonical.com]
Sent: Saturday, September 30, 2017 7:31 PM
To: Chen2, Dave <Dave_Chen2 at Dell.com>
Cc: juju <juju at lists.ubuntu.com>
Subject: RE: juju hangs during bootstrapping

Can u try nslookup for archive.ubuntu.com<http://archive.ubuntu.com> looks like upstream dns issue as well apart from network as you mentioned it was broken.
Thanks and regards,
Narinder Gupta
1-2817365150

On Sep 30, 2017 04:54, <Dave.Chen at dell.com<mailto:Dave.Chen at dell.com>> wrote:
Hi Narinder,

Here is the log from the deployed node, actually, I can deploy Operation System successfully with MAAS or “juju bootstrap” but failed at some final steps, Our external network is exactly broken for some reason yesterday and won’t be recovered in short term, but I guess the network broken is  happened after I saw the below error message during bootstrapping.
“Waiting for address
Attempting to connect to 10.20.3.254:22”

I can see apt-get update works from the log at the beginning, and the network is broken couple of hours after I saw those error message.

Does the error message like below has any connection with connection refused error message? I am not quite sure.

“0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Fetching Juju agent version 2.2.4 for amd64
Attempt 1 to download tools from https://streams.canonical.com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz...
curl: (6) Could not resolve host: streams.canonical.com<http://streams.canonical.com>
“

Anyway, I will try it again when our network back to normal, it would be great if you can see any other issue beside the network, thanks again for your help!

Best Regards,
Dave Chen

From: Narinder Gupta [mailto:narinder.gupta at canonical.com<mailto:narinder.gupta at canonical.com>]
Sent: Friday, September 29, 2017 10:52 PM
To: Chen2, Dave <Dave_Chen2 at Dell.com<mailto:Dave_Chen2 at Dell.com>>
Cc: juju <juju at lists.ubuntu.com<mailto:juju at lists.ubuntu.com>>
Subject: Re: juju hangs during bootstrapping

Hi Dave,
May I know which division of Dell you are working on? As i have setup Openstack deployed t Dell multiple time with MAAS and have not seen this issue so far.

So please send me log /var/log/cloud-init-output.log which will let us know what is wrong. Also try sudo apt-get update on the bootstrap node to confirm you have external access.

In MAAS you can always add the ssh keys to land into the installed nodes though.



Thanks and Regards,

Narinder Gupta (PMP)                   narinder.gupta at canonical.com<mailto:narinder.gupta at canonical.com>

Canonical, Ltd.                    narindergupta [irc.freenode.net<http://irc.freenode.net>]

+1.281.736.5150<tel:(281)%20736-5150>                            narindergupta2007[skype]



Ubuntu- Linux for human beings | www.ubuntu.com<http://www.ubuntu.com> | www.canonical.com<http://www.canonical.com>

On Fri, Sep 29, 2017 at 9:42 AM, <Dave.Chen at dell.com<mailto:Dave.Chen at dell.com>> wrote:
Hi All,

I am trying to bootstrap a MAAS cloud based on juju’s official guide (https://jujucharms.com/docs/2.2/clouds-maas), everything seems correct but after the Operation System (Ubuntu 16.04 or Ubuntu14.0) has been installed, juju hangs when attempting to connect to the MAAS node, here is what I can see from the terminal,


$ juju bootstrap maas-cloud

Creating Juju controller "maas-cloud" on maas-cloud

Looking for packaged Juju agent version 2.2.4 for amd64

Launching controller instance(s) on maas-cloud...

- cka68p (arch=amd64 mem=32G cores=12)

Fetching Juju GUI 2.9.2

Waiting for address

Attempting to connect to 10.20.3.254:22<http://10.20.3.254:22> (JUJU hangs here!)

And it’s pending here forever, so I tried it again with the debug mode,

$ juju bootstrap --show-log --debug --bootstrap-series=trusty maas-cloud maas-cloud-controller



I saw some detail information like below,

Attempting to connect to 10.20.3.254:22<http://10.20.3.254:22>

19:33:11 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22: Connection refused

19:33:16 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22: Connection refused

19:33:21 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22: Connection refused

19:33:56 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist

19:34:32 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist

19:35:08 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist

19:35:43 INFO  juju.cloudconfig userdatacfg_unix.go:410 Fetching agent: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz]><https://streams.canonical.com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz%5d%3e>

I have no idea what’s going wrong since I can telnet to the node and ssh to that node is also possible, I just need type “yes” then I can login to the node,
$ ssh ubuntu at 10.20.3.254<mailto:ubuntu at 10.20.3.254>
The authenticity of host ' 10.20.3.254 (10.20.3.254)' can't be established.
ECDSA key fingerprint is SHA256:4FVm21s4dx7gc0/yDgz0+QAMGK4qWODoIqeoWtZg9RI.
Are you sure you want to continue connecting (yes/no)?

From the console of that node, I can find the controller’s public key has been injected to the node,

-----BEGIN SSH HOST KEY KEYS-------

…

-----END SSH KEY FINGERPRINTS--------
…
Cloud-init v. 0.7.9 finished at … Datasource DataSourceMAAS [http://...:5240/MAAS/metadata/]. Up 153.77 seconds.       (cloud-init hangs here!)


I googled it and found someone said it is because “authorized-keys-path” is commented out in the “environments.yaml” [1], but the juju version I am using is “2.2.4-xenial-amd64”, the MAAS version is 2.2.2,
Initially, I installed juju 1.25 and configured environments.yaml, but now I have uninstalled juju 1.25, removed all those file in $home/.juju/ and start it over again with juju 2.2.4.
I really cannot figure out why it always hangs at this step, is there any cache persisted anywhere that masked the  “authorized-keys-path” even after the uninstallation of juju1.25? or there is any step I missed with juju 2.2.4?

Where is user-data of cloud-init persisted on the filesystem? Any more detail logs I can refer to?


I feel frustration after trying several days without any progress, pls help me out, many many thanks for any inputs!


[1]  https://serverfault.com/questions/588967/juju-bootstrap-fails-connection-refused-port-22


Best Regards,
Dave Chen

Best Regards,
Dave Chen


--
Juju mailing list
Juju at lists.ubuntu.com<mailto:Juju at lists.ubuntu.com>
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20171011/027fee4f/attachment.html>


More information about the Juju mailing list