network spaces - aws support

Wed Aug 3 10:17:01 UTC 2016

Hey James,

Thanks for the detailed information! I think you did find a bug, let me
try to explain.

First, it looks like your vpc-ff069a98 meets the minimum connectivity
requirements of Juju (i.e. it's deemed suitable to host a controller),
so passing --config vpc-id-force=true at bootstrap shouldn't be
necessary (if it is, I'd like to know more about how your VPC's subnets
and route tables look like, if possible!).

The bug I think you've discovered is related to how Juju does automatic
distribution of instances across AZ for redundancy. Since EC2 subnets
can only be part of one AZ, and when using a non-default VPC the AWS API
requires you to pass SubnetID, not just AvailZone to RunInstances used
to start an instance. Security groups are VPC-specific as well.

The essence of the issue I think is demonstrated here:

2016-08-02 16:57:13 INFO juju.provider.ec2 environ.go:516 selected
random subnet "subnet-f30d3fd9" from all matching in zone "us-east-1a":
[subnet-9b013ab1 subnet-f30d3fd9 subnet-fb380ad1]

I strongly suspect those other 2 subnets: subnet-9b013ab1,
subnet-fb380ad1 in AZ us-east-1a are part of a different VPC, but are
still picked up "randomly" during StartInstance in the EC2 provider.

In other words Juju is not filtering subnets in a given AZ by the
user-specified vpc-id when deciding which one to pick.

I looked at the code and it looks easy to fix this, fortunately.
If I give you a patch to apply on top of the tip of juju/juju master
branch, would you be willing to give it a try to see if it works?

Thanks for your patience! I'll file a bug about what you've discovered
and start working on a fix for beta14!

Cheers,
Dimiter

On 08/02/2016 08:33 PM, James Beedy wrote:
> Ok, here we go .... hopefully this will provide a better overhead view.
> 
> `juju bootstrap creativedrive aws --credential creativedrive --config
> vpc-id=vpc-ff069a98 --config vpc-id-force='true' --upload-tools --debug
> --config logging-config='<root>=TRACE'` <- http://paste.ubuntu.com/21914113/
> 
> cat machine-0.log <- http://paste.ubuntu.com/21914847/
> 
> 
> ### debug add-model
> 
> `juju add-model consul --credential creativedrive --config
> vpc-id=vpc-ff069a98 --config vpc-id-force='true' --debug` <-
> http://paste.ubuntu.com/21915182/
> 
> ### juju status on new model
> 
> juju status --format yaml <- http://paste.ubuntu.com/21915512/
> 
> ### debug add space and subnet
> 
> `juju add-space common-infrastructure --debug` <-
> http://paste.ubuntu.com/21915888/
> 
> #### add a subnet in my vpc to the newly created space
> `juju add-subnet subnet-9b013ab1 common-infrastructure us-east-1a
> --debug` <- http://paste.ubuntu.com/21916055/
> 
> ### list spaces and subnets
> 
> `juju subnets && juju spaces` <- http://paste.ubuntu.com/21916258/
> 
> 
> ### deploy something to the model
> `juju deploy ubuntu --debug` <- http://paste.ubuntu.com/21916540/
> 
> 
> ### deploy something to a network space
> 
> `juju deploy ubuntu ubuntu-spaces --constraints
> spaces=common-infrastructure --debug` <- http://paste.ubuntu.com/21916804/
> 
> 
> ### juju status now shows a success for the machine deployed to the
> model w/out a space constraint, and and error for the instance deployed
> to the space.
> 
> `juju status --format yaml` <- http://paste.ubuntu.com/21917006/
> 
> 
> No matter what, I can't seem to get anything deployed to a "space"....
> 
> So strange ... possibly I have stumbled upon a bug?
> 
> 
> Thanks again for your insight here.
> 
> 
> 
> On Tue, Aug 2, 2016 at 9:51 AM, James Beedy <jamesbeedy at gmail.com
> <mailto:jamesbeedy at gmail.com>> wrote:
> 
>     To my utter dismay, setting the correct config 'vpc-id-force' gave
>     me the same result....
> 
>     Let me scrub and collect my machine-0.log for you.
> 
> 
>      
> 
>     On Tue, Aug 2, 2016 at 9:36 AM, James Beedy <jamesbeedy at gmail.com
>     <mailto:jamesbeedy at gmail.com>> wrote:
> 
>         Dimiter,
> 
>         Thanks for the insight.
>         /
>         /
>         /Can you please also paste the full logs (scrubbed of secrets)
>         of `juju
>         bootstrap ... --debug` (with the vpc-id etc., but please also
>         include
>         `--config logging-config='<root>=TRACE'`), and machine-0.log from
>         /var/log/juju on the bootstrap node, once completed? That will help
>         figuring out the issue.
>         /
>         `juju bootstrap creativedrive aws --credential creativedrive
>         --config vpc-id=vpc-ff069a98 --config force-vpc-id='true'
>         --config loggin-config='<root>=TRACE' --upload-tools --debug` <-
>         http://paste.ubuntu.com/21908548/
> 
>         machine-0.log shows  "2016-08-02 16:16:16 TRACE juju.apiserver
>         request_notifier.go:127 -> [2] machine-0
>         {"request-id":53,"response":{"config":{"access-key":"","agent-version":"2.0-beta13","authorized-keys":"juju-client-key\nssh-rsa
>         ssh-rsa
>         juju-system-key\n","automatically-retry-hooks":true,"default-series":"","development":false,"disable-network-management":false,"firewall-mode":"instance","force-vpc-id":true,"ignore-machine-addresses":false,"logging-config":"\u003croot\u003e=TRACE;unit=DEBUG","name":"controller","proxy-ssh":false,"region":"us-east-1","secret-key":"/E","ssl-hostname-verification":true,"storage-default-block-source":"ebs","test-mode":false,"type":"ec2","uuid":"259be235-a255-416d-8bbf-75e128d05794","vpc-id":"vpc-ff069a98","vpc-id-force":false}}}"
> 
> 
>         Just realizing now, I have been specifying 'vpc-force-id', not
>         'vpc-id-force' (grrrr).
> 
>         I would expect to see this resolved when I apply the correct
>         config. I'll report back shortly.
> 
>         Thanks for your time!
> 
>         /From what I can understand, you're trying to bootstrap on a
>         non-default,
>         possibly private VPC (accessed via its internal address over a VPN
>         connection maybe?), and then add a model with the same VPC and
>         credentials.
> 
>         /
>         ^ Exactly/
>         /
>         /
>          If OTOH, the VPC used for add-model is different, the
>         machines there won't be able to talk to the controller's VPC
>         unless it
>         has a public address (cross VPC communication currently relies
>         on having
>         that, fancier setups with VPN gateways is not yet supported)./
> 
>         ^
> 
>         The error in status implies 2 separate VPCs are used (or a VPC and
>         EC2-Classic - i.e. no VPC) for the controller and hosted model.
> 
>         Cheers,
>         Dimiter
> 
> 
> 
> 
> 

-- 
Dimiter Naydenov <dimiter.naydenov at canonical.com>
Juju Core Sapphire team <http://juju.ubuntu.com>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20160803/afc5885b/attachment.pgp>