[ubuntu-cloud] Fwd: Notes on using OpenStack with Juju
Jorge O. Castro
jorge at ubuntu.com
Mon Nov 26 14:58:00 UTC 2012
For those of you not on the juju list...
---------- Forwarded message ----------
From: Thomas Leonard <tal at it-innovation.soton.ac.uk>
Date: Fri, Nov 23, 2012 at 11:38 AM
Subject: Notes on using OpenStack with Juju
To: juju <juju at lists.ubuntu.com>
Hi,
We decided to set up an OpenStack system on a hosted machine to run
Juju services. I'm new to OpenStack and found it fairly
difficult/confusing. I'm sending a (slightly sanitised) version of the
notes I made during the process in case it's useful for others or
inspires someone to make it easier in future...
Hostname
Set the desired hostname first. OpenStack doesn't like it if you try
to change it later (like I did) and references to localhost will
remain.
LVM
OpeStack requires an LVM volume group, which must be named
“nova-volumes”. Create it on a spare partition with e.g.
vgcreate nova-volumes /dev/sda7
VPN
We don’t have enough IP addresses to give every VM a public IP.
Instead, the users must establish a VPN connection and use that to
manage the VMs. The few services which need to be public can be
assigned one of the limited public IPs available.
OpenStack supports a system called cloudpipe for automatically creating VPNs:
http://docs.openstack.org/trunk/openstack-compute/admin/content/cloudpipe-per-project-vpns.html
However, the instructions have many typos and other errors and I
couldn’t get it to work, so I installed OpenVPN manually instead:
http://openvpn.net/index.php/open-source/documentation/howto.html
I initially installed it in a VM, but there were some complex routing
issues with that, so now it's running on the main OpenStack host. Note
that the VPN is not for security (if we had IPv6, we’d probably just
make them all public IPs anyway), so I made a single client key-pair
for everyone (and configured the server to accept multiple clients
with the same certificate).
OpenStack
Ubuntu provides a guide for installing OpenStack:
https://help.ubuntu.com/community/UbuntuCloudInfrastructure
It gives three options:
- a live image: I imagine this won't persist after a reboot
- using MAAS: requires at least 10 physical machines and we only have one
- "the hard way": not documented
This seems to be the best guide for setting up OpenStack on Ubuntu
12.04 manually:
http://www.hastexo.com/resources/docs/installing-openstack-essex-4-ubuntu-1204-precise-pangolin/step-1-prepare-your-system
Fixed IPs
These are the internal (VPN-only) IPs assigned automatically to
machines. I created a 192.168.0.* network:
nova-manage network create private 192.168.0.0/24 1 256 --bridge=br100
Note: if you get it wrong and have to delete the network, it doesn’t
update the database properly and you'll get all kinds of weird bugs
about the old network no longer existing. You have to go in and edit
the database table like this:
delete from fixed_ips where network_id = 1;
Floating IPs
These are public IP addresses which can be assigned to machines as
needed. Register them with:
nova-manage floating create 1.2.3.4
...
To list them:
nova-manage floating list
Warning: if you delete a project, the floating IPs will not be
released. Use some SQL to null out the project_id field to recover the
IPs.
Diagnostics
You can get a list of running services like this:
# nova-manage service list
Binary Host Zone Status
State Updated_At
nova-consoleauth localhost nova enabled
XXX 2012-11-06 09:35:58
nova-scheduler localhost nova enabled
XXX 2012-11-06 09:36:02
nova-cert localhost nova enabled
XXX 2012-11-06 09:36:02
nova-compute localhost nova enabled
XXX 2012-11-06 09:35:53
nova-network localhost nova enabled
XXX 2012-11-06 09:36:01
nova-cert openstack nova enabled
:-) 2012-11-07 11:00:34
nova-consoleauth openstack nova enabled
:-) 2012-11-07 11:00:34
nova-scheduler openstack nova enabled
:-) 2012-11-07 11:00:34
nova-compute openstack nova enabled
:-) 2012-11-07 11:00:33
nova-network openstack nova enabled
:-) 2012-11-07 11:00:34
nova-volume openstack nova enabled
:-) 2012-11-07 11:00:34
Note that on my system it thinks there should be a set of "localhost"
services too because I renamed the machine after installing.
Volumes
Nova will detect the LVM group created above and allow creating
volumes via the web interface. However, you can’t attach them to nodes
until you follow this guide (it will just hang in the “attaching”
state):
http://docs.openstack.org/essex/openstack-compute/admin/content/managing-volumes.html
To unjam the volumes, some SQL is needed (see linked instructions
above). I was never able to get this working with Essex, but upgrading
to Folsom fixed the problems attaching volumes to instances.
Folsom
Folsom is available here:
http://blog.canonical.com/2012/09/14/now-you-can-have-your-openstack-cake-and-eat-it/
http://wiki.openstack.org/ReleaseNotes/Folsom
https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst
Note: the dashboard moves from http://myhost/ to http://myhost/horizon/
It requires a reboot before you can log in (restarting Apache,
keystone, etc, is not enough).
Swift
http://docs.openstack.org/trunk/openstack-compute/install/content/ch_installing-openstack-object-storage.html
Created a partition to hold the Swift data (I used LVM but you don't have to):
vgcreate storage /dev/sdb1
lvcreate --name swift --size 100G storage
Formatted as ext4 and added to fstab:
/dev/storage/swift /srv/node/storage/swift ext4
noatime,nodiratime,user_xattr 0 0
Create the rings. Make sure you set "1" for the number of replicas
(not 3 as in the tutorial) or it will fail silently later because you
only have one server:
cd /etc/swift
swift-ring-builder account.builder create 18 1 1
swift-ring-builder container.builder create 18 1 1
swift-ring-builder object.builder create 18 1 1
swift-ring-builder account.builder add z1-192.168.0.1:6002/swift 100
swift-ring-builder container.builder add z1-192.168.0.1:6001/swift 100
swift-ring-builder object.builder add z1-192.168.0.1:6000/swift 100
Note: DO NOT put a “/” in the drive name (e.g. storage/swift). It will
appear to work, but the Swift service will fail silently when you try
to use it.
Start the server:
# swift-init proxy start
OSError: [Errno 13] Permission denied: '/root/keystone-signing'
See: https://bugs.launchpad.net/keystone/+bug/1036847
chown swift /var/cache/swift
Note: change port to match the one in
/etc/keystone/default_catalog.templates (8888 -> 8080).
Test:
swift -V 2.0 -A http://localhost:5000/v2.0 -U admin -K $OS_PASSWORD stat
Account: AUTH_02ac15eeaab246a5a1ccfdcf8b9fc2d4
Containers: 0
Objects: 0
Bytes: 0
Accept-Ranges: bytes
X-Timestamp: 1352368359.21927
X-Trans-Id: txe265c083c6604d15b519b8f4c18d484c
Note: "401 Unauthorized" just means "an error occurred". Not
necessarily an authorisation failure (could be e.g. insufficient
storage available because the storage partition isn’t mounted, because
there's a "/" in the name).
Ignore the curl-based auth checks. They don’t work.
swift -V 2.0 -A http://localhost:5000/v2.0 -U admin -K $OS_PASSWORD
upload mycontainer myfile.tgz
The file uploads, but the web GUI fails to display it (“Unable to
retrieve container list").
keystone role-create --name swiftoperator
keystone user-role-add --user-id 67bb173cd78a47798c1a755884d1ccd3
--role-id 2e8bbe57a65c41a7a0a0898a1a4dd978 --tenant_id
0beec81cd4c34cd78c058b94823ca847
An unexpected error prevented the server from fulfilling your request.
'NoneType' object has no attribute 'get' (HTTP 500)
Despite the error, it does add the role.
Containers and folders can then be created by ordinary users via the web GUI.
S3
http://docs.openstack.org/trunk/openstack-compute/admin/content/configuring-swift-with-s3-emulation-to-use-keystone.html
Entry point 'swift3' not found in egg 'swift'.
Instructions are wrong. The entry point should be:
[filter:swift3]
use = egg:swift3#swift3
Trying to download Juju config from Web GUI gives:
“Error: Could not generate Juju environment config: Invalid service
catalog service: s3”
To check:
keystone service-list
+----------------------------------+----------+--------------+---------------------------+
| id | name | type |
description |
+----------------------------------+----------+--------------+---------------------------+
| 0ecd251795fb408da5d30ac562149d9e | nova | compute |
OpenStack Compute Service |
| 303e58c465ec458591091902f54bc913 | glance | image |
OpenStack Image Service |
| 3054d7477f3c4a1b9e3c4c6873d81075 | swift | object-store |
OpenStack Storage Service |
| ac57776cbd6f43bf87b8d3874fa92098 | s3 | s3 | s3
objectstore service |
| ea4a7a9829264f689ef5493f34c0b880 | volume | volume |
OpenStack Volume Service |
| eaa016ede163470ca62220b905c2884e | ec2 | ec2 |
OpenStack EC2 service |
| fb7ef4dbf4b44fffaa921b7d83fe24d8 | keystone | identity |
OpenStack Identity |
+----------------------------------+----------+--------------+---------------------------+
Shows "s3". But: "keystone catalog" doesn't.
note: /etc/keystone/default_catalog.templates is a decoy.
keystone endpoint-create --service-id ac57776cbd6f43bf87b8d3874fa92098
This will make Keystone stop working ('NoneType' object has no
attribute 'replace'). To fix that:
mysql -uroot keystone
update endpoint set extra='{"adminurl": "http://myhost:8080",
"internalurl": "http://myhost:8080", "publicurl":
"http://myhost:8080"}' where id='4de64caaaa6343d4beb67da8cdebd06f';
"keystone catalog" now includes "s3". However, the GUI gives the same error.
Downloads OK for admin user and admin project, but not for admin user
on regular project.
Mysteriously started working after a while.
Juju
After downloading the environment.yaml file from the web GUI
(settings), juju is able to bootstrap:
$ juju bootstrap
2012-11-08 11:31:06,467 WARNING ssl-hostname-verification is disabled
for this environment
2012-11-08 11:31:06,468 WARNING EC2 API calls not using secure transport
2012-11-08 11:31:06,468 WARNING S3 API calls not using secure transport
2012-11-08 11:31:06,468 WARNING Ubuntu Cloud Image lookups encrypted
but not authenticated
2012-11-08 11:31:06,469 INFO Bootstrapping environment 'openstack'
(origin: distro type: ec2)...
2012-11-08 11:31:10,146 INFO 'bootstrap' command finished successfully
However, it can’t connect to the new instance:
2012-11-08 11:31:48,832 ERROR Invalid host for SSH forwarding: ssh:
Could not resolve hostname
server-49fe7773-7ec4-4ddc-b947-762bde2ec9cf: Name or service not known
After adding the name to /etc/hosts manually:
$ juju status
2012-11-08 11:35:21,942 WARNING ssl-hostname-verification is disabled
for this environment
2012-11-08 11:35:21,943 WARNING EC2 API calls not using secure transport
2012-11-08 11:35:21,943 WARNING S3 API calls not using secure transport
2012-11-08 11:35:21,943 WARNING Ubuntu Cloud Image lookups encrypted
but not authenticated
2012-11-08 11:35:21,944 INFO Connecting to environment...
The authenticity of host 'server-49fe7773-7ec4-4ddc-b947-762bde2ec9cf
(192.168.0.3)' can't be established.
ECDSA key fingerprint is 9c:08:95:ea:a2:87:2c:10:eb:ce:f6:6b:e4:ab:e6:59.
Are you sure you want to continue connecting (yes/no)? yes
Warning: the ECDSA host key for
'server-49fe7773-7ec4-4ddc-b947-762bde2ec9cf' differs from the key for
the IP address '192.168.0.3'
Offending key for IP in /home/tal/.ssh/known_hosts:237
Are you sure you want to continue connecting (yes/no)? yes
2012-11-08 11:35:31,846 INFO Connected to environment.
machines:
0:
agent-state: running
dns-name: server-49fe7773-7ec4-4ddc-b947-762bde2ec9cf
instance-id: i-0000001f
instance-state: running
services: {}
2012-11-08 11:35:32,094 INFO 'status' command finished successfully
Gets stuck: S3 downloads to VMs aren’t working.
Note: the web-GUI caches your AWS keys. If you recreate a project then
it will continue to serve up the old keys in the environment.yaml.
Restart Apache.
There are two ways to fetch a file with S3: you can include an HTTP
header, or put the parameters in the HTTP GET URL. Seems that the
latter doesn’t work.
This is due to a problem checking the date. Swift expects a parsable
date, but S3 uses seconds since the epoch. Edit
/usr/lib/python2.7/dist-packages/swift3/middleware.py:483 to fix.
Am able to deploy the Juju wordpress/mysql example and make it public
(note that "juju expose" updates the firewall but doesn't assign a
public IP; use the web GUI to do that).
EC2 vs OpenStack API
By default, the EC2 API (used by the default environment.yaml returned
by horizon) returns made-up non-routable names and therefore doesn't
work. Changing nova to assign only IP addresses makes it mostly work,
but then the machines get numeric hostnames, which confuses e.g. the
postgresql charm because it then can't configure the mail server [ why
is it doing this anyway? ].
An alternative is to use the OpenStack provider type. A suitable
configuration looks like this:
juju: environments
default: openstack
environments:
openstack:
type: openstack_s3
control-bucket: juju-openstack-myproject-95ec-8c2083e67721
admin-secret: ...
auth-mode: userpass
auth-url: http://myhost:5000/v2.0/
username: tal
password: ...
project-name: myproject
default-series: precise
default-instance-type: m1.small
default-image-id: 5d7c1800-d778-47dd-946d-ebd22ee21e34
s3-uri: http://myhost:8080
combined-key: ... (EC2 access-key)
secret-key: (as for EC2)
The image ID is from "glance index". The provider must be
"openstack_s3", not "openstack". Otherwise, it fails to download the
metadata (Authentication failure).
The zookeeper machine still fails to boot because it gets an access
denied error trying to contact Swift (check
/var/log/cloud-init-output.log):
juju-admin: error: unrecognized arguments: <head> <title>401
Unauthorized</title> </head> <body> <h1>401 Unauthorized</h1> This
server could not verify that you are authorized to access the document
you requested. Either you supplied the wrong credentials (e.g., bad
password), or your browser does not understand how to supply the
credentials required.<br /><br /> Authentication required </body>
</html>
The command it tries is in /var/lib/cloud/instance/cloud-config.txt:
juju-admin initialize --instance-id=$(curl
http://myhost:8080/juju-openstack-myproject-95ec-8c2083e67721/juju_master_id?Signature=B3ec2HhzK0RoCxDRhS1BSUYEHkA%3D&Expires=1668074520&AWSAccessKeyId=902987dd775f4701b6c607b52c0ed26b)
--admin-identity=admin:Ulx9XFU1PJJE2+KNWbNENrT83fI=
--constraints-data=e3Byb3ZpZGVyLXR5cGU6IG9wZW5zdGFja19zMywgdWJ1bnR1LXNlcmllczogcHJlY2lzZX0K
--provider-type=openstack_s3'
The problem is a lack of quoting in the curl command. Edit
juju/providers/openstack/launch.py to add quotes.
It then bootstraps correctly.
Adding a new user
Create a new user using the dash. Make them a "swiftoperator". The
user should log in to the dash and go to Settings -> Download Juju
Environment Config.
They save this as ~/.juju/environment.yaml, adding their password in
the indicated places.
Then just "juju bootstrap".
Quotas
Quotas can be set in /etc/nova/nova.conf. It’s worth increasing some
of them a bit, e.g.
quota_instances=30
quota_security_groups=50
To change the quota for an existing project, use e.g.
nova quota-update 7ed23fe65dd24434838cef60cff37c75 --instances 50
SSL
Keystone has an SSL option, but enabling it breaks everything since it
just switches the existing port to SSL, which nothing else is
expecting.
The recommendation seems to be to put it behind Apache SSL.
Enable Apache SSL as usual (you can reuse the VPN certificates here).
This at least gets SSL for the dashboard.
https://myhost/horizon/
To use SSL with keystone, enable the mod_proxy and mod_proxy_http and
then add this to default-ssl:
ProxyPass /v2.0/ http://localhost:5000/v2.0/
Note: you have to use /v2.0/ (not e.g. /keystore/v2.0/).
Then, in your environments.yaml:
auth-url: https://myhost/v2.0/
Put the certificate in /usr/local/share/ca-certificates/myproject.crt
and run update-ca-certificates so Juju will find it. Note: MUST end
with ".crt".
S3 doesn’t have a fixed prefix like /v2.0/, so you need to make Apache
listen on a new port (e.g. 8443) and proxy that to 8080.
Problem: Juju fails to deploy successfully when using SSL with S3 (SSL
with keystone is fine):
2012-11-15 15:18:48,473 ERROR Machine provider information missing: machine 0
Cause: the cloudinit script tries to use https to download into the
initial VM, but it doesn’t trust the certificate. To make the VMs
trust our certificate, edit the template VM (see below):
Put certificates in /usr/local/share/ca-certificates/
sudo update-ca-certificates
Then curl works:
$ curl https://myhost/
<body><h1>It works!</h1>
(that it works with keystone suggests that keystone doesn't bother to
verify that the certificate is valid)
Next, proxy port 18774 (https) to 8774 (http) in the same way for
nova. Update the publicurl in the keystone/endpoint table. Make sure
you use the DNS name in the endpoint, not the IP address, or it won’t
verify.
Finally, update the endpoint table for the "object-store" (use
"keystone catalog" to get the ID). Redirect this to 8443 (which we
already forwarded for S3).
Now Juju runs without printing any warnings about security.
You also need to add the CA certificate to the main OpenStack machine
(the one running Horizon) or that will stop working.
Note: possible problem uploading large charms with SSL ("Connection to
the other side was lost in a non-clean fashion"). Needs investigating
further.
Horizon
Horizon is the web GUI dashboard for OpenStack. I had to make a few
changes to the Juju template generation:
The Juju configuration download was including the wrong credentials.
I’ve fixed it and filed a bug:
https://bugs.launchpad.net/ubuntu/+source/horizon/+bug/1079611
I also updated the forms code to pass the username and project name,
allowing those to be filled in automatically too.
I changed /usr/lib/python2.7/dist-packages/horizon/dashboards/settings/templates/settings/juju/environments.yaml.template
to:
juju: environments
environments:
{{ username }}:
type: openstack_s3
control-bucket: juju-{{ username }}
admin-secret: {{ juju_admin_secret }}
auth-mode: userpass
auth-url: https://myhost/v2.0/ (HTTPS URL)
username: {{ username }}
password: USER_PASSWORD
project-name: {{ project_name }}
default-series: precise
default-instance-type: m1.small
default-image-id: 247e6ec1-0d38-4927-a6ea-601a78d16c7f (not
sure this is needed actually)
s3-uri: https://myhost:8443/ (HTTPS S3 API)
combined-key: {{ ec2_access_key }}
secret-key: {{ ec2_secret_key }}
ssl-hostname-verification: true
juju-origin: ppa (Juju 0.5 doesn’t support openstack_s3)
Custom image
Deploying all the VMs with Juju is quite slow (~4 min per VM). Also,
the machines need to be pre-configured to trust the CA certificate. To
do this:
Create a new instance
Add all packages likely to be needed (dependencies, not the actual services)
Sync and snapshot
nova image-list
Wait until it finishes SAVING
The new image is in /var/lib/glance/images (it will be the most recent one)
glance add name="Ubuntu 12.04 Juju worker" is_public=true
container_format=ovf disk_format=qcow2 < image-file
This will display the new image ID. Update your environments.yaml to
refer to the new image.
Using the Juju client (these are the instructions we give to our users)
These instructions assume an Ubuntu 12.10 client.
1. In "Access & Security", ensure at least ICMP (ping) and SSH are enabled.
2. Enable the OpenStack VPN on your own machine
3. apt-get install juju
4. Fix it:
sudo vi /usr/lib/python2.7/dist-packages/juju/providers/openstack/launch.py
At line 74, add the missing quotes:
:74s/%s)/'%s')/
:wq
5. Download your Juju configuration from the web GUI (Settings)
Save as ~/.juju/environments.yaml
Edit to include your password where indicated
6. juju bootstrap
Wait a few minutes for the new machine to boot and install everything
7. juju status
--
Dr Thomas Leonard
IT Innovation Centre
Gamma House, Enterprise Road,
Southampton SO16 7NS, UK
tel: +44 23 8059 8866
mailto:tal at it-innovation.soton.ac.uk
http://www.it-innovation.soton.ac.uk/
--
Juju mailing list
Juju at lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju
--
Jorge Castro
Canonical Ltd.
http://juju.ubuntu.com
More information about the Ubuntu-cloud
mailing list