Re: Multi install with existing MAAS starts all services except for “IP Pending” on Glance Simplestreams Image Sync
Jeff McLamb
mclamb at gmail.com
Wed Jul 29 20:15:54 UTC 2015
OK, making progress now. Per your recommendation I removed and added
back in the trusty sync charm manually.
Now, I can log in to the horizon dashboard!
However, several tabs result in a generic OpenStack (not
Ubuntu-customized like the general dashboard pages) "Something went
wrong! An unexpected error has occurred. Try refreshing the page..."
The tabs in question that give those results are Compute -> Access &
Security, Network -> Network Topology,
When I go to pages like Network -> Routers, it does render, but there
are error popup boxes in the page itself with:
Error: Unable to retrieve router list.
and
Error: Unable to retrieve a list of external networks "Connection to
neutron failed: HTTPConnectionPool(host='192.168.1.45', port=9696):
Max retries exceeded with url:
/v2.0/networks.json?router%3Aexternal=True (Caused by <class
'httplib.BadStatusLine'>: '')”.
If I do a `juju ssh openstack-dashboard/0` and tail -f
/var/log/apache2/error.log I get the following when accessing one of
the failed pages:
http://paste.ubuntu.com/11961863/
Furthermore, looking at the neutron server logs, I see non-stop traces
about the neutron.ml2_gre_allocations table not existing:
http://paste.ubuntu.com/11961891/
Getting closer, bit by bit.
Thanks for all the help,
Jeff
On Wed, Jul 29, 2015 at 1:49 PM, Mike McCracken
<mike.mccracken at canonical.com> wrote:
> Jeff, you can use --edit-placement to edit what goes where, and remove the
> sync service.
> Then you will just have to import a cloud image into glance yourself.
>
> Or, you could let it try to deploy the sync service, then do a little bit of
> tweaking and manual juju work to deploy the sync script as trusty instead of
> vivid, on a default install where everything else is trusty.
>
> on the host you're running the openstack installer from
> export JUJU_HOME=~/.cloud-install/juju
>
> since you have a failed sync service hanging around, first remove it with
> this:
> juju remove-service glance-simplestreams-sync
>
> then look in ~/.cloud-install/local-charms/
> there should be a directory 'vivid' with subdir glance-simplestreams-sync.
> if you rename that directory to 'trusty', then re-deploy the sync charm
> manually, it should work.
>
> to redeploy, do this:
> juju deploy --repository=~/.cloud-install/local-charms
> local:trusty/glance-simplestreams-sync --to lxc:3 --config
> ~/.cloud-install/charmconf.yaml
>
> -mike
>
> On Wed, Jul 29, 2015 at 10:32 AM, Jeff McLamb <mclamb at gmail.com> wrote:
>>
>> Hey Mike -
>>
>> I am running MAAS Version 1.8.0+bzr4001-0ubuntu2 (vivid1) according to
>> the version strip at the bottom of the MAAS Web GUI.
>>
>> Sure, will gladly file the bug report.
>>
>> At this point, in hopes of getting something up and running to a
>> horizon dashboard I can log into, I am going to try to do the
>> install-only mode of openstack-install, then manually select services
>> but not glance sync. I presume I can get a working setup without sync,
>> which I don't really need in the first place, right?
>>
>> Thanks,
>>
>> Jeff
>>
>>
>>
>> On Wed, Jul 29, 2015 at 1:26 PM, Mike McCracken
>> <mike.mccracken at canonical.com> wrote:
>> > Hi Jeff, your pastebin is showing that MAAS is complaining about the
>> > distro
>> > series, and the installer is just forwarding that error along.
>> > What version of MAAS are you running?
>> >
>> > As for the error with running a vivid container, that seems like a bug
>> > in
>> > juju, and if you have the time, I'd appreciate it if you could file it
>> > as a
>> > bug against lp:juju-core, including the info in your earlier email, in
>> > particular the stuff around "Failed to mount cgroup at
>> > /sys/fs/cgroup/systemd: Permission denied".
>> >
>> > It seems like there's a systemd-changeover related issue here that the
>> > juju
>> > folks will want to know about.
>> >
>> > Thanks!
>> > -mike
>> >
>> > On Wed, Jul 29, 2015 at 10:21 AM, Jeff McLamb <mclamb at gmail.com> wrote:
>> >>
>> >> This bug report seems to be related, if not the same thing:
>> >>
>> >> https://bugs.launchpad.net/landscape/+bug/1451385
>> >>
>> >> On Wed, Jul 29, 2015 at 1:10 PM, Jeff McLamb <mclamb at gmail.com> wrote:
>> >> > Here is the output from trying with --series vivid:
>> >> >
>> >> > http://paste.ubuntu.com/11960777/
>> >>
>> >> >
>> >> > I also did a re-install with --series trusty (really no different
>> >> > from
>> >> > the first time) and everything turned out the same, except the order
>> >> > in which containers came up and what got assigned to which container.
>> >> >
>> >> > However, logging into the physical node (machine 3) that hosts the
>> >> > containers, I found the console log of the actual vivid template lxc
>> >> > when it is in the RUNNING state and the glance sync container is the
>> >> > only one left to provision.
>> >> >
>> >> > The errors below seem to make sense with respect to the complaint in
>> >> > `juju status` that says it could not clone properly because the
>> >> > template container did not terminate:
>> >> >
>> >> > ubuntu at cooperative-crowd:~$ sudo cat
>> >> > /var/lib/juju/containers/juju-vivid-lxc-template/console.log
>> >> > Failed to mount cgroup at /sys/fs/cgroup/systemd: Permission denied
>> >> > systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA
>> >> > +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS +ACL
>> >> > +XZ -LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN)
>> >> > Detected virtualization lxc.
>> >> > Detected architecture x86-64.
>> >> >
>> >> > Welcome to Ubuntu 15.04!
>> >> >
>> >> > Set hostname to <juju-vivid-lxc-template>.
>> >> > Initializing machine ID from random generator.
>> >> > Failed to install release agent, ignoring: No such file or directory
>> >> > Failed to create root cgroup hierarchy: No such file or directory
>> >> > Failed to allocate manager object: No such file or directory
>> >> > [!!!!!!] Failed to allocate manager object, freezing.
>> >> >
>> >> > On Wed, Jul 29, 2015 at 12:35 PM, Mike McCracken
>> >> > <mike.mccracken at canonical.com> wrote:
>> >> >> Can you share what goes wrong when you tried --series vivid ?
>> >> >> It's always supported specifying whatever you like there, so it
>> >> >> should
>> >> >> work,
>> >> >> and if something complained, that's a bug.
>> >> >>
>> >> >> Thanks,
>> >> >> -mike
>> >> >>
>> >> >> On Wed, Jul 29, 2015 at 9:20 AM, Jeff McLamb <mclamb at gmail.com>
>> >> >> wrote:
>> >> >>>
>> >> >>> It seems as though vivid is not a valid --series name in my version
>> >> >>> of
>> >> >>> openstack-installer 0.99.19
>> >> >>>
>> >> >>> I can run it with trusty just fine, but I suspect that that will
>> >> >>> lead
>> >> >>> to the same issue where glance is installed as vivid and the rest
>> >> >>> are
>> >> >>> installed as trusty?
>> >> >>>
>> >> >>> Thanks,
>> >> >>>
>> >> >>> Jeff
>> >> >>>
>> >> >>>
>> >> >>> On Wed, Jul 29, 2015 at 11:48 AM, Mike McCracken
>> >> >>> <mike.mccracken at canonical.com> wrote:
>> >> >>> > If you're able, I think the easiest way is to tear it down and
>> >> >>> > re-deploy.
>> >> >>> > Anything else would be a lot of manual steps. Juju has some
>> >> >>> > upgrade
>> >> >>> > capabilities, but the openstack installer pre-dates them, and so
>> >> >>> > you'd
>> >> >>> > have
>> >> >>> > to hand-edit some config files, etc.
>> >> >>> >
>> >> >>> > -mike
>> >> >>> >
>> >> >>> > On Wed, Jul 29, 2015 at 8:44 AM, Jeff McLamb <mclamb at gmail.com>
>> >> >>> > wrote:
>> >> >>> >>
>> >> >>> >> Cool, thanks for the tip, Mike.
>> >> >>> >>
>> >> >>> >> Yes the host is vivid running juju 1.24.3-vivid-amd64
>> >> >>> >>
>> >> >>> >> Given my current state, should I kill the environment and start
>> >> >>> >> again,
>> >> >>> >> or is there a way I can deploy or update what's currently there
>> >> >>> >> without having to start over?
>> >> >>> >>
>> >> >>> >> Thanks,
>> >> >>> >>
>> >> >>> >> Jeff
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> On Wed, Jul 29, 2015 at 11:39 AM, Mike McCracken
>> >> >>> >> <mike.mccracken at canonical.com> wrote:
>> >> >>> >> > Hi Jeff, so it looks like you're running vivid on the host, is
>> >> >>> >> > that
>> >> >>> >> > right?
>> >> >>> >> > Your point that the sync charm is the only one deployed as
>> >> >>> >> > vivid
>> >> >>> >> > is a
>> >> >>> >> > good
>> >> >>> >> > catch - while this should still work, it might be causing
>> >> >>> >> > problems
>> >> >>> >> > for
>> >> >>> >> > juju's local provider.
>> >> >>> >> > I will fix the deploy code for the sync charm to use the same
>> >> >>> >> > series
>> >> >>> >> > as
>> >> >>> >> > the
>> >> >>> >> > other charms, but one thing you can try in the meantime is use
>> >> >>> >> > --series
>> >> >>> >> > vivid to just deploy the whole thing as vivid.
>> >> >>> >> > This has been tested and should work.
>> >> >>> >> >
>> >> >>> >> > -mike
>> >> >>> >> >
>> >> >>> >> > On Wed, Jul 29, 2015 at 8:25 AM, Jeff McLamb
>> >> >>> >> > <mclamb at gmail.com>
>> >>
>> >> >>> >> > wrote:
>> >> >>> >> >>
>> >> >>> >> >> Hey Adam, Mike -
>> >> >>> >> >>
>> >> >>> >> >> The environment is up, but the container that should contain
>> >> >>> >> >> glance-simplestreams is not up at all. Here is the lxc-ls
>> >> >>> >> >> output
>> >> >>> >> >> from
>> >> >>> >> >> the node that contains all of the infrastructure containers:
>> >> >>> >> >>
>> >> >>> >> >> ubuntu at cooperative-crowd:~$ sudo lxc-ls --fancy
>> >> >>> >> >> NAME STATE IPV4 IPV6
>> >> >>> >> >> AUTOSTART
>> >> >>> >> >>
>> >> >>> >> >> ----------------------------------------------------------------
>> >> >>> >> >> juju-machine-3-lxc-0 RUNNING 192.168.1.17 - YES
>> >> >>> >> >> juju-machine-3-lxc-1 RUNNING 192.168.1.18 - YES
>> >> >>> >> >> juju-machine-3-lxc-2 RUNNING 192.168.1.19 - YES
>> >> >>> >> >> juju-machine-3-lxc-3 RUNNING 192.168.1.20 - YES
>> >> >>> >> >> juju-machine-3-lxc-4 RUNNING 192.168.1.21 - YES
>> >> >>> >> >> juju-machine-3-lxc-6 RUNNING 192.168.1.22 - YES
>> >> >>> >> >> juju-machine-3-lxc-7 RUNNING 192.168.1.23 - YES
>> >> >>> >> >> juju-trusty-lxc-template STOPPED - - NO
>> >> >>> >> >> juju-vivid-lxc-template STOPPED - - NO
>> >> >>> >> >> ubuntu at cooperative-crowd:~$
>> >> >>> >> >>
>> >> >>> >> >> As you can see, the lxc-5 machine that should be running is
>> >> >>> >> >> not.
>> >> >>> >> >>
>> >> >>> >> >> When I look at openstack-status all of the services have a
>> >> >>> >> >> green
>> >> >>> >> >> checkmark and are [started] except for Glance Simplestreams
>> >> >>> >> >> which is
>> >> >>> >> >> in the [pending] stage.
>> >> >>> >> >>
>> >> >>> >> >> In the place of an IP address it says IP Pending and "waiting
>> >> >>> >> >> for
>> >> >>> >> >> initial status".
>> >> >>> >> >>
>> >> >>> >> >> I was wondering if I was running out of IP addresses in that
>> >> >>> >> >> range
>> >> >>> >> >> on
>> >> >>> >> >> the MAAS DHCP server, but there are only 8 containers and
>> >> >>> >> >> there
>> >> >>> >> >> are
>> >> >>> >> >> plenty of addresses left in that range.
>> >> >>> >> >>
>> >> >>> >> >> Here is the output of juju status where it is revealed that:
>> >> >>> >> >>
>> >> >>> >> >> 3/lxc/5:
>> >> >>> >> >> agent-state-info: 'failed to retrieve the template to
>> >> >>> >> >> clone:
>> >> >>> >> >> template container
>> >> >>> >> >> "juju-vivid-lxc-template" did not stop'
>> >> >>> >> >> instance-id: pending
>> >> >>> >> >> series: vivid
>> >> >>> >> >>
>> >> >>> >> >> For some reason this is the only container that it tries to
>> >> >>> >> >> install
>> >> >>> >> >> using the vivid series. All other containers use trusty.
>> >> >>> >> >>
>> >> >>> >> >> Here is the full juju status output:
>> >> >>> >> >>
>> >> >>> >> >> http://paste.ubuntu.com/11960193/
>> >> >>> >> >>
>> >> >>> >> >> Thanks!
>> >> >>> >> >>
>> >> >>> >> >> Jeff
>> >> >>> >> >>
>> >> >>> >> >> On Wed, Jul 29, 2015 at 11:15 AM, Adam Stokes
>> >> >>> >> >> <adam.stokes at canonical.com>
>> >> >>> >> >> wrote:
>> >> >>> >> >> > Jeff,
>> >> >>> >> >> >
>> >> >>> >> >> > Is the environment still up? Can you `juju ssh
>> >> >>> >> >> > glance-simplestreams-sync/0`
>> >> >>> >> >> > and grab the unit logs from `/var/log/juju/`.
>> >> >>> >> >> >
>> >> >>> >> >> > CCing Mike for further investigation into that service
>> >> >>> >> >> >
>> >> >>> >> >> > On Wed, Jul 29, 2015 at 10:25 AM, Jeff McLamb
>> >> >>> >> >> > <mclamb at gmail.com>
>> >> >>> >> >> > wrote:
>> >> >>> >> >> >>
>> >> >>> >> >> >> Hey Adam -
>> >> >>> >> >> >>
>> >> >>> >> >> >> Sure, here it is:
>> >> >>> >> >> >>
>> >> >>> >> >> >> http://paste.ubuntu.com/11959888/
>> >> >>> >> >> >>
>> >> >>> >> >> >> Thanks!
>> >> >>> >> >> >>
>> >> >>> >> >> >> Jeff
>> >> >>> >> >> >>
>> >> >>> >> >> >>
>> >> >>> >> >> >> On Wed, Jul 29, 2015 at 10:07 AM, Adam Stokes
>> >> >>> >> >> >> <adam.stokes at canonical.com>
>> >> >>> >> >> >> wrote:
>> >> >>> >> >> >> > Hey Jeff,
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > Can you paste.ubuntu.com your
>> >> >>> >> >> >> > ~/.cloud-install/commands.log?
>> >> >>> >> >>
>> >> >>> >> >> >>
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > On Wed, Jul 29, 2015 at 1:38 AM, Jeff McLamb
>> >> >>> >> >> >> > <mclamb at gmail.com>
>>
>> >>
>> >> >>>
>> >> >>> >>
>> >> >>> >> >> >> > wrote:
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> Hello -
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> Due to Adam’s suggestion in a previous thread about
>> >> >>> >> >> >> >> Autopilot
>> >> >>> >> >> >> >> not
>> >> >>> >> >> >> >> working properly with a proxy running (e.g. masa-proxy)
>> >> >>> >> >> >> >> on
>> >> >>> >> >> >> >> the
>> >> >>> >> >> >> >> MAAS
>> >> >>> >> >> >> >> server, I killed the environment and started over with
>> >> >>> >> >> >> >> the
>> >> >>> >> >> >> >> Multi
>> >> >>> >> >> >> >> option.
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> Interesting aside: you must openstack-install -k and
>> >> >>> >> >> >> >> openstack-install
>> >> >>> >> >> >> >> -u or otherwise remove ~/.cloud-install entirely
>> >> >>> >> >> >> >> because
>> >> >>> >> >> >> >> if
>> >> >>> >> >> >> >> you
>> >> >>> >> >> >> >> just
>> >> >>> >> >> >> >> do a -k and do not remove the landscape related files,
>> >> >>> >> >> >> >> it
>> >> >>> >> >> >> >> will
>> >> >>> >> >> >> >> still
>> >> >>> >> >> >> >> try to install an Autopilot-based setup even if you
>> >> >>> >> >> >> >> choose
>> >> >>> >> >> >> >> Multi.
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> Anyway, the Multi install progressed well,
>> >> >>> >> >> >> >> bootstrapping
>> >> >>> >> >> >> >> juju
>> >> >>> >> >> >> >> with
>> >> >>> >> >> >> >> the
>> >> >>> >> >> >> >> VM node, followed by deploying services across 3 other
>> >> >>> >> >> >> >> physical
>> >> >>> >> >> >> >> nodes
>> >> >>> >> >> >> >> that are all big RAM and big core.
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> One physical node was chosen as dedicated compute, one
>> >> >>> >> >> >> >> as
>> >> >>> >> >> >> >> dedicated
>> >> >>> >> >> >> >> Neutron, and the third was chosen for infrastructure,
>> >> >>> >> >> >> >> deploying
>> >> >>> >> >> >> >> what
>> >> >>> >> >> >> >> should be 8 containers.
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> In the openstack-status view I see that all services
>> >> >>> >> >> >> >> have
>> >> >>> >> >> >> >> “started”
>> >> >>> >> >> >> >> except for the Glance - Simplestreams Image Sync
>> >> >>> >> >> >> >> service
>> >> >>> >> >> >> >> which
>> >> >>> >> >> >> >> is
>> >> >>> >> >> >> >> to
>> >> >>> >> >> >> >> be deployed as Container 5 on the infrastructure
>> >> >>> >> >> >> >> machine
>> >> >>> >> >> >> >> (Machine
>> >> >>> >> >> >> >> 3
>> >> >>> >> >> >> >> in
>> >> >>> >> >> >> >> my case). It is stuck in “IP Pending” state.
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> I have rebooted the infrastructure node and all lxc
>> >> >>> >> >> >> >> containers
>> >> >>> >> >> >> >> came
>> >> >>> >> >> >> >> back up, except a lxc-ls —fancy does not show the
>> >> >>> >> >> >> >> existence of
>> >> >>> >> >> >> >> juju-machine-3-lxc-5. It does have STOPPED
>> >> >>> >> >> >> >> juju-trusty-lxc-template
>> >> >>> >> >> >> >> and juju-vivid-lxc-template. The vivid one was actually
>> >> >>> >> >> >> >> RUNNING
>> >> >>> >> >> >> >> before
>> >> >>> >> >> >> >> I rebooted the system, but now it is STOPPED.
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> I can get to the horizon dashboard GUI but I cannot
>> >> >>> >> >> >> >> login
>> >> >>> >> >> >> >> with
>> >> >>> >> >> >> >> a
>> >> >>> >> >> >> >> “An
>> >> >>> >> >> >> >> error occurred authenticating. Please try again later.”
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> Any ideas how to get that last service running so I can
>> >> >>> >> >> >> >> hopefully
>> >> >>> >> >> >> >> log
>> >> >>> >> >> >> >> into the horizon GUI for the first time?
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> Thanks,
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> Jeff
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> --
>> >> >>> >> >> >> >> ubuntu-openstack-installer mailing list
>> >> >>> >> >> >> >> ubuntu-openstack-installer at lists.ubuntu.com
>> >> >>> >> >> >> >> Modify settings or unsubscribe at:
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> https://lists.ubuntu.com/mailman/listinfo/ubuntu-openstack-installer
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >
>> >> >>> >> >> >
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >
>> >> >>> >
>> >> >>
>> >> >>
>> >
>> >
>
>
More information about the ubuntu-openstack-installer
mailing list