LXD polish for xenial

Tue Apr 19 00:06:24 UTC 2016

On Mon, Apr 18, 2016, 6:12 PM Tycho Andersen <tycho.andersen at canonical.com>
wrote:

> Hi Martin,
>
> Thanks for looking into this.
>
> On Mon, Apr 18, 2016 at 07:17:29PM +0100, Martin Packman wrote:
> > With the LXD 2.0 release at the start of last week and the prospect of
> > some stability, I spent a good chunk of the week testing of Juju and
> > LXD.
> >
> > What CI has been doing so far this cycle has been running our standard
> > deployment tests with the lxd provider on a pre-prepared machine with
> > a known-working package set. My two goals beyond this were:
> >
> > - Validating the first install experience of the LXD provider on xenial
> > - Using lxd in place of lxc in containerised workloads across clouds
> >
> > The conclusion being at present, we don't have an experience that
> > works for either of these.
> >
> > For the lxd provider, I understand we're resigned to the user having
> > to manually configure a bridge for lxd before bootstrap can work.
> > Currently the documentation is confused as to what exactly the steps
> > are, the release notes refer to these two links:
> >
> > <https://linuxcontainers.org/lxd/getting-started-cli/>
> > <http://insights.ubuntu.com/2016/04/07/lxd-networking-lxdbr0-explained/>
> >
> > But I think the latest advice is as committed with this change to our
> > documentation:
> >
> > <https://github.com/juju/docs/pull/998/files>
> >
> > Note that just running dpkg-reconfigure is not enough, you have to
> > poke a service or run `lxc` afterwards or you get this error with
> > beta4:
> >
> >     $ juju bootstrap --config default-series=xenial lxd-test lxd
> >     ERROR cannot find network interface "lxcbr0": route ip+net: no
> > such network interface
> >     ERROR invalid config: route ip+net: no such network interface
> >
> > That's probably the cause of the other confusion in the updated docs -
> > now we *do* want the bridge named lxdbr0 not lxcbr0. If the user
> > already has lxc setup in some fashion there's a different error from
> > juju telling them to run the dpkg-reconfigure command. That should
> > presumably always appear whenever there's no usable bridge.
> >
>
> Actually, you might get this error with current master too now that I
> think about it. I'll see about testing that and sending a PR which
> fixes it.
>
> > This also presents a challenge for automated testing of the lxd
> > provider in a clean environment, dpkg-reconfigure isn't the nicest
> > thing to use non-interactively, and I can't find clear reference to
> > what the exact required pieces are for the juju provider.
>
> In fact you shouldn't need anything if all you're trying to do is boot
> a container. The dpkg-reconfigure is only required if you want to
> actually have ipv4 connectivity from inside the containers, which you
> may or may not want to do.
>

Most workloads that people deploy, they'll want to connect to them. Many
people, myself included, who have had ipv6 issues on their machines, have
it disabled or do not have an ipv6 compatible connection/nat (unfortunately)

Note that you don't have to do dpkg-reconfigure, you can just edit
> /etc/default/lxd-bridge and restart lxd-bridge && lxd, see
> editLXDBridgeFile in /container/lxd/initialisation.go in the juju tree
> for an example. That may be easier for scripting purposes (you can
> probably just overwrite that file with whatever config you want if
> it's a throwaway VM).
>
> > As part of the juju 2.0 packaging for Ubuntu, we need an
> > autopackagetest that will run in a fresh xenial machine, so this
> > script is what I added to do the lxd configuration:
> >
> > <
> http://bazaar.launchpad.net/~juju-qa/ubuntu/xenial/juju/xenial-2.0-beta4/view/head:/debian/tests/setup-lxd.sh
> >
> >
> > With the additional step afterwards to call `lxc finger` that works
> > (with caveats) for me. In the autopkgtest.ubuntu.com infrastructure
> > however it does not, and it has also failed in two different ways for
> > Steve Langasek and Martin Pitt:
> >
> > "autopkgtest lxd provider tests fail for 2.0"
> > <https://bugs.launchpad.net/ubuntu/+source/juju-core/+bug/1571082>
> >
> > So, at present we don't have confidence that the LXD provider will
> > work, even with the manual configuration step, for users installing
> > Xenial for the first time.
> >
> >
> > When it comes to using lxd in clouds, as I understand it we've settled
> > on retaining the 'lxc' and 'lxd' name distinction in 2.0 - which does
> > mean bundles have to be manually changed at present to start using
> > lxd. Most of the CI bundle testing is using real bundles out of the
> > store, which all still say 'lxc' and therefore don't exercise the lxd
> > container code at all.
> >
> > We do have the container networking test which uses 'juju add-machine
> > ... lxd:0' - and fails due to the networking setup:
> >
> > "container networking lxd 'Missing parent for bridged type nic'"
> > <https://bugs.launchpad.net/juju-core/+bug/1571053>
>
> Looks like this is probably related to the network profiles code that
> was added recently, frobware probably knows more (cc'd). If you test
> on a cloud that /isn't/ network spaces aware (e.g. GCE) using the
> current next branch, it will fall back to using the default network
> config without trying to do anything fancy. This should work better
> for you.
>
> > That is probably less interesting than the default behaviour without
> > the feature flag.
> >
> > As a separate test, I updated one of our simple bundles just to say
> > 'lxd' in two places where it had 'lxc' for a service before. The
> > deployment timed out after 24 minutes, where the normal test with lxc
> > takes 12 minutes.
> >
> > The reason for that turns out to be pretty simple. Looking back at the
> > lxd provider test, it hung for over 20 minutes just updating packages
> > when setting up the first container:
> >
> >     In container /var/log/apt/history.log
> >     Start-Date: 2016-04-15  22:11:16
> >     ...
> >     End-Date: 2016-04-15  22:33:03
> >
> > Unlike other providers, lxd exposes no way to use the daily images
> > instead of release images, so at present any machine using lxd
> > containers with juju for the first time will get the xenial beta2
> > image then upgrade basically every package. This issue goes away next
> > week, but gets in the way of testing before then.
>
> Yes, it does, jam added it. You can use the image-stream=daily option:
>
> juju set-model-config image-stream="daily"
> juju bootstrap --config image-stream=daily ...
>
> The daily streams haven't been regenerated since April 12 (another
> known issue that CPC is working on) so you'll still see some churn.
>
> > In a related note, the lxc container handling in juju manages images
> > on the state server, but from what I see of the lxd code, each
> > deployed machine will fetch images from cloud-images.ubuntu.com and
> > keep a separate set of images. That makes the above problem much worse
> > for any bundle with multiple machines that use containers.
> >
> > Finally, we'll need to update the log gathering code in CI to know how
> > to look inside lxd containers. At the moment, only the machine log
> > seems to be linked into the /var/log/lxd/ directory, so the cloud-init
> > logs and other pieces are currently missing. It does seem we can peek
> > inside using paths like:
> >
> >
> /var/lib/lxd/containers/juju-d9c2c426-f268-47d9-8b96-4468b3f60b51-machine-0/rootfs/var/log/apt/term.log
> >
> > But I'm not sure if that's behaviour we can rely on with all lxd
> configurations.
>
> You'll be able to do this while the container is running, but
> depending on your storage backend, if it is stopped you can't. We
> could use the LXD API to access files in the container regardless of
> whether it is running, though.
>
> Tycho
>
> --
> Juju-dev mailing list
> Juju-dev at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20160419/97592619/attachment.html>