Cloud instance with reduced MTU, container uses 1500, connections stall

[Serge Hallyn]

Quoting Andreas Hasenack (andreas at canonical.com):
> > > Bridge:
> > > # brctl show lxcbr0
> > > bridge name    bridge id        STP enabled    interfaces
> > > lxcbr0        8000.feee96ca7280    no        veth13ME9N
> > >
> > > I'm not sure how different MTUs are handled on a bridge.
> >
> > Right so then perhaps /etc/init/lxc-net.conf should be setting the
> > mtu on lxcbr0 to match eth0's.  Of course eth0's mtu could be changed
> > later and we can't catch that.
> >
> > Can you check whether dropping lxcbr0's mtu fixes your problem by
> > itself?
> >
> 
> 
> Interesting. The moment I start the container, the lxcbr0 MTU is back to
> 1500. I guess creating the veth device with a MTU of 1500 and adding it to
> the lxcbr0 bridge triggers that:
> (...)
> 3: lxcbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1454 qdisc noqueue state
> DOWN group default
>     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
>     inet 10.0.3.1/24 brd 10.0.3.255 scope global lxcbr0
>        valid_lft forever preferred_lft forever
>     inet6 fe80::38b8:13ff:feac:a6b0/64 scope link
>        valid_lft forever preferred_lft forever
> 
> root at juju-fakestack-machine-3:~# lxc-start -n juju-trusty-template -d
> 
> root at juju-fakestack-machine-3:~# ip addr
> (...)
> 3: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
> UP group default
>     link/ether fe:22:eb:2c:3a:78 brd ff:ff:ff:ff:ff:ff
>     inet 10.0.3.1/24 brd 10.0.3.255 scope global lxcbr0
>        valid_lft forever preferred_lft forever
>     inet6 fe80::38b8:13ff:feac:a6b0/64 scope link
>        valid_lft forever preferred_lft forever
> 11: vethNH178P: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> master lxcbr0 state UP group default qlen 1000
>     link/ether fe:22:eb:2c:3a:78 brd ff:ff:ff:ff:ff:ff
>     inet6 fe80::fc22:ebff:fe2c:3a78/64 scope link
>        valid_lft forever preferred_lft forever
> 
> 
> But ok, I put it back at 1454 and login on the container, which has 1500,
> and things don't work. What matters is the MTU of the container interface
> I'm afraid.

Right, the problem is that if lxc was going to try to autoset the veth
mtu then it should set it to equal the mtu of its bridge, lxcbr0.  So
if we don't set lxcbr0 to match the host NIC then there's nothing lxc
could realistically do anyway.

Anyway this still seems ok to me - if we update /etc/init/lxc-net.conf
to have lxcbr0 mtu match the host nic's mtu, and then have lxc drop
the veth mtu (if not specified in configuration file) to match
the veth link's (lxcbr0, br0, whatever is specified in the config).

I'm open to other suggestions.

-serge