Netplan and high availability

Daniel Axtens daniel.axtens at canonical.com
Wed Oct 17 04:30:34 UTC 2018


Hi,

>From a netplan point of view, it's a bit of a tricky one. (Note that
I'm a netplan contributor but *not* a netplan maintainer or architect,
so take this as unofficial.)

I haven't tested pacemaker specifically, but I can replicate the
disappearance of addresses easily[*].

I think it's systemd-networkd, not netplan, that is mangling the IP address here.

Firstly, 'netplan apply' (considering only the networkd backend):
 - creates systemd-networkd config files (netplan generate, src/networkd.c)
 - stops systemd-networkd (netplan/cli/commands/apply.py::command_apply:62)
 - applies any changes in .link files via udev (:87)
 - applies some changes to DOWN devices - which will not be relevant here (:95)
 - starts networkd (:109)

So netplan isn't knowingly doing anything to the IP address.

networkd, on the other hand, sees the address, notes that it is not in
the configuration, and removes it. This is known behaviour from
networkd [1], and appears to be intentional. Quoting from commit
5e5b137acc70 [2]:

  Before we would never remove any state, only add to it, we now drop unwanted
  state from any link the moment we start managing it.

  Note however, that we still will not remove any foreign state added at runtime,
  to avoid any feedback loops. However, we make no guarantees about coexisting
  with third-party tools that change the state of the links we manage.

Because netplan restarts systemd, the VIP address is there when
systemd starts managing the link, and it is not in the config, so it's
removed. When it's later re-added, it then counts as "added at
runtime" and so isn't re-removed.

I'm not sure if there is a networkd option that changes this
behaviour. Even if there was, we would need to make it opt-in rather
than default - most people, most of the time expect that after running
netplan apply, the config they provide to netplan represents the
totality of the state of the configured interfaces, and we regularly
get bug reports where we violate this expectation.

One way of dealing with the systemd behaviour (in general) would be to
have a networkd-dispatcher script[3] that notices when the interface
is configured and pokes any service that expect to provide additional
configuration on top of networkd. In this case, the backport sounds
like a good idea.

In short, as usual, blame systemd. :P

Regards,
Daniel

[1] https://unix.stackexchange.com/questions/348766/ip-address-removed-from-network-interface-managed-by-systemd-networkd
[2] https://github.com/systemd/systemd/commit/5e5b137acc7094c9a1b8e7634b426dd445688bf0
[3] https://netplan.io/faq#use-pre-up-post-up-etc-hook-scripts
[*] ubuntu at netplan2:~$ ip a show dev ens12
7: ens12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel
state UP group default qlen 1000
    link/ether 52:54:00:aa:3d:c3 brd ff:ff:ff:ff:ff:ff
    inet 75.191.156.72/30 brd 75.191.156.75 scope global ens12
       valid_lft forever preferred_lft forever
ubuntu at netplan2:~$ sudo ip a add dev ens12 123.45.67.89/24
ubuntu at netplan2:~$ ip a show dev ens12
7: ens12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel
state UP group default qlen 1000
    link/ether 52:54:00:aa:3d:c3 brd ff:ff:ff:ff:ff:ff
    inet 75.191.156.72/30 brd 75.191.156.75 scope global ens12
       valid_lft forever preferred_lft forever
    inet 123.45.67.89/24 scope global ens12
       valid_lft forever preferred_lft forever
ubuntu at netplan2:~$ sudo netplan apply
ubuntu at netplan2ip a show dev ens12
7: ens12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel
state UP group default qlen 1000
    link/ether 52:54:00:aa:3d:c3 brd ff:ff:ff:ff:ff:ff
    inet 75.191.156.72/30 brd 75.191.156.75 scope global ens12
       valid_lft forever preferred_lft forever



More information about the ubuntu-server mailing list