NetworkManager reports disconnect/reconnect many times a day

[Paul Smith]

I am seeing a behavior of NetworkManager where it reports that it is
disconnecting for a few seconds then reconnecting, multiple times a day
(once every few hours).  However, it's not ACTUALLY disconnected.

I'm using Ubuntu 20.04.

I have a WIRED connection; this is not wifi.

My little LAN icon on the control panel goes to a "?" then comes back,
and my Spotify app goes dim then comes back.

I do a lot of remote work, including a VPN and SSH to remote systems,
and at no time do I notice any lag in typing etc. even when the above
happens.  So, in some ways this is not a real problem (and in fact I've
lived with it for quite a while).

But, a bunch of our wireless devices DO lose connection multiple times
a day although I can't say if it's the same times or not.  I don't know
if this is related: it obviously could be completely different things
but I thought I'd start with the simpler-to-debug problem.

I enabled DEBUG logging for NetworkManager and whenever this happens I
get output like this in journalctl:

Dec 05 05:55:01 NetworkManager[968]: <debug> [1607165701.8724] connectivity: (enx106530e29458,IPv4,38386) start request to 'http://connectivity-check.ubuntu.com/' (try resolving 'connectivity-check.ubuntu.com' using systemd-resolved)
Dec 05 05:55:21 NetworkManager[968]: <debug> [1607165721.7958] connectivity: (enx106530e29458,IPv4,38386) check completed: LIMITED; timeout
Dec 05 05:55:21 NetworkManager[968]: <debug> [1607165721.7959] device[7d9e8e19a7fc97e0] (enx106530e29458): connectivity state changed from FULL to LIMITED
Dec 05 05:55:21 NetworkManager[968]: <debug> [1607165721.7963] manager: connectivity checking indicates LIMITED
Dec 05 05:55:21 NetworkManager[968]: <info>  [1607165721.7964] manager: NetworkManager state is now CONNECTED_SITE
Dec 05 05:55:21 NetworkManager[968]: <debug> [1607165721.7974] dispatcher: (760) dispatching action 'connectivity-change'
Dec 05 05:55:21 NetworkManager[968]: <debug> [1607165721.7980] device[7d9e8e19a7fc97e0] (enx106530e29458): ip4-config: update (commit=1, new-config=0x55e94d2649c0)

then there's a bunch of debug about state change then I see:

Dec 05 05:55:21 NetworkManager[968]: <debug> [1607165721.8005] dns-mgr: (device_ip_config_changed): no DNS changes to commit (0)
Dec 05 05:55:21 NetworkManager[968]: <debug> [1607165721.8006] connectivity: (enx106530e29458,IPv4,38393) start request to 'http://connectivity-check.ubuntu.com/' (try resolving 'connectivity-check.ubuntu.com' using systemd-resolved)
Dec 05 05:55:21 NetworkManager[968]: <debug> [1607165721.8007] device[7d9e8e19a7fc97e0] (enx106530e29458): ip4-config: update (commit=0, new-config=0x55e94d2649c0)
Dec 05 05:55:21 NetworkManager[968]: <debug> [1607165721.8294] dispatcher: (760) /etc/NetworkManager/dispatcher.d/01-ifupdown succeeded
Dec 05 05:55:21 NetworkManager[968]: <debug> [1607165721.8294] dispatcher: (760) /etc/NetworkManager/dispatcher.d/99-disable-wireless-when-wired succeeded
Dec 05 05:55:22 NetworkManager[968]: <debug> [1607165722.7972] connectivity: (enx106530e29458,IPv4,38394) start request to 'http://connectivity-check.ubuntu.com/' (try resolving 'connectivity-check.ubuntu.com' using systemd-resolved)
Dec 05 05:55:23 NetworkManager[968]: <debug> [1607165723.1788] connectivity: (enx106530e29458,IPv4,38394) check completed: FULL; status header found
Dec 05 05:55:23 NetworkManager[968]: <debug> [1607165723.1789] device[7d9e8e19a7fc97e0] (enx106530e29458): connectivity state changed from LIMITED to FULL
Dec 05 05:55:23 NetworkManager[968]: <debug> [1607165723.1791] manager: connectivity checking indicates FULL
Dec 05 05:55:23 NetworkManager[968]: <info>  [1607165723.1791] manager: NetworkManager state is now CONNECTED_GLOBAL
Dec 05 05:55:23 NetworkManager[968]: <debug> [1607165723.1796] dispatcher: (761) dispatching action 'connectivity-change'

and my icon shows connected again.

It looks like checks to connectivity-check.ubuntu.com sometimes fail.
 I don't know if that's because my network is not allowing new
connections or something weird like that (as I said I haven't noticed
any hangs while typing in remote sessions), or if there's a DNS issue,
or if this Ubuntu server is too busy and doesn't always respond, or
what it is.

Has anyone else noticed this?  If you run:

  journalct --unit NetworkManager --since yesterday

do you see the CONNECTED_SITE / CONNECTED_GLOBAL messages (these are
the only ones printed at "INFO" level which is the default log level)
appear randomly / not when you expect your network connection to go
down?  If lots of people are seeing it then it's most likely an Ubuntu
server problem.

Does anyone know if there's a way to get more details from
NetworkManager about exactly what kind of failure it detected?  Does
"timedout" mean a 408 timeout?  A 504 timeout?  Something else?