[Bug 2095432] Re: Ping health-monitor does not work
Rodrigo Barbieri
2095432 at bugs.launchpad.net
Wed Jan 22 20:03:40 UTC 2025
** Also affects: octavia
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to octavia in Ubuntu.
https://bugs.launchpad.net/bugs/2095432
Title:
Ping health-monitor does not work
Status in octavia:
New
Status in snap-octavia-diskimage-retrofit:
New
Status in octavia package in Ubuntu:
New
Bug description:
Creating a ping health monitor always results in ERROR state. This
happens because the haproxy service running in the amphora image does
not have enough permissions to run the file /var/lib/octavia/ping-
wrapper.sh.
In Octavia Logs:
2025-01-17 17:47:06.714 40655 DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] member 8c4c9a83-96ff-4533-867b-6b451acdc255 status has changed from OFFLINE to ERROR, updating db. _update_status /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:229
2025-01-17 17:47:06.719 40655 DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] pool 00d6affb-3e1e-46df-9162-7b47c21da98f status has changed from ONLINE to ERROR, updating db. _update_status /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:229
2025-01-17 17:47:06.724 40655 DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] loadbalancer 8f720506-28fc-4589-a33b-de1cba13c93f status has changed from ONLINE to ERROR, updating db. _update_status /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:229
In the amphora:
Jan 21 16:53:14 amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145 ip[19071]: [ALERT] (19071) : Failed to exec process for external health check: Permission denied. Aborting.
Jan 21 16:53:19 amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145 ip[19073]: [ALERT] (19073) : Failed to exec process for external health check: Permission denied. Aborting.
Looking at the permissions for /var/lib/octavia:
root at amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145:/var/lib# ls -lha octavia
total 36K
drwxr-x--- 4 octavia octavia 4.0K Jan 21 15:49 .
drwxr-xr-x 43 root root 4.0K Jan 20 19:42 ..
drwxr-xr-x 3 root root 4.0K Jan 20 19:42 certs
drwxr-xr-x 2 root root 4.0K Jan 21 16:52 d0a9cf91-1c0c-499d-a86c-3553099fa43d
srw-rw-rw- 1 root root 0 Jan 20 19:43 d0a9cf91-1c0c-499d-a86c-3553099fa43d.sock
-rw-r--r-- 1 root root 25 Jan 20 19:11 haproxy-default-user-group.conf
-rwxr-xr-x 1 root root 207 Jan 20 20:05 ping-wrapper.sh
-rw-r--r-- 1 root root 23 Jan 20 19:42 plugged_interfaces
-rw-r--r-- 1 root root 23 Jan 20 19:42 plugged_interfaces.sorted
-rw-r----- 1 root root 85 Jan 21 16:54 stats_counters.json
root at amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145:/var/lib#
The haproxy runs in a separate namespace as the nobody:nogroup config:
root at amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145:/var/lib# grep user octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d/haproxy.cfg
user nobody
stats socket /var/lib/octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d.sock mode 0666 level user
root at amphora-ae26c49f-53c8-4f74-ac29-74046f6a5145:/var/lib# grep group octavia/haproxy-default-user-group.conf
group nogroup
Service config:
[Service]
# Force context as we start haproxy under "ip netns exec"
SELinuxContext=system_u:system_r:haproxy_t:s0
Environment="CONFIG=/var/lib/octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d/haproxy.cfg"
"USERCONFIG=/var/lib/octavia/haproxy-default-user-group.conf"
"PIDFILE=/var/lib/octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d/d0a9cf91-1c0c-499d-a86c-3553099fa43d.pid"
ExecStartPre=/usr/sbin/haproxy -f $CONFIG -f $USERCONFIG -c -q -L
PhDoY7WOeBqhwysZ0Dzu6CxNZgU
ExecReload=/usr/sbin/haproxy -c -f $CONFIG -f $USERCONFIG -L
PhDoY7WOeBqhwysZ0Dzu6CxNZgU
process:
nobody 1728 0.0 1.3 113372 13716 ? S Jan20 0:10
/usr/sbin/haproxy -sf 1651 -Ws -f
/var/lib/octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d/haproxy.cfg -f
/var/lib/octavia/haproxy-default-user-group.conf -p
/var/lib/octavia/d0a9cf91-1c0c-499d-a86c-3553099fa43d/d0a9cf91-1c0c-499d-a86c-3553099fa43d.pid
-L PhDoY7WOeBqhwysZ0Dzu6CxNZgU
Given that the haproxy user configuration is set in the octavia source
files, some possible solutions to run the ping-wrapper.sh are:
1) Change the /var/lib/octavia folder permissions to 755 instead of
750. This will allow the haproxy service to run the ping-wrapper.sh
file which already has a 755 permission. This however exposes all the
folder contents. This can be done either during the octavia-diskimage-
retrofit build process or in the octavia package sources files.
2) Change the /var/lib/octavia ownership to nobody:nogroup. This
allows access to the folder but also gives permission to create files
in the folder. This can be done either during the octavia-diskimage-
retrofit build process or in the octavia package sources files.
3) Move ping-wrapper.sh out of the /var/lib/octavia folder, to a
public place. This will also require changing the octavia source
haproxy template files which have the location of the file hardcoded
to /var/lib/octavia.
To manage notifications about this bug go to:
https://bugs.launchpad.net/octavia/+bug/2095432/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list