[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
Nobuto Murata
nobuto at nobuto-murata.org
Sat May 12 19:32:21 UTC 2018
I may be completely wrong, but one possible reason to cause 503 from
haproxy is AppArmor.
@Xav, what happens if you disable apparmor, i.e. aa-disable /usr/bin
/neutron-lbaasv2-agent?
As you see in an unrelated bug[1], the apparmor profile installed by
neutron-gateway charm blocks lbaasv2 if it's set in enforced mode.
[kernel log]
Sep 21 19:46:44 HOSTNAME kernel: audit: type=1400 audit(1506023204.857:304): apparmor="DENIED" operation="connect" info="Failed name lookup - disconnected path" error=-13 profile="/usr/bin/neutron-lbaasv2-agent" name="var/lib/neutron/lbaas/v2/496d6d2b-8bf7-42b7-822f-c3f31d8db43f/haproxy_stats.sock" pid=736613 comm="neutron-lbaasv2" requested_mask="wr" denied_mask="wr" fsuid=115 ouid=0
[/var/log/neutron/neutron-lbaasv2-agent.log]
2017-09-21 19:44:44.850 736613 WARNING neutron_lbaas.drivers.haproxy.namespace_driver [-] Error while connecting to stats socket: [Errno 13] EACCES
In complain mode, if you see "ALLOWED" message for operation="connect" and info="Failed name lookup - disconnected path", but still see EACCES from lbaasv2 log. It may be hit by a bug in apparmor which blocks operation="connect" even in complain mode[2][3].
[1] https://bugs.launchpad.net/charm-neutron-gateway/+bug/1718768
[2] https://bugs.launchpad.net/apparmor/+bug/1624497
[3] https://bugs.launchpad.net/apparmor/+bug/1624300
** Changed in: charm-neutron-gateway
Status: Invalid => Incomplete
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron-lbaas in Ubuntu.
https://bugs.launchpad.net/bugs/1770040
Title:
lbaas load balancer does not forward traffic unless agent restarted
Status in OpenStack neutron-gateway charm:
Incomplete
Status in neutron-lbaas package in Ubuntu:
Incomplete
Bug description:
Juju 3.2.7, current 18.02 charms. Deployed an OpenStack cloud
including Neutron and LBaaS.
Made a fresh set of 3 instances, assigned floating IPS to all 3. Made
a security group to allow port 80 in.
Made a fresh load balancer, listener, tcp based healthcheck. Used nc
on all 3 instances to listen on port 80.
Connected to the load balancer floating IP on port 80, immediately
discoonects me - there's no backends listening.
Initial haproxy status:
root at neut001:~# echo 'show stat;show table' | socat stdio /var/lib/neutron/lbaas/v2/b100e451-f3ba-46f3-bde4-274a6d15ae6d/haproxy_stats.sock
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,
171808be-5e02-4bb9-8dbd-e559d541f473,FRONTEND,,,0,1,2000,1,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,1,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,
(no backends)
Restarted neutron-lbaasv2-agent
root at neut001:~# echo 'show stat;show table' | socat stdio /var/lib/neutron/lbaas/v2/b100e451-f3ba-46f3-bde4-274a6d15ae6d/haproxy_stats.sock
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,
171808be-5e02-4bb9-8dbd-e559d541f473,FRONTEND,,,0,0,2000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,0,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,
7ef8ab16-8fec-4ed5-9883-2613d4295476,632625a8-50cf-40e2-9c18-bdfdf79cac3c,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,13,0,,1,3,1,,0,,2,0,,0,L4OK,,0,,,,,,,0,,,,0,0,,,,,-1,,,0,0,0,0,
7ef8ab16-8fec-4ed5-9883-2613d4295476,16584f9d-e035-4550-9094-b26078e1fc87,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,13,0,,1,3,2,,0,,2,0,,0,L4OK,,0,,,,,,,0,,,,0,0,,,,,-1,,,0,0,0,0,
7ef8ab16-8fec-4ed5-9883-2613d4295476,d39e8de3-95a6-417d-81eb-4fca9079e57e,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,13,0,,1,3,3,,0,,2,0,,0,L4OK,,0,,,,,,,0,,,,0,0,,,,,-1,,,0,0,0,0,
7ef8ab16-8fec-4ed5-9883-2613d4295476,BACKEND,0,0,0,0,200,0,0,0,0,0,,0,0,0,0,UP,3,3,0,,0,13,0,,1,3,0,,0,,1,0,,0,,,,,,,,,,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,
(all 3 backends)
After restarting the service, all the traffic passes perfectly. The
only thing I did was restart the service - I suspect there's some kind
of race condition where we need to restart the services after the
config is changed.
To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list