[Bug 1522130] Re: Openstack API Charms have a naive haproxy connection timeout

[Liam Young]

** Changed in: glance (Juju Charms Collection)
       Status: New => Invalid

** Changed in: nova-cloud-controller (Juju Charms Collection)
       Status: New => Invalid

** Changed in: cinder (Juju Charms Collection)
       Status: New => In Progress

** Changed in: keystone (Juju Charms Collection)
       Status: New => In Progress

** Changed in: neutron-api (Juju Charms Collection)
       Status: New => In Progress

** Changed in: cinder (Juju Charms Collection)
   Importance: Undecided => High

** Changed in: keystone (Juju Charms Collection)
   Importance: Undecided => High

** Changed in: neutron-api (Juju Charms Collection)
   Importance: Undecided => High

** Changed in: cinder (Juju Charms Collection)
     Assignee: (unassigned) => Liam Young (gnuoy)

** Changed in: keystone (Juju Charms Collection)
     Assignee: (unassigned) => Liam Young (gnuoy)

** Changed in: neutron-api (Juju Charms Collection)
     Assignee: (unassigned) => Liam Young (gnuoy)

** Changed in: cinder (Juju Charms Collection)
    Milestone: None => 16.01

** Changed in: keystone (Juju Charms Collection)
    Milestone: None => 16.01

** Changed in: neutron-api (Juju Charms Collection)
    Milestone: None => 16.01

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Juju Charms Collection.
Matching subscriptions: charm-bugs
https://bugs.launchpad.net/bugs/1522130

Title:
  Openstack API Charms have a naive haproxy connection timeout

Status in cinder package in Juju Charms Collection:
  In Progress
Status in glance package in Juju Charms Collection:
  Invalid
Status in keystone package in Juju Charms Collection:
  In Progress
Status in neutron-api package in Juju Charms Collection:
  In Progress
Status in nova-cloud-controller package in Juju Charms Collection:
  Invalid

Bug description:
  On OpenStack API charms haproxy sets timeout connection to 1 second.
  This is reasonable only in an ideal scenario. Connections to backends
  on a busy cloud may take much longer. Even haproxy docs suggest 4 to 5
  seconds [1].

  defaults
      log global
      mode tcp
      option tcplog
      option dontlognull
      retries 3
      timeout queue 1000
      timeout connect 1000

  This causes mysterious errors like the following:

  On neutron-api:
  ConnectionRefused: Unable to establish connection to http://10.245.161.159:9696/v2.0/security-groups.json

  On keystone:
  ERROR: HTTPConnectionPool(host='10.5.51.133', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'httplib.BadStatusLine'>: '')

  On nova-cloud-controller:
  requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.245.161.158', port=8774): Max retries exceeded with url: /v2/ba51369fa6444c6d9c2be7264675741f/servers/373ed532-9819-4f2b-b32a-4d27147d0338/os-interface (Caused by <class 'httplib.BadStatusLine'>: '')

  When max_retries is forced > 1 it becomes only slightly less opaque:

  urllib3.connectionpool - WARNING - Retrying (5 attempts remain) after
  connection broken by 'BadStatusLine("''",)':
  /v2/ba51369fa6444c6d9c2be7264675741f/servers/32eed1a0-6c9d-497f-b9fb-
  aac0fa748858/os-interface

  Set timeout connect (and timeout queue while we are at it) to sane
  defaults and optionally make this configurable.

  [1] https://cbonte.github.io/haproxy-
  dconv/configuration-1.5.html#4-timeout%20connect

To manage notifications about this bug go to:
https://bugs.launchpad.net/charms/+source/cinder/+bug/1522130/+subscriptions