[Bug 1620897] Re: Fix race on getting close notifier channel

[Xiang Hui]

** Description changed:

  [OS]
  Ubuntu Xenial
  etcd 2.2.5
  
  [Error Log]
  Sep 2 16:08:16 ubuntu etcd[18180]: panic: net/http: CloseNotify called after ServeHTTP finished
  Sep 2 16:08:16 ubuntu etcd[18180]: goroutine 421 [running]:
  Sep 2 16:08:16 ubuntu etcd[18180]: net/http.(*response).CloseNotify(0xc8202f64e0, 0x0)
  Sep 2 16:08:16 ubuntu etcd[18180]: #011/usr/lib/go/src/net/http/server.go:1535 +0x9d
  Sep 2 16:08:16 ubuntu etcd[18180]: github.com/coreos/etcd/proxy.(*reverseProxy).ServeHTTP.func1(0x7fb0024d1a80, 0xc8202f64e0, 0xc8203e4f78, 0xc8202e8a80, 0xc8203e4fa0, 0xc820464f50)
  Sep 2 16:08:16 ubuntu etcd[18180]: #011/build/etcd-tG_CNV/etcd-2.2.5+dfsg/obj-x86_64-linux-gnu/src/github.com/coreos/etcd/proxy/reverse.go:107 +0x39
  Sep 2 16:08:16 ubuntu etcd[18180]: created by github.com/coreos/etcd/proxy.(*reverseProxy).ServeHTTP
  Sep 2 16:08:16 ubuntu etcd[18180]: #011/build/etcd-tG_CNV/etcd-2.2.5+dfsg/obj-x86_64-linux-gnu/src/github.com/coreos/etcd/proxy/reverse.go:113 +0x691
  Sep 2 16:08:16 ubuntu systemd[1]: etcd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
  Sep 2 16:08:16 ubuntu systemd[1]: etcd.service: Unit entered failed state.
  Sep 2 16:08:16 ubuntu systemd[1]: etcd.service: Failed with result 'exit-code'.
  
  etcd proxy process dies and is not restarted by systemd, Components that
  depend on etcd report connection errors like this:
  
  - Request to server http://127.0.0.1:4001 failed:
  MaxRetryError("HTTPConnectionPool(host='127.0.0.1', port=4001): Max
  retries exceeded with url:
  /v2/keys/calico/dhcp/v1/subnet?waitIndex=1625&recursive=true&wait=true
  (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object
  at 0x7f6f0c5cae10>: Failed to establish a new connection: [Errno 111]
  ECONNREFUSED',))",)
  
  [Root Cause]
  etcd proxy panic occur in etcd 2.2.5, which is fixed in etcd 2.3.4 (https://github.com/coreos/etcd/pull/5269/files#r62072134), currently only have 2.2.5 in xenial, we need it backported.
  
  -------------------------------------------------------
  
  [Impact]
  This patch fix race on getting close notifier channel when a panic reported as 'net/http: CloseNotify called after ServeHTTP finished'.
  
  [Test Case]
+ No special configuration, running etcd in proxy mode.
  
  [Regression Potential]
+ etcd no longer crashes every so often while in proxy mode...

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1620897

Title:
  Fix race on getting close notifier channel

Status in etcd package in Ubuntu:
  Confirmed

Bug description:
  [OS]
  Ubuntu Xenial
  etcd 2.2.5

  [Error Log]
  Sep 2 16:08:16 ubuntu etcd[18180]: panic: net/http: CloseNotify called after ServeHTTP finished
  Sep 2 16:08:16 ubuntu etcd[18180]: goroutine 421 [running]:
  Sep 2 16:08:16 ubuntu etcd[18180]: net/http.(*response).CloseNotify(0xc8202f64e0, 0x0)
  Sep 2 16:08:16 ubuntu etcd[18180]: #011/usr/lib/go/src/net/http/server.go:1535 +0x9d
  Sep 2 16:08:16 ubuntu etcd[18180]: github.com/coreos/etcd/proxy.(*reverseProxy).ServeHTTP.func1(0x7fb0024d1a80, 0xc8202f64e0, 0xc8203e4f78, 0xc8202e8a80, 0xc8203e4fa0, 0xc820464f50)
  Sep 2 16:08:16 ubuntu etcd[18180]: #011/build/etcd-tG_CNV/etcd-2.2.5+dfsg/obj-x86_64-linux-gnu/src/github.com/coreos/etcd/proxy/reverse.go:107 +0x39
  Sep 2 16:08:16 ubuntu etcd[18180]: created by github.com/coreos/etcd/proxy.(*reverseProxy).ServeHTTP
  Sep 2 16:08:16 ubuntu etcd[18180]: #011/build/etcd-tG_CNV/etcd-2.2.5+dfsg/obj-x86_64-linux-gnu/src/github.com/coreos/etcd/proxy/reverse.go:113 +0x691
  Sep 2 16:08:16 ubuntu systemd[1]: etcd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
  Sep 2 16:08:16 ubuntu systemd[1]: etcd.service: Unit entered failed state.
  Sep 2 16:08:16 ubuntu systemd[1]: etcd.service: Failed with result 'exit-code'.

  etcd proxy process dies and is not restarted by systemd, Components
  that depend on etcd report connection errors like this:

  - Request to server http://127.0.0.1:4001 failed:
  MaxRetryError("HTTPConnectionPool(host='127.0.0.1', port=4001): Max
  retries exceeded with url:
  /v2/keys/calico/dhcp/v1/subnet?waitIndex=1625&recursive=true&wait=true
  (Caused by NewConnectionError('<urllib3.connection.HTTPConnection
  object at 0x7f6f0c5cae10>: Failed to establish a new connection:
  [Errno 111] ECONNREFUSED',))",)

  [Root Cause]
  etcd proxy panic occur in etcd 2.2.5, which is fixed in etcd 2.3.4 (https://github.com/coreos/etcd/pull/5269/files#r62072134), currently only have 2.2.5 in xenial, we need it backported.

  -------------------------------------------------------

  [Impact]
  This patch fix race on getting close notifier channel when a panic reported as 'net/http: CloseNotify called after ServeHTTP finished'.

  [Test Case]
  No special configuration, running etcd in proxy mode.

  [Regression Potential]
  etcd no longer crashes every so often while in proxy mode...

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/etcd/+bug/1620897/+subscriptions