Plans for HTTP2 in Xenial and future SRUs

Tue Jan 5 21:29:39 UTC 2016

Question for the release team: is it acceptable for us to MIR nghttp2
in xenial-updates after release?

This question follows a quick ad-hoc meeting between mdeslaur, sarnold,
teward and I[1], and applies to apache2 and nginx equally (both are in
main).

Upstream support for HTTP2 is available to build in the archive, but
currently we disable the build in both as support is considered
experimental upstream. We (security team and server team) don't want to
be pinned down into supporting these for five years if it is subject to
change in this way, as this could be unreasonably difficult to do.

However, given our release schedule, we expect that we will want to add
HTTP2 support to Xenial after Xenial's release, following the new SRU
policy.

For nginx this is just a tweak of configure flags we could do in an SRU.

For apache2 we will additionally need to MIR nghttp2. Can I confirm that
this will be OK to do post-release?

Thanks,

Robie

[1] #ubuntu-hardened.log, attached.
-------------- next part --------------
20:49 <mdeslaur> let me start with a bit of history...
20:49 <mdeslaur> When I merged apache into xenial, the new version had http2 enabled in debian
20:49 <mdeslaur> but it's marked as "experimental" by apache upstream, and it requires an external library libnghttp2
20:49 <mdeslaur> which is in universe and would need a security audit and a MIR
20:50 <mdeslaur> so I disabled http2 support, as I wasn't sure we wanted to ship something marked "experimental" in an lts release
20:50 <mdeslaur> it was more for support reasons than for security reasons, though all the new code certainly is scary
20:51 <rbasak> For reference, where does upstream declare it experimental please, so that we can see when that tag gets removed?
20:51 <mdeslaur> that being said, I do think we need to support http2 in the LTS release. So there are two options 1- enable the "experimental" code, 2- backport http2 support once upstream declares it stable
20:51 <mdeslaur> rbasak: for ref: https://httpd.apache.org/docs/2.4/mod/mod_http2.html
20:52 <mdeslaur> "This module is experimental. Its behaviors, directives, and defaults are subject to more change from release to release relative to other standard modules."
20:52 <mdeslaur> at the time I did the merge, I was unaware that the http/2 standard was pretty much done
20:52 <rbasak> Great, thanks.
20:52 <mdeslaur> I thought it was experimental also
20:52 <mdeslaur> but browsers actually support it now
20:53 <mdeslaur> from a security point of vue, it's a bit scary, and libnghttp2 needs a security audit
20:54 <mdeslaur> but if we intend on shipping it anyway somewhere down the line, perhaps we should do the work now
20:54 <mdeslaur> but someone has to commit to supporting it and doing the mir paperwork
20:54 <rbasak> OK
20:54 <mdeslaur> now, for nginx, http2 support was disabled too, but it apparently doesn't use that library
20:55 <mdeslaur> it uses it's own brand new code
20:55 <teward> correct, however the module is also listed as experimental there, as well
20:55 <mdeslaur> hrm, ok
20:55 <rbasak> So libnghttp2 needs an MIR and so you will be able to consider it more carefully doing a security review for the MIR.
20:55 <teward> mdeslaur: http://nginx.org/en/docs/http/ngx_http_v2_module.html
20:55 <teward> to quote:
20:55 <teward> Known Issues ||  The module is experimental, caveat emptor applies.
20:55 <rbasak> What about apache2's mod_http2? Do you want to do a review on that before it is enabled, or from a security perspective are you happy for us to enable it?
20:56 <mdeslaur> I'll let sarnold comment on that
20:56 <mdeslaur> procedure wise, in theory, we don't need to audit it
20:57 <mdeslaur> my fear is that the protocol will get changed, or the code will be refactored in some big way, and then we'll have to backport basically a whole new module
20:57 <mdeslaur> but...we'll be doing that whether we enable it now or later, as the LTS release will eventually require http/2
20:58 <mdeslaur> now, I know sarnold has concerns also
20:58 <mdeslaur> I'll let him summarize them
20:58 <sarnold> normally we'd be happy to say that if it comes from the same upstream in the same package then it wouldn't need a review, but http/2 is scary enough that a quick once-over first seems appropriate.
20:59 <sarnold> the kind of code that http/2 represents is almost impossible to code correctly in C; there's just too many edge cases of packets coming in a few bytes at a time, or declaring that they provide overlapping boundaries, etc
20:59 -!- hggdh [~hggdh at ubuntu/member/hggdh] has quit [Remote host closed the connection]
21:00 <sarnold> and since it's a protocol rather than e.g. simple file olperations, fuzzers are insanely primitive here; it's up the ingenuity of fuzzer authors to figureout what might break these things.
21:01 -!- hggdh has joined #ubuntu-hardened
21:01 <sarnold> I just didn't want us shipping code immediately that had never seen production use anywhere else first; there's bound to be something scary found in these, and I suspect the rate at which issues are found will drop off over time
21:02 <sarnold> the timing with the nginx stable release is a bit unfortunate; I'd be less worried if their stable release came even a month or two before ours
21:02 <rbasak> FWIW, I just asked in #httpd. Experimental tag removal is "Unlikely for April".
21:03 <mdeslaur> sarnold: I assume http/2 would be disabled by default. Knowing we will need to support it as some point in time, would adding a "experimental, not supported" warning to the readme be sufficient?
21:03 <rbasak> As it's a module, I wonder about the feability of actually shipping it universe. That might not be worth the effort though.
21:04 <sarnold> mdeslaur: maybe. I don't really like the idea of "this little file over here said it might not be safe" ..
21:04 <sarnold> rbasak: at least for nginx there's already some modules that live in a universe binary package
21:05 <rbasak> Right, so it's easier for nginx to arrange in term of packaging, since we (teward really) do it already.
21:05 <mdeslaur> as I said, this is certainly something that we're going to have to turn on in 6m-1y anyway
21:05 <sarnold> but I think http/2 ought to be supported in an LTS, and moving it from one binary package to another in an update in the future seems like a lot of trouble and a lot to ask of teward.
21:05  * teward points at two configure rules flags to enable it
21:06 <teward> really not hard to enable or disable it in nginx, either way, since it's shipped in the tarballs
21:06 <mdeslaur> so perhaps we should leave it disabled until upstreams declare it stable, and then look at it again?
21:06  * rbasak puts "messaging" on the agenda to discuss later.
21:06 <mdeslaur> it's basically just a build option, so it's trivial for someone to stick it in a ppa
21:07 <rbasak> mdeslaur: for both apache2 and nginx?
21:07 <teward> mdeslaur: which we already do in nginx, the mainline ppa keeps the Debian build rules which include HTTP/2
21:07 <teward> rbasak: ^
21:07 <mdeslaur> rbasak: if someone wants to have their own version with the experimental code enabled
21:07 <rbasak> Should we MIR nghttp2 now maybe, and even put it into main in advance?
21:07 <mdeslaur> rbasak: not necessarily us
21:08 <mdeslaur> sarnold: opinion on nghttp2 ^
21:08 <mdeslaur> so it would be there for the day we do decide to turn it on in apache...
21:08 <sarnold> mdeslaur: hrmm, that probably would be easiest. well "easiest" :)
21:08 <rbasak> However if it is main now you will still need to attend CVEs for it.
21:09 <rbasak> So that's more work for you.
21:09 <mdeslaur> and with no way to test it :)
21:09 <sarnold> we've got two dozen new go package to mir before FF too..
21:09 <sarnold> hah
21:09 <rbasak> OTOH if a future nghttp2 ends up in main, you'll have to at least do a one-off update which could be equally painful.
21:09 <rbasak> That might be easier though, on second thoughts.
21:09 <sarnold> moving nghttp2 from universe to main in an lts also sounds painful.
21:10 <sarnold> having it in main before the release is doubtrless going to make flipping a switch inthe apache build easier when the time comes
21:10 <rbasak> As long as the release team and SRU team are OK with it, I don't think it's too bad to move it to main later.
21:10 <rbasak> We'll need to check with them first, but I think it should be OK.
21:11 <mdeslaur> I have to go, but I think we're now all on the same page with regards to the issues
21:11 <rbasak> OK
21:11 <mdeslaur> personally, I think the server team should make the call...sarnold?
21:11 <sarnold> mdeslaur: that's fine by me.
21:11 <rbasak> I'd like this release noted whatever happens. So let me summarise what I've understood.
21:12 <rbasak> 1. No HTTP2 support in nginx nor apache2 because both upstreams still consider it experimental.
21:12 <sarnold> mdeslaur: I think they're well aware of our concerns at this point and I think they'd take them into proper consideration when makin a decision.
21:12 <rbasak> 2. We expect to add HTTP2 support when they're ready upstreams through SRU, including a future MIR of nghttp2.
21:12 <rbasak> 3. Once added we expect support in main.
21:13 <mdeslaur> rbasak: that sounds accurate to me
21:13 <rbasak> Additionally we're not even building HTTP2 support right now, not even in universe.
21:13 <mdeslaur> rbasak: the only thing we probably should do is ask the release team about the feasibility of promoting nghttp2 after release
21:13 <rbasak> And we'll release Xenial this way.
21:13 <rbasak> ack. I'll take an action to ask that.
21:13 <teward> so then merely because we believe upstreams mark it as experimental, we're not shipping it, which is fine, but I will point out that videolan.org (and likely many others) have the NGINX http/2 running in production
21:13 <mdeslaur> I don't think it's an issue, but we'd better ask
21:14 <teward> (though I have no issue either way)
21:14 <mdeslaur> teward: it's about supporting it...if it's badly broken in some way and it's enabled, we are commited to fixing it
21:14 <teward> mdeslaur: right, i just wanted to make that one point
21:14 <rbasak> teward: that's fine I think. I think mdeslaur said earlier he'd like to see people using this in production before shipping in an LTS, so by definition something like that is expected.
21:14 <teward> right
21:14 <mdeslaur> ok, gotta go, thanks everyone
21:15 <rbasak> Thanks all.
21:15 <teward> rbasak: mdeslaur: not trying to say otherwise, just putting one example out there
21:15 <sarnold> rbasak: sounds great to me, thanks
21:15 <teward> and I doubt the 'experimental' state will go away in http/2 in nginx until Stable 1.10.x is cut from 1.9.x
21:15 <teward> (in April, potentially after release)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/ubuntu-release/attachments/20160105/b21bc981/attachment.pgp>