Patch Pilot report

[Stephen J. Turnbull]

John Arbash Meinel writes:

 > To be honest, something doesn't seem right with our process if it
 > takes 50% of a core developers time just to keep up with the review
 > queue.

That's a seriously bogus calculation.  You *must* do this in terms of
man-hours per submission, and evaluate its usefulness in terms of the
ratio of review time to submitter time and the value of the
contributions themselves.  You're aware of all that, yet you are
focusing on core developer time spent.  Don't do that yet; you simply
don't have enough information about the benefits to make a judgment at
this point.

 > Certainly increasing community contributions is a good thing. Especially
 > if the time spent is beneficial. But a contributor spending 10 min
 > causes you to spend an hour, then our balance is out of whack.

This is another bogus calculation.  Let's take it to the limit.  If
you pick a bug on the tracker and implement a fix in an hour, then the
contributor spent zero minutes writing the patch and you spent sixty.
Wow, that 60/0 ratio sucks infinitely, doesn't it!?  Of course not.

So, if in fact a contributor spending 10 minutes on a patch means you
spend 60 instead of 65, and it's "something that should be done
eventually" anyway, you're probably ahead of the game already.  Also,
I doubt you're getting very many 10 minute contributions.  Probably
few of those contributions take less time for the contributor
(localization, reading and understanding the code, etc) than for the
reviewer.  What you're really comparing when you say "10 minutes" is
how long it would take you to write the patch once you understood the
problem, right?

I would suggest (1) *first*, look at the results (# of patches landed,
value of patches landed, both divided into committer and
non-committer), and (2) all committers keeping a log of time spent
reviewing, divided into patch pilot time and other reviewing, for
comparison purposes.

(2) is of course a Massive Pain In The Butt.  But that will give you
the data needed for sensible kaizenning.  I suggest intuitively that
the right answer may be more non-Canonical committers, or perhaps
consulting contracts to non-Canonical committers to function as patch
pilots, but you'll have to look at the data to decide.

Also, patch piloting is an investment in future contributions.  (1) It
encourages contributors to come back.  (2) Depending on how the patch
pilots function, there may be substantial educational benefits to the
process for contributors, who will use less patch pilot time in future
contributions.  Obviously that's pretty nebulous, but as time goes on
you'll get a feel for whether those benefits are accruing, and how
much.  Unless you really hit a crunch on some features, I'd recommend
letting the patch pilots spend as much of their time on piloting as
they can stand ;-) until mid-Spring.  It will take that long for that
kind of result to show.  It should also show in the data, but only 6
months' worth will probably still be prety fuzzy.