ARB Review

[Allison Randal]

On 04/12/2012 02:36 PM, Jono Bacon wrote:
> I agree, Allison. As I mentioned in my original mail to the ARB a few
> months ago, I think the primary bottleneck we are facing right now is
> a people one and getting enough ARB eyeballs in front of the apps.
> From what I have seen of the ARB in the last few months some members
> of the board have been much more active than others (as is common with
> any community board), and your contributions have been a good example
> of an active member. While I think that getting more ARB bums on seats
> will help open up potential capacity, I worry that we might just get
> more people on board who will also get busy with real life.

At the moment the recruiting language for new ARB members says that the
time commitment will be "no more than 5 hours a week", with nothing
about a minimum commitment of time each week.

https://wiki.ubuntu.com/AppReviewBoard/Responsibilities

I couldn't remember where the 5 hours came from, so dug back through
emails. Turns out it was from you. :) I gather from re-reading that at
the time you were trying to make it sound non-scary so we'd have an
easier time recruiting new members. We should probably change that to
"at least 4 hours per week". I certainly don't blame any of the current
members for not having enough time available, since we took such pains
to reassure them that we wouldn't need much time from them. Seems like a
bit of "bait and switch".

On the other hand, I have doubts how many new recruits we'll get if we
tell them they have to sign up for 4 hours every week. That's a pretty
hefty volunteer commitment. And, at the same time, a 4 hour block really
is the minimum needed from each volunteer to make real progress on the
queue. Maybe that will change once we manage to switch everyone over to
submitting by PPA.

>> More people will require more coordination, to make sure all those
>> volunteers aren't duplicating effort (working on the same packages).
>> But, we can ramp up coordination as we increase the number of volunteers.
> 
> I agree; more members will need more coordination and good leadership
> to ensure everyone is motivated and keeping on top of things.

This part seems totally doable, with a combination of human effort and
some technical improvements to the review system. Finding the bodies to
coordinate seems like the hard part.

>> The
>> greatest time seems to be spent responding to developers, explaining
>> what's wrong with their submission, and then re-reviewing when they
>> resubmit (often looping several times, when their second, third, and
>> fourth submissions also have problems).
> 
> This does indeed seem to be an area where things slow down the most.
> On one hand, as I have suggested, for developers that take weeks to
> respond if ever, I think we should have a fairly explicit date in
> which a response is required.

<shrug> We aren't blocking on time-limits at the moment. Maybe when we
have some way to see submissions waiting for a response from the
developer it'll become relevant. Right now, there really is no effective
difference between requesting more information and rejecting the
submission: the app drops out of our queue for both, and the developer
can resubmit an update to both, which brings it back into the queue. For
that matter, the automated message for a "Rejection" currently asks the
developer to fix the app and resubmit. :)

> I also wonder if there are any other ways in which we could automate
> some of the manual reviewing and checks going on. Maybe we could
> develop scripts or improvements to ease this?

One thing that would be helpful is if the submission system:

a) had a field for a link to a PPA for the submission (as an alternative
to uploading files), and

b) required the PPA field to be filled for ARB submissions (zero-cost,
FLOSS license)

We get all kinds of random junk submitted now, from binary packages, to
.jar files, to tarballs with no compilation instructions. And, we spend
a lot of time doing the "first cut" review, explaining to developers
what's wrong with their submission and how to fix it. Having the
submission form bounce them back until they get it right would be a very
helpful filter.

>> The weekend before last I revamped the submission guide, and started
>> adding links to it in responses to developers. This is helpful in giving
>> the developers more details, but in a standard way so the reviewer
>> doesn't have to spend much time on any one reply to a developer:
>>
>> https://wiki.ubuntu.com/AppReviewBoard/Submissions
> 
> Nice work!

Thanks. :)

>> I'd like to get this linked in from developer.ubuntu.com, so more
>> developers see it *before* they submit.
> 
> Agreed. This is something David Planella in the 12.10 cycle.

Great!

Allison