Communicating effectively with the QA Community

[Nicholas Skaggs]

As asked in today's release meeting, I'm starting a thread to continue 
the discussion of how the development and release teams communicate and 
work with the QA community. This was prompted by the recent changes 
landing into quantal with little warning to most of the community. 
During the meeting the nvidia/xorg change and the unity2d changes were 
discussed. To the extent it's helpful in understanding the issues, I 
will quickly discuss them. I just want to ensure we don't over-analyze 
these incidents as they are intended to be examples only :-)

So quickly to give the example that was discussed in the meeting. Xorg 
1.13 was pushed into quantal, causing anyone running the nvidia 
proprietary drivers to no longer boot to a desktop (x crashes). This 
caused alot of stress and grief on the part of those running qantal and 
those testing quantal. I had no previous knowledge this was going to 
occur, and certainly no one in our QA community did ether. It occurred 
right before the weekend and the next week, I worked with popey and team 
to setup some unity testing. To summarize life for the average qa 
community member, here's what happened in order:

Unity2d is dropped, and unity may or may not be running anymore on my 
hardware
Unity no longer works in a vm (due to no more unity2d)
Xorg breaks, rendering me unable to even boot into something like lxde 
or xfce in the interim
I'm asked to help test unity and report bugs

Now clearly it's easy to see how the cards got stacked against them, and 
in general they did not receive a good experience. To some of these 
folks, me then asking them to help test is almost an insult, since there 
ability to test has been severly hampered or removed.

Now, I want to be clear that I don't believe that the changes should or 
should not have been done. That's a different discussion. Indeed, I 
think the community would have reacted just fine had we done EXACTLY the 
same thing, but simply told them in advance. If we're not communicating 
with the community, we have a one-sided and abusive relationship.

Now, for part of this I assume responsibility for. For my part I have 
been trying to pry more information out of the development teams on 
what's happening and when and using that information to alert the folks 
in the testing community. However, I am not always aware (and indeed as 
the discussion today showed, the leads coming to the meeting are not 
always aware) of changes that might impact everyone. I'd like us to 
continue to be diligent in communicating to each other changes that will 
be impactful.

The larger piece is how we might better communicate that information to 
everyone. Some immediate ideas that came to mind are for me to funnel 
information from the tech leads through the same outlets I do now for 
things like testing events -- perhaps even on a weekly basis.. A bit of 
a pulse of what's going on in development. This has been happening on an 
ad-hoc and more limited basis so far this cycle. Going further, a 
longer-term idea was integrating a package into the development release 
to push out new information to anyone running it about what's going on. 
Things like when important changes are landing and or when we want 
testing. It would be helpful to push that out to everyone on the 
desktop, with no requirements for them to "sign-up". Last UDS I 
presented this idea roughly as an indicator that could be hooked into to 
push out testing notifications. That same idea could be extended to push 
general purpose information as well. Obviously that's a very rough idea, 
but I'm curious to hear thoughts on the matter.

Now, in closing, the alternate images are likely to be officially 
dropped at beta (as Steve shared today). This is a good chance next week 
to try and be more proactive about communicating this change with 
everyone. I plan to keep an eye out for Steve's thread on -devel (Steve 
you pinging me when you drop it would be wonderful), and sharing with 
everyone in the community in advance that the discussion (and outcome) 
will be happening for beta. I think Steve mentioning it and then 
following up to make sure it's communicated would be a wonderful change 
and would help prevent future issues like what we encountered recently.

Other ideas / thoughts? I just want to make sure the QA community is a 
well integrated and important part of the development process. This 
discussion is part of making that happen.

Cheers,

Nicholas