Communicating effectively with the QA Community

[Scott Kitterman]

Nicholas Skaggs <nicholas.skaggs at canonical.com> wrote:

>As asked in today's release meeting, I'm starting a thread to continue 
>the discussion of how the development and release teams communicate and
>work with the QA community. This was prompted by the recent changes 
>landing into quantal with little warning to most of the community. 

This should be considered the normal case.  Before FF and when there's not an Alpha freeze on most changes are just uploaded without any kind of pre-coordination.  Cleary people try to minimize issues, but things are going to happen and, no matter how hard we try to avoid it, people running the development release need to expect some breakage now and then.

>During the meeting the nvidia/xorg change and the unity2d changes were 
>discussed. To the extent it's helpful in understanding the issues, I 
>will quickly discuss them. I just want to ensure we don't over-analyze 
>these incidents as they are intended to be examples only :-)

AFAIK, the release team wasn't significantly involved in the Unity 2D changes.  Those were, AIUI, handled in the desktop team, which is what I'd have expected.

>So quickly to give the example that was discussed in the meeting. Xorg 
>1.13 was pushed into quantal, causing anyone running the nvidia 
>proprietary drivers to no longer boot to a desktop (x crashes). This 
>caused alot of stress and grief on the part of those running qantal and
>those testing quantal. I had no previous knowledge this was going to 
>occur, and certainly no one in our QA community did ether. It occurred 
>right before the weekend and the next week, I worked with popey and
>team 
>to setup some unity testing. To summarize life for the average qa 
>community member, here's what happened in order:

There was a packaging bug.  What should have happened is that Nvidia users should have been unable to upgrade into the broken configuration.  I don't think some held packages are a significant issue worthy of pre-coordination.  If there hadn't been the bug, this would have been minor.

>Unity2d is dropped, and unity may or may not be running anymore on my 
>hardware
>Unity no longer works in a vm (due to no more unity2d)
>Xorg breaks, rendering me unable to even boot into something like lxde 
>or xfce in the interim
>I'm asked to help test unity and report bugs
>
>Now clearly it's easy to see how the cards got stacked against them,
>and 
>in general they did not receive a good experience. To some of these 
>folks, me then asking them to help test is almost an insult, since
>there 
>ability to test has been severly hampered or removed.
>
>Now, I want to be clear that I don't believe that the changes should or
>should not have been done. That's a different discussion. Indeed, I 
>think the community would have reacted just fine had we done EXACTLY
>the 
>same thing, but simply told them in advance. If we're not communicating
>with the community, we have a one-sided and abusive relationship.

I read this list and hang out on #ubuntu-release and I managed to be aware of at least the X change (but not the bug) well in advance.  For people who care to know, it's not that hard to know what the release team knows.  Is the we in that sentence Canonical?

>Now, for part of this I assume responsibility for. For my part I have 
>been trying to pry more information out of the development teams on 
>what's happening and when and using that information to alert the folks
>in the testing community. However, I am not always aware (and indeed as
>the discussion today showed, the leads coming to the meeting are not 
>always aware) of changes that might impact everyone. I'd like us to 
>continue to be diligent in communicating to each other changes that
>will 
>be impactful.

Pre-coordinating anything but the largest of changes is a large burden.  Any teams I'm involved in do virtually all their work in public view.  I'm glad to help people figure out where to find it, but I think there is no where near the developer time available to pre-communicate every change that might break something.

>The larger piece is how we might better communicate that information to
>everyone. Some immediate ideas that came to mind are for me to funnel 
>information from the tech leads through the same outlets I do now for 
>things like testing events -- perhaps even on a weekly basis.. A bit of
>a pulse of what's going on in development. This has been happening on
>an 
>ad-hoc and more limited basis so far this cycle. Going further, a 
>longer-term idea was integrating a package into the development release
>to push out new information to anyone running it about what's going on.
>
>Things like when important changes are landing and or when we want 
>testing. It would be helpful to push that out to everyone on the 
>desktop, with no requirements for them to "sign-up". Last UDS I 
>presented this idea roughly as an indicator that could be hooked into
>to 
>push out testing notifications. That same idea could be extended to
>push 
>general purpose information as well. Obviously that's a very rough
>idea, 
>but I'm curious to hear thoughts on the matter.
>
>Now, in closing, the alternate images are likely to be officially 
>dropped at beta (as Steve shared today). This is a good chance next
>week 
>to try and be more proactive about communicating this change with 
>everyone. I plan to keep an eye out for Steve's thread on -devel (Steve
>you pinging me when you drop it would be wonderful), and sharing with 
>everyone in the community in advance that the discussion (and outcome) 
>will be happening for beta. I think Steve mentioning it and then 
>following up to make sure it's communicated would be a wonderful change
>and would help prevent future issues like what we encountered recently.
>
>Other ideas / thoughts? I just want to make sure the QA community is a 
>well integrated and important part of the development process. This 
>discussion is part of making that happen.

Now that it's post feature freeze, you can look at FFe requests as an indicator of other things that might be coming.

Scott K