[maemo-developers] Follow up from QA meeting on IRC

[From: Jeremiah Foster]

On Nov 12, 2009, at 9:17, Dave Neary wrote:

> Hi,
> 
> Edward Page wrote:
>> To help remind people that there is a checklist and whats on it,
>> should the rating page link to or include the criteria?

Yes - that makes sense.
>> 
>> I see there were no notes on the algorithm.  A threshold of 10 was
>> annoying as a developer.  As a tester, a threshold of 10 made me feel
>> more comfortable not doing a full blown /opt check or power management
>> check because of 10 people I could hope someone else would do it and I
>> could worry about other issues like application stability.  With a
>> smaller threshold I would feel more of a burden to do all of the steps
>> which would discourage me.
> 
> Ed's point definitely resonates with me. The great thing about QA is
> that you can crowd-source it effectively if you don't require much of
> the user/tester. It seems like the Maemo QA process is more
> developer-focussed than user-focussed at the moment, and is as such
> pushing a lot of the responsibility for the QA process to the user.

QA is logically the next step after development and is often done by an engineer, at least in the corporate world. Even in maemo, those who do QA so far have also been developers; either they have done QA on their own package or they have done it informally for other's and use the extras-testing interface.

In short - the expectation is that you are pretty deep into maemo.
> 
> This seems like an ideal opportunity to lower the barrier to
> participation to tiny levels, but only if it is
> 
> 1. easy to give a +1/-1

Well, we need to keep this distinct from merely rating the package. We are not rating here, we are promoting quality packages. You may be asked to promote a package you think frivolous, but if it meets the technical criteria you are asked to promote it.

> 2. We don't require intimate knowledge of the Maemo community for
> feedback (I'm thinking of the checklist, what "optification" means, etc)

Feedback is welcome through a variety of means - rating the package with stars on the download page is one for example. Feedback is welcome in QA too, but let's not pretend that this isn't a serious technical process requiring at least some knowledge of the underlying operating system and perhaps the package system and even how the development process works (on a high level). 

We need to insure that those who are doing QA, can handle the technical demands so that the software does not damage the device and maemo's reputation. This really does require a clear specification, i.e. checklist, and knowing what optification means in the maemo context.

> 3. We require enough feedback that most of the code paths in the
> application will be tested before we OK an application
> 
> Lowering the threshold to 5 is implicitly saying "we're not getting
> feedback quickly enough",

I think it is really saying that the bar is currently set to high.

> which in turn is saying "the feedback process
> is overly cumbersome for a casual user".

Yes and it should be. This is critically important and has serious implications and technical challenges. Anyone at all is welcome, but you may need to learn some stuff about the QA process. The QA process is hard because developing is hard, writing code is hard, integrating into an OS is hard - we need to make sure those things work.

> It seems to me that that's the
> problem, rather than the contents of the checklist or the threshold in
> place. If giving feedback was trivially easy (as it is, for example, in
> Android Market) you would be getting hundreds of votes when new versions
> of applications are released, as people installed & used them.

We need a separate mechanism for this then, perhaps this is similar to the app rating system we already have on downloads?
> 
> So - how can we make giving the feedback and voting on applications easier?

I don't think the idea of "Quality Software" fits with the idea of "Easy to do". Exhibit A: Windows. Many developers are familiar with this saying said to management only half in jest: Quality, Time, Cost - pick two.

Seriously, we don't want people surfing by and clicking pejoratively. We need a set of serious technical criteria to assure quality software, not a warm and fuzzy interface where the hoi polloi randomly click shiny buttons.

Jereamiah