[maemo-developers] QA from extras-devel to extras-testing

[From: Quim Gil]

Hi,

ext Graham Cobb wrote:
> Not sure about the requirement to have Nitro installed.  In particular, what 
> happens to the (potentially large number) of people using maemo-testing who 
> do not have Nitro installed?  What happens if none of the people who want to 
> beta test this package have Nitro installed.

I don't know why on earth would someone refuse to install Nitro (silent,
convenient, informative) when is actively joining a testing program to
assess the quality of the software that is trying out.

In any case, using extras-testing and not using Nitro is not the big
deal. The big deal would be if people would actively rate "Yes, this app
is OK" without having a crash reporter installed.

If you mean that 20 people well organized can push a buggy package of
extras bypassing Nitro... well, that would be true if they are the only
20 testing it. If a 21st tries and makes Nitro detect crashes, with one
reporter you have enough to start suspecting of a package.


> Also, what if this is a brand new application (e.g. liqbase, last year) -- how 
> does the developer recruit beta testers?

It's extras-testing who needs to recrit betatesters, not a specific app.
Imagine 200 people getting their 'betatesters driving license' and going
to a page where they see the status of apps tested, how many votes, how
many days, what needs special attention...

Installing a new app and giving it a try of 10 minute and then rating if
there were showstoppers or not is not a big-big deal.


>  What if this is a commercial 
> company who has created an app and wants it available as soon as possible?

maemo.org doesn't need to bother about commercial companies with special
needs. Forum Nokia does that.

>> 3a. If they find severe bugs the packages go back to extras devel.
> 
> This at the judgement of the developer, presumably (one man's "severe" may be 
> different from another's).  Downgrading packages in repositories does not 
> work well so this would have to be a rare occurence.

Let me bring again the

Requirements for extras (humans are needed to test)
- Don't crash or freeze systems.
- Don't drain batteries.
- Are feature complete: everything inside works.
- Have been tested by someone trusted before.

Crash and freezes are easy to detect. Nitro helps reporting them
automatically with traces to a server.

About draining batteries we hope to have a tool easy to use for betatesters.

Everything inside works is also relatively objective: you go through the
menu options and you check that there is nothing obvious. Normal and
minor bugs are ok, happen to everybody.

The latter is achieved by the requirement of 20 approvers.

Also, let's say that downgrading from testing to extras is less severe
than downgrading from extras to testing. I agree that stronger reasons
need to be provided for that.


> 
>> 3b. If nobody finds a showstopper the app goes to extras after N weeks
>> and N votes.
> 
> Certainly not if the developer has not requested it.

Good point. The option by default might be "Not releasable", and this
would stop a package even with 500 favorable ranks. This might be useful
to organize releases, and no just find out that your app made it to
extras with big success (or awful feedback) precisely when you were 3
weeks on holidays.

Then for small releases developers in a hurry might decide to select
"Releasable" when they push the app to extras-testing, and forget about
it (unless it comes back).


> I don't see the humans as "more trustworthy", just more flexible.  Maemo is a 
> small community, with a very small number of packages compared with (say) 
> Debian.  And as a "mobile device" oriented distribution we need to be 
> encouraging developers to make availabe neat applications, of good quality, 
> as quickly as possible.   And I don't see a fixed rule as being the best way 
> to achieve that.  
> 
> I would rather that any one of a team of (say) Jeremiah, Jaffa, Qole and a few 
> others reviewed a submission request form allowing the developer to explain 
> what testing has happened, by whom, over what period of time and then made 
> the upgrade if they are convinced.  The "N weeks, M testers" would be the 
> standard guideline but the reviewer can change the criteria if they feel it 
> is appropriate to the circumstances.

At this point I can only say: you know better than me. I have just tried
to propose in broad terms a community process that both developers,
community and Nokia could agree upon. You have our total trust defining
all the details you think are better to pursue the main goal.

-- 
Quim Gil
open source advocate
Maemo Software @ Nokia