[maemo-developers] QA from extras-devel to extras-testing

[From: Graham Cobb]

On Monday 11 May 2009 15:13:05 Quim Gil wrote:
> 1. Developer pushes packages from extras-devel to extras testing.
>
> 2. Automated testing does the checks.
>
> 2a. If the test fails, the packages don't get to extras-testing.
>
> 2b. If the test is successful, the packages go to extras testing.
>
> 3. In extras-testing the betatesters put the software into stress,
> equipped with Nitro (crash reporter installed in their devices) plus
> whatever tools they can use voluntarily.

Not sure about the requirement to have Nitro installed.  In particular, what 
happens to the (potentially large number) of people using maemo-testing who 
do not have Nitro installed?  What happens if none of the people who want to 
beta test this package have Nitro installed.

Also, what if this is a brand new application (e.g. liqbase, last year) -- how 
does the developer recruit beta testers?  What if this is a commercial 
company who has created an app and wants it available as soon as possible?

> 3a. If they find severe bugs the packages go back to extras devel.

This at the judgement of the developer, presumably (one man's "severe" may be 
different from another's).  Downgrading packages in repositories does not 
work well so this would have to be a rare occurence.

> 3b. If nobody finds a showstopper the app goes to extras after N weeks
> and N votes.

Certainly not if the developer has not requested it.

For GPE, I release one or two beta releases over a period of several weeks or 
months in the run up to a production release.  These need to go into 
extras-testing because that is where the people who like to run and test new 
stuff live.  They are not candidates to be promoted to extras because they 
are beta releases -- even if they are perfect they would need descriptions 
changing.  And, more realistically, there will be bugs reported, some of 
which I will want to fix before release.

Eventually I create a production release.  This needs a brief sanity check 
(for the last production release I gave this 2 days and about 3-4 people 
tried it out) and then should go into extras.  I would prefer that sanity 
check to be faster than 2 weeks (if previous, similar but not identical, 
versions have already been in extras-testing for several months).  But for 
GPE I could live with the N weeks, M testers for this sanity check.

> I see the point of asking for qualified humans giving the green light as
> something more trustworthy than community testers rating. The problem I
> see is that these people are usually busy and being the filter can put a
> lot of stress in them (holidays, exams, problems at home... and a dozen
> of developers waiting for you to press a green button).

I don't see the humans as "more trustworthy", just more flexible.  Maemo is a 
small community, with a very small number of packages compared with (say) 
Debian.  And as a "mobile device" oriented distribution we need to be 
encouraging developers to make availabe neat applications, of good quality, 
as quickly as possible.   And I don't see a fixed rule as being the best way 
to achieve that.  

I would rather that any one of a team of (say) Jeremiah, Jaffa, Qole and a few 
others reviewed a submission request form allowing the developer to explain 
what testing has happened, by whom, over what period of time and then made 
the upgrade if they are convinced.  The "N weeks, M testers" would be the 
standard guideline but the reviewer can change the criteria if they feel it 
is appropriate to the circumstances.

> In fact, what we probably should and will do is test more those apps
> that are most downloaded, for a simple logical reason: they are more
> used. Also, more automated testing and torture could be put on the most
> downloaded apps. But doing intensive and reliable testing on every
> package uploaded is not scalable, probably not even when automated since
> there are all kinds of apps, languages, dependencies...
>
> If a major bug could make it inside a final release of an app downloaded
> 50 times and the chance for the bug to explode is 1/1000... tough luck.
> It happens all the time.

No single N weeks and M testers criterion will work well both for apps 
downloaded 50 times and apps downloaded 50,000 times.

Graham