[maemo-developers] QA from extras-devel to extras-testing

[From: Quim Gil]

ext gary liquid wrote:
> automated unit testing will not find all bugs or problems in perfectly
> packages software.
> manual human testing will not find all bugs or problems in perfectly
> packaged software.
> 
> a combination of both involving common sense and prior history should
> minimize the risk of problems though.

... and this is exactly what I proposed!

1. Developer pushes packages from extras-devel to extras testing.

2. Automated testing does the checks.

2a. If the test fails, the packages don't get to extras-testing.

2b. If the test is successful, the packages go to extras testing.

3. In extras-testing the betatesters put the software into stress,
equipped with Nitro (crash reporter installed in their devices) plus
whatever tools they can use voluntarily.

3a. If they find severe bugs the packages go back to extras devel.

3b. If nobody finds a showstopper the app goes to extras after N weeks
and N votes.

I see the point of asking for qualified humans giving the green light as
something more trustworthy than community testers rating. The problem I
see is that these people are usually busy and being the filter can put a
lot of stress in them (holidays, exams, problems at home... and a dozen
of developers waiting for you to press a green button).

This is why this option is preferable. Skilled humans can still test
whatever comes and raise the blocker bugs, before or after the final
release.

In fact, what we probably should and will do is test more those apps
that are most downloaded, for a simple logical reason: they are more
used. Also, more automated testing and torture could be put on the most
downloaded apps. But doing intensive and reliable testing on every
package uploaded is not scalable, probably not even when automated since
there are all kinds of apps, languages, dependencies...

If a major bug could make it inside a final release of an app downloaded
50 times and the chance for the bug to explode is 1/1000... tough luck.
It happens all the time.


2 weeks in the pipeline in exchange of free true testing and the trust
of Nokia for potential promotion and collaboration sounds like a good
deal to me. Testing the latest software 2 weeks before any regular use
even sees the releases sounds also like fan and a good deal for the
betatesters.

-- 
Quim Gil
open source advocate
Maemo Software @ Nokia