[maemo-developers] QA from extras-devel to extras-testing

[From: Quim Gil]

ext Jeremiah Foster wrote:
> On May 5, 2009, at 15:41, Quim Gil wrote:
> The problem is that many package  
> maintainers don't know the programming language of the software they  
> are packaging. If you are packaging something written in erlang you  
> will not be able to quickly fix bugs in that package if you don't know  
> erlang. This problem is a big one in debian, which is why they pass  
> bugs upstream. How many package maintainers know the code of the  
> package they maintain in maemo?

No idea and actually not the business of the QA community process. If an
application is causing real pain to users *now* it needs to be demoted
asap and be promoted again only when the pain is fixed.

The fact that a severe bug can't be fixed because there is only one
maintainer interested in it not knowing much about the code and lazy
about reporting upstream... would tell a bit about the quality of that
piece of software, by the way.


>> In any case it is not the business of the maemo.org extras QA  
>> process to
>> deal with the bugfix itself. This process should make sure the quality
>> of the apps offered to end users is good, promoting what is good and
>> demoting what is bad.
> 
> Fair enough, but "quality" needs to be specifically defined. I.e.  
> installs, de-installs, has a source tarball, description, etc.

Looking at the list proposed in the first email of this thread, this
could be:

Requirements for extras-testing  (should be testable automatically)
- Install and deinstall flawlessly.
- Don't bring conflicts in dependencies.
- Their info in the app manager is complete (icon, summary, URL to
project, updates info).
- Have decent page in maemo.org/downloads.
- Have a place to report issues to the developers.

Requirements for extras (humans are needed to test)
- Don't crash or freeze systems.
- Don't drain batteries.
- Are feature complete: everything inside works.
- Have been tested by someone trusted before.



>> Goals
>>
>> - We need to know when an application in extras-devel is ready to be
>> tested by power users (betatesters).
> 
> As soon as it is uploaded?

Developers upload to extras-devel, betatesters watch extras-testing.

A betatester could be a regular users willing to invest time going
trough fresh stuff and reporting about the findings. If we can save them
the major bugs automated testing can find, all the better.

> 
>> - We need to know when an application in extras-testing is ready for  
>> end
>> users.
> 
> After going through a policy check and two weeks of power users' tests?

2 weeks minimum and 10 testers OK with no blockers?

We can fine tune how many days and how many testers are required.

-- 
Quim Gil
open source advocate
Maemo Software @ Nokia