[maemo-developers] QA from extras-devel to extras-testing

[From: Jeremiah Foster]

On May 5, 2009, at 17:30, Quim Gil wrote:


> ext Jeremiah Foster wrote:
>> On May 5, 2009, at 15:41, Quim Gil wrote:
>> The problem is that many package
>> maintainers don't know the programming language of the software they
>> are packaging. If you are packaging something written in erlang you
>> will not be able to quickly fix bugs in that package if you don't  
>> know
>> erlang. This problem is a big one in debian, which is why they pass
>> bugs upstream. How many package maintainers know the code of the
>> package they maintain in maemo?
>
> No idea and actually not the business of the QA community process.  
> If an
> application is causing real pain to users *now* it needs to be demoted
> asap and be promoted again only when the pain is fixed.

Okay.
>
> The fact that a severe bug can't be fixed because there is only one
> maintainer interested in it not knowing much about the code and lazy
> about reporting upstream... would tell a bit about the quality of that
> piece of software, by the way.

Yes, true. I just seems a little tough, but perhaps this is what is  
required to increase quality.
>
>>> In any case it is not the business of the maemo.org extras QA
>>> process to
>>> deal with the bugfix itself. This process should make sure the  
>>> quality
>>> of the apps offered to end users is good, promoting what is good and
>>> demoting what is bad.
>>
>> Fair enough, but "quality" needs to be specifically defined. I.e.
>> installs, de-installs, has a source tarball, description, etc.
>
> Looking at the list proposed in the first email of this thread, this
> could be:
>
> Requirements for extras-testing  (should be testable automatically)
> - Install and deinstall flawlessly.
> - Don't bring conflicts in dependencies.
> - Their info in the app manager is complete (icon, summary, URL to
> project, updates info).
> - Have decent page in maemo.org/downloads.
> - Have a place to report issues to the developers.
>
> Requirements for extras (humans are needed to test)
> - Don't crash or freeze systems.
> - Don't drain batteries.
> - Are feature complete: everything inside works.
> - Have been tested by someone trusted before.

These seem reasonable to me.

>>> Goals
>>>
>>> - We need to know when an application in extras-devel is ready to be
>>> tested by power users (betatesters).
>>
>> As soon as it is uploaded?
>
> Developers upload to extras-devel, betatesters watch extras-testing.
>
> A betatester could be a regular users willing to invest time going
> trough fresh stuff and reporting about the findings. If we can save  
> them
> the major bugs automated testing can find, all the better.

This seems like an excellent solution to me.

>>
>>> - We need to know when an application in extras-testing is ready for
>>> end
>>> users.
>>
>> After going through a policy check and two weeks of power users'  
>> tests?
>
> 2 weeks minimum and 10 testers OK with no blockers?

Yes.

> We can fine tune how many days and how many testers are required.

Yes, this allows us to get started and adjust along the way.

Jeremiah