[maemo-developers] QA from extras-devel to extras-testing

[From: Jeremiah Foster]

On May 10, 2009, at 2:02, Graham Cobb wrote:

> On Tuesday 05 May 2009 16:30:41 Quim Gil wrote:
>> ext Jeremiah Foster wrote:
>>> On May 5, 2009, at 15:41, Quim Gil wrote:
>>>> - We need to know when an application in extras-testing is ready  
>>>> for
>>>> end
>>>> users.
>>>
>>> After going through a policy check and two weeks of power users'  
>>> tests?
>>
>> 2 weeks minimum and 10 testers OK with no blockers?
>>
>> We can fine tune how many days and how many testers are required.
>
> I agreed with most of Quim's and Jeremiah's points until this one.
>
> Absolutely, definitely not!!  A specified time and a number of  
> testers is not
> a reasonable or practical way to decide when the application is  
> ready for
> promotion: a human being needs to make a judgement.

Unfortunately, having a human decide if an app is ready for promotion  
is fraught with problems, making the human promotion process nearly  
impossible.
>
> For a fairly popular app like GPE (and particularly for very popular  
> apps like
> Canola), 2 weeks and 10 people is not be enough.  I released a Beta  
> version
> of GPE months ago and have not promoted it to Extras because I know  
> of one
> problem which is potentially serious and I haven't fixed it yet --  
> people are
> better off sticking with the previous version unless they are  
> willing to act
> as beta testers.  But none of the beta testers has been able to log a
> reproducible report (which is why I haven't been able to fix it --  
> but it
> occurs enough to clearly be a real problem).

Don't you think you should label packages as a "beta" in the version  
number so that potential downloaders are aware that this is a problem?  
There will be a mechanism to allow packages to stay in testing and not  
get automatically promoted.

> On the other hand, a small application may only have 20 users so  
> finding 10
> beta testers may be impossible!

This is why it needs to be an automated process. _Every_ package needs  
to be tested.

> And 2 weeks may be much too long to wait if the update is a security  
> update or
> fixes a serious bug or provides a much-needed library that is  
> blocking other
> packages.

Debian fortunately has a mechanism for promoting packages more quickly  
if there is a security problem. I think incorporating this ability to  
quickly promote packages might be a good idea.
>
> The bottom line is that I do not believe an automatic promotion is  
> possible:

There is no other way. Here's why;

	1. Too many packages - humans don't scale well.
	2. The promotion policy needs to be tested by an automated process  
with proper regression and unit tests - this is a standard part of  
professional quality assurance, there is no reason that maemo  
shouldn't have the same professional standards.
	3. The process should be codified and clear - if you build a well- 
made package, if it passes all the requirements, you get promoted.  
This eliminates favoritism and any biases and rewards meritocratic  
adherence to the packaging and development policy.

>
> there should be a set of people who have the privilege to do the  
> promotion
> and the submitter should request the update and have to persuade one  
> of the
> set of people to do the promotion.

Who will do this? How long will they stick around? What happens if  
they like one type of package and not another? How long will it take  
to decide who does this and the process of choosing them?

> 2 weeks and 10 testers may be a guideline
> but we should trust the people who are given upgrade privileges to  
> make the
> decision.

Maemo does and will continue to do so, but will also run some tests on  
new software so Maemo can assure users (who ought to always be the  
focus) that the software will work as advertised.

Jeremiah