[maemo-developers] QA from extras-devel to extras-testing

[From: Graham Cobb]

On Tuesday 05 May 2009 16:30:41 Quim Gil wrote:
> ext Jeremiah Foster wrote:
> > On May 5, 2009, at 15:41, Quim Gil wrote:
> >> - We need to know when an application in extras-testing is ready for
> >> end
> >> users.
> >
> > After going through a policy check and two weeks of power users' tests?
>
> 2 weeks minimum and 10 testers OK with no blockers?
>
> We can fine tune how many days and how many testers are required.

Sorry, for the late reply: I have been away all week with no access to 
personal email.

I agreed with most of Quim's and Jeremiah's points until this one.

Absolutely, definitely not!!  A specified time and a number of testers is not 
a reasonable or practical way to decide when the application is ready for 
promotion: a human being needs to make a judgement.

For a fairly popular app like GPE (and particularly for very popular apps like 
Canola), 2 weeks and 10 people is not be enough.  I released a Beta version 
of GPE months ago and have not promoted it to Extras because I know of one 
problem which is potentially serious and I haven't fixed it yet -- people are 
better off sticking with the previous version unless they are willing to act 
as beta testers.  But none of the beta testers has been able to log a 
reproducible report (which is why I haven't been able to fix it -- but it 
occurs enough to clearly be a real problem).

On the other hand, a small application may only have 20 users so finding 10 
beta testers may be impossible!

And 2 weeks may be much too long to wait if the update is a security update or 
fixes a serious bug or provides a much-needed library that is blocking other 
packages.

The bottom line is that I do not believe an automatic promotion is possible: 
there should be a set of people who have the privilege to do the promotion 
and the submitter should request the update and have to persuade one of the 
set of people to do the promotion.  2 weeks and 10 testers may be a guideline 
but we should trust the people who are given upgrade privileges to make the 
decision.

Graham