[maemo-developers] QA from extras-devel to extras-testing

[From: gary liquid]

automated unit testing will not find all bugs or problems in perfectly
packages software.
manual human testing will not find all bugs or problems in perfectly
packaged software.

a combination of both involving common sense and prior history should
minimize the risk of problems though.

gary

On Mon, May 11, 2009 at 10:42 AM, Jeremiah Foster <
jeremiah at jeremiahfoster.com> wrote:

>
> On May 10, 2009, at 2:02, Graham Cobb wrote:
>
> > On Tuesday 05 May 2009 16:30:41 Quim Gil wrote:
> >> ext Jeremiah Foster wrote:
> >>> On May 5, 2009, at 15:41, Quim Gil wrote:
> >>>> - We need to know when an application in extras-testing is ready
> >>>> for
> >>>> end
> >>>> users.
> >>>
> >>> After going through a policy check and two weeks of power users'
> >>> tests?
> >>
> >> 2 weeks minimum and 10 testers OK with no blockers?
> >>
> >> We can fine tune how many days and how many testers are required.
> >
> > I agreed with most of Quim's and Jeremiah's points until this one.
> >
> > Absolutely, definitely not!!  A specified time and a number of
> > testers is not
> > a reasonable or practical way to decide when the application is
> > ready for
> > promotion: a human being needs to make a judgement.
>
> Unfortunately, having a human decide if an app is ready for promotion
> is fraught with problems, making the human promotion process nearly
> impossible.
> >
> > For a fairly popular app like GPE (and particularly for very popular
> > apps like
> > Canola), 2 weeks and 10 people is not be enough.  I released a Beta
> > version
> > of GPE months ago and have not promoted it to Extras because I know
> > of one
> > problem which is potentially serious and I haven't fixed it yet --
> > people are
> > better off sticking with the previous version unless they are
> > willing to act
> > as beta testers.  But none of the beta testers has been able to log a
> > reproducible report (which is why I haven't been able to fix it --
> > but it
> > occurs enough to clearly be a real problem).
>
> Don't you think you should label packages as a "beta" in the version
> number so that potential downloaders are aware that this is a problem?
> There will be a mechanism to allow packages to stay in testing and not
> get automatically promoted.
>
> > On the other hand, a small application may only have 20 users so
> > finding 10
> > beta testers may be impossible!
>
> This is why it needs to be an automated process. _Every_ package needs
> to be tested.
>
> > And 2 weeks may be much too long to wait if the update is a security
> > update or
> > fixes a serious bug or provides a much-needed library that is
> > blocking other
> > packages.
>
> Debian fortunately has a mechanism for promoting packages more quickly
> if there is a security problem. I think incorporating this ability to
> quickly promote packages might be a good idea.
> >
> > The bottom line is that I do not believe an automatic promotion is
> > possible:
>
> There is no other way. Here's why;
>
>        1. Too many packages - humans don't scale well.
>        2. The promotion policy needs to be tested by an automated process
> with proper regression and unit tests - this is a standard part of
> professional quality assurance, there is no reason that maemo
> shouldn't have the same professional standards.
>        3. The process should be codified and clear - if you build a well-
> made package, if it passes all the requirements, you get promoted.
> This eliminates favoritism and any biases and rewards meritocratic
> adherence to the packaging and development policy.
>
> >
> > there should be a set of people who have the privilege to do the
> > promotion
> > and the submitter should request the update and have to persuade one
> > of the
> > set of people to do the promotion.
>
> Who will do this? How long will they stick around? What happens if
> they like one type of package and not another? How long will it take
> to decide who does this and the process of choosing them?
>
> > 2 weeks and 10 testers may be a guideline
> > but we should trust the people who are given upgrade privileges to
> > make the
> > decision.
>
> Maemo does and will continue to do so, but will also run some tests on
> new software so Maemo can assure users (who ought to always be the
> focus) that the software will work as advertised.
>
> Jeremiah
> _______________________________________________
> maemo-developers mailing list
> maemo-developers at maemo.org
> https://lists.maemo.org/mailman/listinfo/maemo-developers
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.maemo.org/pipermail/maemo-developers/attachments/20090511/0e5d2f29/attachment.htm