[maemo-developers] QA from extras-devel to extras-testing

[From: gary liquid]

absolutely quim,
I felt nervous being able to push the first release of liqbase from -devel
to extras all on my own, I initially thought back then these ^^^^ steps
being discussed now were already in place and that I couldnt promote my own
package.

i do think there should be the flexibility of a fastrack for
security/serious bugfix push because the principle package has already had
the majority of the work done and already spent time in serious testing, it
will just be something to play by ear, different packages effect systems in
different and inventive ways (reminded of such with my playtest)

gary

On Mon, May 11, 2009 at 3:13 PM, Quim Gil <quim.gil at nokia.com> wrote:

>
>
> ext gary liquid wrote:
> > automated unit testing will not find all bugs or problems in perfectly
> > packages software.
> > manual human testing will not find all bugs or problems in perfectly
> > packaged software.
> >
> > a combination of both involving common sense and prior history should
> > minimize the risk of problems though.
>
> ... and this is exactly what I proposed!
>
> 1. Developer pushes packages from extras-devel to extras testing.
>
> 2. Automated testing does the checks.
>
> 2a. If the test fails, the packages don't get to extras-testing.
>
> 2b. If the test is successful, the packages go to extras testing.
>
> 3. In extras-testing the betatesters put the software into stress,
> equipped with Nitro (crash reporter installed in their devices) plus
> whatever tools they can use voluntarily.
>
> 3a. If they find severe bugs the packages go back to extras devel.
>
> 3b. If nobody finds a showstopper the app goes to extras after N weeks
> and N votes.
>
> I see the point of asking for qualified humans giving the green light as
> something more trustworthy than community testers rating. The problem I
> see is that these people are usually busy and being the filter can put a
> lot of stress in them (holidays, exams, problems at home... and a dozen
> of developers waiting for you to press a green button).
>
> This is why this option is preferable. Skilled humans can still test
> whatever comes and raise the blocker bugs, before or after the final
> release.
>
> In fact, what we probably should and will do is test more those apps
> that are most downloaded, for a simple logical reason: they are more
> used. Also, more automated testing and torture could be put on the most
> downloaded apps. But doing intensive and reliable testing on every
> package uploaded is not scalable, probably not even when automated since
> there are all kinds of apps, languages, dependencies...
>
> If a major bug could make it inside a final release of an app downloaded
> 50 times and the chance for the bug to explode is 1/1000... tough luck.
> It happens all the time.
>
>
> 2 weeks in the pipeline in exchange of free true testing and the trust
> of Nokia for potential promotion and collaboration sounds like a good
> deal to me. Testing the latest software 2 weeks before any regular use
> even sees the releases sounds also like fan and a good deal for the
> betatesters.
>
> --
> Quim Gil
> open source advocate
> Maemo Software @ Nokia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.maemo.org/pipermail/maemo-developers/attachments/20090511/a2750779/attachment.htm