[maemo-developers] QA from extras-devel to extras-testing

[From: Graham Cobb]

On Monday 11 May 2009 17:58:46 Quim Gil wrote:
> > Also, what if this is a brand new application (e.g. liqbase, last year)
> > -- how does the developer recruit beta testers?
>
> It's extras-testing who needs to recrit betatesters, not a specific app.
> Imagine 200 people getting their 'betatesters driving license' and going
> to a page where they see the status of apps tested, how many votes, how
> many days, what needs special attention...

Now I am very confused.  Your view of extras-testing is very different from 
mine.

I think we are agreed that extras-testing is going to be a place to put apps 
that the developers think are of beta quality and are trying to go down the 
path to release in extras.  I also thought that a lot of people (including 
probably most of those who actually contribute to maemo-users or ITT) would 
choose to have extras-testing permanently enabled on their tablet and would 
tend to update new software from it frequently (both things that are 
dangerous to do with extras-devel today).

However, I did not think that the bulk of the people using extras-testing 
would actively think of themselves as beta testers, would apply for 
a "betatesters driving license" or would even rate applications.  I would 
hope that a proportion of them would be willing to log a bug report if they 
found a problem but would be surprised if it was more than 10%.

I assumed that the "N weeks and M testers" requirement would really be "in N 
weeks, M people downloaded it and no one reported any serious problems".  If 
we are expecting an actual report on it, or any active testing (trying out 
menu options, etc.), then even for popular apps I don't see M being larger 
than about 3!  When I release a Beta version of GPE and announce it on ITT, 
asking for people to reply to let me know they have downloaded it and tried 
it, I get about 3-5 responses.  I know others have also downloaded it because 
I sometimes get later replies asking questions, reporting problems, etc.  I 
have to assume that even more people downloaded it and haven't had a problem.

> Installing a new app and giving it a try of 10 minute and then rating if
> there were showstoppers or not is not a big-big deal.

If Nokia or the Maemo community can come up with a way to recruit/incentivise 
people to do that then I am very supportive.  It would be great!  But I have 
not heard any ideas on how to do that.  I don't think 200 people will do that 
without some serious incentive.

> >> 3a. If they find severe bugs the packages go back to extras devel.
> >
> > This at the judgement of the developer, presumably (one man's "severe"
> > may be different from another's).  Downgrading packages in repositories
> > does not work well so this would have to be a rare occurence.
>
> Let me bring again the
>
> Requirements for extras (humans are needed to test)
> - Don't crash or freeze systems.
> - Don't drain batteries.
> - Are feature complete: everything inside works.
> - Have been tested by someone trusted before.
>
> Crash and freezes are easy to detect. Nitro helps reporting them
> automatically with traces to a server.
>
> About draining batteries we hope to have a tool easy to use for
> betatesters.
>
> Everything inside works is also relatively objective: you go through the
> menu options and you check that there is nothing obvious. Normal and
> minor bugs are ok, happen to everybody.

This is the one that I think is very subjective.  I don't see beta testing 
being about testing menu options (I would hope the developer can do that 
themselves),  I think beta testing is about running something for a period of 
time and not having unexplained or incorrect behaviours.  

Take a real example: the current Beta GPE meets all your criteria.  But, very, 
very occasionally the homepage applet crashes, causing all homepage applets 
to disappear.  This doesn't happen for me and doesn't happen for lots of 
other people.  But for a few people (4 or 5 of the beta testers) it does 
happen -- maybe every few weeks.  

If this app was going to be downloaded by 50 people it probably wouldn't be 
worth worrying about this problem.  But close to 40,000 have downloaded the 
current version of gpe-calendar and if I release this at least a few thousand 
people are likely to hit the problem. So, I need to find and fix this.  

It is finding this sort of problem that I see the task of beta testing being 
about -- not going through the menu options.  Ultimately, you are going to 
have to rely on the developer to make the judgement call on whether the bug 
is actually serious or not.

Graham