[maemo-developers] How to use extras-testing correctly?

[From: Henrik Hedberg]

    Extras-testing QA is not working as it is implemented now! There are
two main issues:

* Comments are stored into a wrong place. Those belong to Bugzilla! It 
is double effort for a developer to track two different places or to
transfer reports into bug tracking system manually.

* Developers are giving karma based on their subjective thinking instead
of agreed (?) QA requirements.

    Let's analyse karma and comments that Mauku has got:

* Two testers of five either add a new bug report or search the existing 
bug reports before entering a comment. Good for those two, bad for the rest.

* Negative karma given at 2009-09-24 04:52 UTC and related comment 
written at 2009-09-24 04:55 UTC. Is there any real reason to give 
negative karma based on the comment? Tester either wants some new 
features (definitely not based on QA requirements) or some minor user 
interface modifications (not a show stopper).

* All testers report at least one issue that is not actually related to 
the application itself but the underlying library. I understand that it 
is hard for end-user to see the difference, but what happens when the 
library is updated? These issues are fixed, and so is the application 
also, but the negative karma stays there.

    My suggestion here is that:

* Negative karma can be given _only_ if it based on the agreed QA 
requirements.

* The package page should have a link to a bug tracker.

* Negative karma can be given _only_ with a link to a bug tracker having 
a bug report about the show stopper. It may be either a new bug report 
written by the tester or an old open bug report just referred in the 
comment.

* Negative karma is automatically removed when the related bug report is 
closed (fixed or other way resolved).

    Graham Cobb wrote:

> Testers should be testing against the agreed
> requirements only.  The subjective element should be eliminated as
> much as possible - we are using human testers because it is
> impossible to do this testing manually, not because we expect
> different opinions.  We are using more than one tester just so that a
> problem doesn't slip through because one tester missed it, not
> because we expect voting.

    I strongly agree!

> By all means add a comment if you are unhappy with the UI but it
> should get a +1 as long as it doesn't conflict with the requirements.
> Ideally every -1 should require a comment saying which of the QA
> requirements is violated.

    I strongly agree. Actually, I would remove the word "ideally". It is 
a must! And with the relationship to bug tracker I described above.

    BR,

    Henrik

-- 
    Henrik Hedberg  -  http://www.henrikhedberg.net/