[maemo-developers] QA process = bug fixing disincentive?

[From: Henrik Hedberg]

Igor.Stoppa at nokia.com wrote:

> I think the problem here is that some braindead system has been introduced,
> which doesn't account for the actual work being done.

    And what is the biggest mistake here is that the new system has been 
put into production before testing it at all.

    Someone just came up with an idea of crowd sourcing the QA and used 
random generator to set the parameters. Soon it was in real use before 
any experiences of its functionaly had got.

    I would have understood if the parameters had been really low (say, 
1 day and 1 thumbs up) at the beginning, or there would have been a 
separate repository (say, extras-selection) to test this new 
functionality. The parameters could have been changed according to 
success of the system and in parallel with the amount of testers (and 
devices available).

    The sad contradiction here is that developers are expected to 
produce high quality outcomes and their results are put under QA, put 
the maemo.org processes and tools are not! Those should have been 
tested, for example, with a smaller group of developers before putting 
into production system.

    The maemo-extras testing marathon did amazing job. Thanks to 
everybody who participated! However, it cannot be a permanent way to 
overcome one of the biggest problems with the new system.

    I am not totally against the new system, and I do recognize the 
importance of quality assurance. I just hope that we can learn from the 
past and react rapidly. Please, do not refer to the better future and 
the possibility to have more users and testers later. Things should work 
now!

    Here are my suggestions now:

1) Fine tune the parameters: say, 5 days and 5 votes. These can be 
changed later when the system is working (has enough testers).

2) Change the system so that user packages that are depended by another 
user package are promoted automatically when the actual user package is 
promoted (like non-user packages are promoted currently). For example, 
when an user is testing Mauku, she is implicitely testing also the 
microfeed package [1]).

3) Find a way to overcome the limitations when an upgraded package is an 
important security fix.

    And here are my older suggestions [2], which, I think, are still valid:

* Negative karma can be given _only_ if it based on the agreed QA 
requirements. (Testers are still giving karma based on their subjective 
thinking instead of QA requirements.)

* The package page should have a link to a bug tracker and the link must 
be used! (Comments are stored into a wrong place currently. It is double 
effort for a developer to track the packages interface in addition to 
bugs.maemo.org.)

* Negative karma can be given _only_ with a link to a bug tracker having 
a bug report about the show stopper. It may be either a new bug report 
written by the tester or an old open bug report just referred in the 
comment.

* Negative karma is automatically removed when the related bug report is 
closed (fixed or other way resolved). (Currently, there is no way to 
remove the negative karma (thumbs down) if the bug is fixed. Please, 
note that the bug may be in the library code, and the bug in the package 
is thus actually fixed when the library is fixed. So, there would not be 
any need to update the application package every time.)

    BR,

    Henrik

    P.S. I do not necessarily see that more testers will make things 
easier and more workable. The more testers there are, the more 
subjective evaluations we will get. If one tester just do not like the 
graphics of an application he may give thumbs down, and even four other 
testers giving thumbs up are needed to fix that misjudgement. I really 
would like to see a discussion about the responsibilities of testers. 
Should there be a mechanism to give negative karma to testers who are 
not following the QA rules, or even way to ban them?

[1] http://talk.maemo.org/showthread.php?p=362575#post362575

[2] 
http://lists.maemo.org/pipermail/maemo-developers/2009-September/020921.html

-- 
    Henrik Hedberg  -  http://www.henrikhedberg.net/