[maemo-developers] Testing marathon & Q&A Feedback

[From: Ville Reijonen]

Hi,

just a bit more fuel on the fire :)

> Just to chime in, thanks to all the folks taking the trouble to test apps they
> might personally not even be interested in.

Is the approval/karma process going to be actually a popularity contest? 
Popular titles get votes fast, a niche software will not. Unless there 
is regular testing marathons, I think this will be an issue.


> We need more people doing this, so the effort could be split (e.g. 
> tester group A checks one app while tester group B checks another).

One can not test everything on virtual machine. For good testing, one 
should have a device with a default setup so the effects can be observed 
- ie. energy usage, system configuration changes, compability, etc. Good 
testing takes quite a lot of time.

Additionally, I would think that one does not want to just put any 
packages for testing on personal device with personal data. You might 
accept the risk for software you like (and trust for some reason), but 
not for all random packages -> popularity contest. As comparison, 
nothing should never be tested on a server in production.

Who could have more than a one device, so one could be used for testing? 
Probably only somebody who is developing on the device for a company. 
I'm not, I could assume that there is many other hobbyists too..


> - Does this bring karma ? I sure think serious testing is worth a LOT and do
> not see that adequately reflected in karma. Could also be motivating in
> bringing more testers to the table. Ways/ideas of making this abuse-proof are
> welcome.

Good QA is worth a lot. The testing karma system at worst is just a gate 
where the entrace criteria is "looks good or I want the new version". 
Why the testing karma is even labeled as karma, when people have karma 
too.. confusing. Should be acceptance points, manna, or something else.

Maybe this manna/karma thing has been though out, but somehow if feel 
that the research for similar systems was not done before rolling it 
out. There seems to be too many holes, and I just though a while. Every 
serious linux distribution has some kind of QA system. Most of the 
software makers have QA systems. Do not invent the wheel again..


There is no separate queue for the security fixes as mentioned elsewhere 
in the thread. How is the security process run anyways?
-> learn and imitate debian/suse/redhat..

How and what the person tested before giving thumb?
-> Checklist which has to be filled before vote, all QA checklist points 
has to be checked before acceptance, some by recognized persons.

How to motivate to do the testing? How are the testers rewarded?
-> Karma, priviledges, titles (devel -> tester -> senior tester->...)

The testing and package history is not visible, just current comments. 
There should be a combined view with bugs etc.
-> Hmm, Launchpad or some other system?

Looking current queue first page, "libeet1" in queue - any takers? How 
one does test a library? There is just procedure for application. A 
library could be a can of worms.. or the next version could be..
-> What are others doing..

Popular packages are voted more than others - popularity contest.
-> Each application area such as command line, tools, games etc. should 
have their own group of people who take care of them.

Until there is testers groups, upload a package, wait 10 days and
register 10 accounts and upvote.. but luckily it going just to extras :)
-> Upvotes should be accepted from known persons only. Otherwise there 
might be trust issues. Testing is already a priviledge. Nokia could 
support testers, for example with personal device and testing device.

-- 
VRe :: http://iki.fi/vre :: +358 40 5775 456