[maemo-developers] Testing marathon & Q&A Feedback

[From: Valerio Valerio]

HI,

some good points, but this isn't easy as it seems, some of the
suggestions here involve a lot of resources that maemo.org can't
provide right now, in my opinion, lets try to concentrate in the basic
improvements, following the actual testing criteria and adopting some
improvements.

On Sun, Nov 1, 2009 at 7:46 PM, Ville Reijonen <vilre at cs.tut.fi> wrote:
>
> Hi,
>
> just a bit more fuel on the fire :)
>
>> Just to chime in, thanks to all the folks taking the trouble to test apps they
>> might personally not even be interested in.
>
> Is the approval/karma process going to be actually a popularity contest?
> Popular titles get votes fast, a niche software will not. Unless there
> is regular testing marathons, I think this will be an issue.

Of course, I expect that people vote for the applications they use.
One idea already suggested in the past is the creation of testing
teams as you mention below.

>
>
>> We need more people doing this, so the effort could be split (e.g.
>> tester group A checks one app while tester group B checks another).
>
> One can not test everything on virtual machine. For good testing, one
> should have a device with a default setup so the effects can be observed
> - ie. energy usage, system configuration changes, compability, etc. Good
> testing takes quite a lot of time.
>
> Additionally, I would think that one does not want to just put any
> packages for testing on personal device with personal data. You might
> accept the risk for software you like (and trust for some reason), but
> not for all random packages -> popularity contest. As comparison,
> nothing should never be tested on a server in production.
>
> Who could have more than a one device, so one could be used for testing?
> Probably only somebody who is developing on the device for a company.
> I'm not, I could assume that there is many other hobbyists too..
>
>
>> - Does this bring karma ? I sure think serious testing is worth a LOT and do
>> not see that adequately reflected in karma. Could also be motivating in
>> bringing more testers to the table. Ways/ideas of making this abuse-proof are
>> welcome.
>
> Good QA is worth a lot. The testing karma system at worst is just a gate
> where the entrace criteria is "looks good or I want the new version".
> Why the testing karma is even labeled as karma, when people have karma
> too.. confusing. Should be acceptance points, manna, or something else.

You get karma for commenting and rating the applications, like in
other 'divisions' of maemo.org (Applications comments, news,
brainstorming,..), it's a form of reward for your collaboration, of
course it isn't perfect and can be abused.

>
> Maybe this manna/karma thing has been though out, but somehow if feel
> that the research for similar systems was not done before rolling it
> out. There seems to be too many holes, and I just though a while. Every
> serious linux distribution has some kind of QA system. Most of the
> software makers have QA systems. Do not invent the wheel again..

Well, if you can suggest a QA system already implemented that fit our
needs, we're listen.

>
>
> There is no separate queue for the security fixes as mentioned elsewhere
> in the thread. How is the security process run anyways?
> -> learn and imitate debian/suse/redhat..

That's one of the miss feature of the package interface among others.

>
> How and what the person tested before giving thumb?
> -> Checklist which has to be filled before vote, all QA checklist points
> has to be checked before acceptance, some by recognized persons.
>
> How to motivate to do the testing? How are the testers rewarded?
> -> Karma, priviledges, titles (devel -> tester -> senior tester->...)

Karma is already there ;).
>
> The testing and package history is not visible, just current comments.
> There should be a combined view with bugs etc.
> -> Hmm, Launchpad or some other system?

Another miss feature...

>
> Looking current queue first page, "libeet1" in queue - any takers? How
> one does test a library? There is just procedure for application. A
> library could be a can of worms.. or the next version could be..
> -> What are others doing..

Errr, wrong queue, try this one:
http://maemo.org/packages/repository/qa/fremantle_extras-testing/
>
> Popular packages are voted more than others - popularity contest.
> -> Each application area such as command line, tools, games etc. should
> have their own group of people who take care of them.

Teams of testers, right.

>
> Until there is testers groups, upload a package, wait 10 days and
> register 10 accounts and upvote.. but luckily it going just to extras :)
> -> Upvotes should be accepted from known persons only. Otherwise there
> might be trust issues. Testing is already a priviledge. Nokia could
> support testers, for example with personal device and testing device.

Not that easy to trick the system, our community is very active, and
believe me, we're not sleeping here.


Best regards,

-- 
Valério Valério
Maemo Community Council Chair

http://www.valeriovalerio.org



>
> --
> VRe :: http://iki.fi/vre :: +358 40 5775 456
> _______________________________________________
> maemo-developers mailing list
> maemo-developers at maemo.org
> https://lists.maemo.org/mailman/listinfo/maemo-developers
>