[maemo-developers] QA process for "middleware" (libfoo, python-bar) packages: some ideas

[From: Anderson Lizardo]

Hi,

I'm not sure if there's a better list to send this (or even if it is
better to user talk.maemo.org), so feel free to redirect me to the
write channel.

In lack of a better name for the packages the packages that exist
solely as "enablers" for developing our great GUI applications.

So far, looks like the QA process is gearing towards the user/GUI
applications. For instance, the "thumbs up/down" system is restricted
to applications is user/* categories, and the QA testers are not
instructed to check for possible problems in packages outside this
category.

I believe we can improve on this area: to have safer and optimized
middleware packages, such as libraries, language bindings, data
packages, build tools, and so on. While the user will obviously notice
bugs with on the UI, problems with the middleware (for instance, some
"unknown symbol" bug or a segfault due to unexpected ABI changes) are
likely to be the most serious ones IMHO.

Let me give some examples of such possible bugs:

* A new version of libfoo is uploaded with unexpected ABI (Application
Binary Interface) changes. At the same time some application is
compiled against it and it works fine, so the package (together with
libfoo) is promoted to extras. OTOH, this new version libfoo does not
play well with the existing packages in extras, which will break or
even fail to start

* A new version of python-foo (some binding for libfoo) is uploaded to
extras-devel, but contains a serious flaw that makes it segfault when
calling method xyz(). At the same time, GUI application is uploaded
which depends on that newer version, but does not use xyz() at all. In
this case, the current QA checks would basically allow the application
and python-foo to enter extras, but some future application which
tries to use xyz() will segfault, blocking it from entering extras
until python-foo is fixed.

I have some ideas to improve (and assure) the quality of middleware packages:

* Require (or strongly recommend) *all* packages to have at least some
sort of unit/functional testing. In fact, most packages imported from
Debian has tests, but what usually happens is that they are disabled
so the package can build on scratchbox.
* Have some way to run these tests automatically from the package
sources when uploading packages to the autobuilder.
* Exercise installation of packages (maybe using piuparts? See
http://packages.debian.org/sid/piuparts), if possible on a real
device.
* Test disk-full conditions, and randon errors during package
installation of packages from Extras.
* Promote usage of ABI check tools.

Any other ideas?

Regards,
-- 
Anderson Lizardo
OpenBossa Labs - INdT
Manaus - Brazil