[maemo-developers] extras: autobuilders

[From: Graham Cobb]

On Friday 09 November 2007 09:11:16 Andrew Flegg wrote:
> On Nov 9, 2007 8:34 AM, Mikhail Sobolev <mss at mawhrin.net> wrote:
> > Let's create a separate thread about autobuilders.
> >
> > A simple question: what would you expect from an autobuilders?
>
> 1) Simple ability to upload a source package(s).
> 2) Have that source package compiled on as many SDKs as possible (OS 2006,
> OS 2007 and OS 2008 should be possible with some/most packages).
> 3) The ability to update a source package in a simple manner (triggering a
>    recompile).
> 4) When a new SDK is released, everything should be recompiled and
> redeployed to extras-{devel,testing,...}.
>
> Ideally, anything which would appear in the Application Manager (ie:
> Section: user/...) should also be updated/auto-created in the
> downloads catalogue; but this is a nice to have.

I would add a few things to Andrew's list in the area of dependency management 
as this is a subject dear to my heart!  GPE consists of 7 user visible 
applications comprising a total of 30 packages with complex dependencies 
between them plus some non-GPE dependencies not in the standard Nokia 
repositories (like libsoup and libsqlite0).  

5) Dependency aware.  At a minimum the build system needs to be dependency 
aware: for example if an application and a library dependency are both 
updated, build the dependency first and then the application.

6) Dependency checking.  One advantage of my current GPE build system is that 
every build starts in a clean environment.  I know that the build doesn't 
depend on anything not listed as a dependency because it would fail if it 
did.  This was one of the first problems I came across when I first started 
trying to build Maemo packages: I used a single scratchbox environment for 
all my development and lots of things had been installed over time: builds 
would be successful even though the list of build dependencies was incomplete 
and the resulting packages would have unexpected dependencies.  So, the 
requirement would be to do builds in a clean environment where only the 
specified dependencies are loaded.

There is an argument that says that dependency checking isn't the job of the 
autobuilder: its job is just to build and developers should be checking their 
dependencies themselves.  But doing this would improve the quality of the 
packages (although not necessarily user-visible quality).

7) Forced rebuilds.  One big problem is with libraries which are not actually 
version-controlled.  This is a fairly small problem with GPE today but there 
are definitely other projects (OpenSync is certainly one) which consider 
themselves in to be in an Alpha phase and don't want to start versioning 
libraries until the APIs are more stable.  This means that there are cases 
where a dependency is updated where all the dependent packages need to be 
rebuilt.  

Of course, this does not only apply in cases where library versioning is not 
in use.  Even in cases where libraries are correctly versioned there may be 
some benefit (e.g. a performance benefit) in rebuilding a dependent 
application.  

What this is really saying is that there needs to be a way to force rebuilding 
a package even though it has not changed.  And it would be useful to have 
that also allow a shorthand of "all packages dependent on package foo".

8) Although I don't think anyone is suggesting that the autobuilder should be 
an autotester, it could at least install the generated packages into an 
environment that also has everything else already installed.  This would 
allow conflicts to be discovered; including conflicts that may never be 
noticed by the development community using extras-devel because no one has 
just the right combination installed.  This could be a separate process I 
suppose: a tool which runs every night and just installs everything it finds 
in extras-devel into a clean scratchbox environment.

Graham