[maemo-developers] extras: autobuilders

[From: Neil Jerram]

Graham Cobb <g+770 at cobb.uk.net> writes:

> I would add a few things to Andrew's list in the area of dependency management 
> as this is a subject dear to my heart!  GPE consists of 7 user visible 
> applications comprising a total of 30 packages with complex dependencies 
> between them plus some non-GPE dependencies not in the standard Nokia 
> repositories (like libsoup and libsqlite0).  

I'm not an expert on dependencies, but some comments anyway...

> 5) Dependency aware.  At a minimum the build system needs to be dependency 
> aware: for example if an application and a library dependency are both 
> updated, build the dependency first and then the application.

Doesn't this just happen anyway?  According to my understanding...

- If the library is a build-dependency of the application, and the
  right version of the library hasn't been uploaded yet, the attempt
  to build the application will bail out immediately.  On the next
  build timer pop, if the library has then been uploaded, the
  application build will proceed.

  (Perhaps uploading the library could cause the application's build
  timer to pop early, but that's well into icing-on-the-cake
  territory, not a basic need.)

- If the library is only a runtime dependency of the application, it
  doesn't matter what order they are built in.  But the auto-builder
  should only publish the application into the repository once all its
  runtime dependencies are already in the repository.

> 6) Dependency checking.  One advantage of my current GPE build system is that 
> every build starts in a clean environment.  I know that the build doesn't 
> depend on anything not listed as a dependency because it would fail if it 
> did.  This was one of the first problems I came across when I first started 
> trying to build Maemo packages: I used a single scratchbox environment for 
> all my development and lots of things had been installed over time: builds 
> would be successful even though the list of build dependencies was incomplete 
> and the resulting packages would have unexpected dependencies.  So, the 
> requirement would be to do builds in a clean environment where only the 
> specified dependencies are loaded.
>
> There is an argument that says that dependency checking isn't the job of the 
> autobuilder: its job is just to build and developers should be checking their 
> dependencies themselves.  But doing this would improve the quality of the 
> packages (although not necessarily user-visible quality).

So, to be concrete, I think this means that the building algorithm has
to be something like this:

build(package)
{
  for (each of package's build-dependencies)
  {
    build(dependency)
    dpkg -i dependency-package.deb
  }
  dpkg-buildpackage(package)
}

main(uploaded_package)
{
  start_from_base_tablet_os()
  build(uploaded_package)
  uninstall_all_installed_packages()
}

> 8) Although I don't think anyone is suggesting that the autobuilder should be 
> an autotester, it could at least install the generated packages into an 
> environment that also has everything else already installed.  This would 
> allow conflicts to be discovered; including conflicts that may never be 
> noticed by the development community using extras-devel because no one has 
> just the right combination installed.  This could be a separate process I 
> suppose: a tool which runs every night and just installs everything it finds 
> in extras-devel into a clean scratchbox environment.

I like how this contrasts with the build algorithm: each uploaded
package is built in its own clean environment, but test-installed in
an "everything" environment.

What kind of conflicts are possible, though?  One that I can think of
is:

- library version 1.1 is built, and published to the repository

- application1 is built, with Depends: library = 1.1; build is
  successful, application1 is published, and the test-install passes

- application2 needs library >= 1.2, so library 1.2 is uploaded and
  built; does the system somehow know, at this point, that
  application1 is now broken?

Is this a valid problem, and/or are there other kinds of conflict?

Regards,
        Neil