[maemo-developers] maemo-developers Digest, Vol 59, Issue 25

[From: Marius Vollmer]

ext Urho Konttori <urho.konttori at gmail.com> writes:

> Also, the current model of centralized gigantic repository does not
> scale up too well.  Just look at the state of using extras-devel is on
> the current devices (hint: slow pain).

[ Urho, thanks for this opportunity to talk about how we want to make
  package management kick ass again.  You guys might think we have
  arranged this (Urho sits two offices down the corridor from me), but
  we really haven't!
]

I don't think the centralized nature is a problem here.  Our current
processes and implementations might not scale well enough, but getting
rid of a centralized repository will not help much, if at all.

The resources needed to maintain a centralized repository can be quite
easily parallized to make the repository itself scale: there can be many
buildbots, many mirrors, a CDN, many testers, and many maintainers.

Now, the repository infrastructure and the processes around it can suck,
of course, and putting another better designed and maintained repository
into place might be needed.  But that's only better because this other
repository is better by itself; it's not better because we now have two
of them.  Shutting down the original sucky repository would be better
still (but might not be easy, of course).


Distributing the packages over many repositories mostly increases
coordination overhead.  It doesn't help to scale.  HAM still has to deal
with the same number of packages, for example.  (And yes, HAM sucks and
badly needs some love, no argument from me against that.)

For HAM performance, the important variable is the number of available
versions, not the number of repositories.  You can regain performance by
reducing the number of available packages and versions.  Forgetting
about repositories altogether and installing everything from standalone
.debs is one way to do that, but it's not a good way.

I believe there is a better way to make package management delightful:
We only let HAM see a (consistent) subset of all available packages, and
we make it possible to change that subset very efficiently at run-time
(at "UI speed" without the need for any "Updating please wait" progress
bars).

That subset would include only the installed packages plus the ones that
the user is currently interested in.  Discovering packages that the user
might be interested in is done without help from apt, via other means.

We are currently writing code for this.  We don't have all the pieces in
place yet, but we have some:

 - The "x-apt" package in Fremantle extras-devel.  This is Harmattans
   version of apt, repackaged to be installable in parallel to
   Fremantle's version of apt.

   The modifications currently include support for "deb-exec" entries in
   sources.list; this allows you to easily use custom protocols between
   apt and repositories for downloading the Packages file, etc.

   Today I'll add the patch for storing the Maemo specific flags in
   apt's binary cache.  This allows us to do our extra OS meta package
   gymnastics without having to read the Packages files.

   http://maemo.gitorious.org/maemo-pkg/apt/commits/x-apt

 - The "mini-pacman" library (not yet in extras-devel).  This is a
   minimal wrapper of libapt-pkg, with a very simplistic API.  Of
   course, it actually uses x-apt, not the platform apt.

   It is also the experimentation ground for different algorithms that
   should get rid of some of the annoyances of the current HAM: better
   error messages, updating the OS when that helps, more agressively in
   general, etc.

   http://maemo.gitorious.org/maemo-af/mini-pacman/trees/master/src

 - A maemo.org specific 'discovery client'.  It interfaces with
   downloads.maemo.org over a custom protocol for browsing available
   applications.  Right now it passes .install files to HAM for the
   actual installation, and my plan is to hook it up with mini-pacman
   instead and then optimize the hell out of the whole stack.

   (Haven't seen the code yet myself, Daniel Wilms and Niels Breet
   should know more.)

> [...] I really do thing we have started to make our home our prison

Then we should remove the bars and locks.  Tearing down the whole house
and going back up the trees would be overreacting quite a bit, no?