[maemo-developers] sbox2 & maemo

[From: Marius Vollmer]

"ext Lauri Leukkunen" <lle at rahina.org> writes:

> Continuing on the topic of build dependencies, SB2 has an open
> design issue regarding them. [...]

I played a bit with sbox2 and I failed in the end, probably because I
didn't use the right toolchain, but while doing that, a picture slowly
formed in my head how I would like things to work.

I don't like sbox1 because it redirects a lot of things into devkits,
which are essentially untouchable and hard to update.

Sbox2 is better since it redirects into the host system, which is much
easier to maintain than the devkits.  But it still redirects from one
system to another.

Also, both sbox1 and sbox2 redirect whole packages, not just binaries
(if I understand sbox2 correctly).  E.g., when redirecting the "perl"
command, the redirected-to perl will take its modules from the
redirected-to prefix, not from the redirected-from prefix.

There are two reasons for redirecting: to avoid emulating the compiler
and other build tools since that is quite slow, and for bootstrapping
where you start with an empty root directory but still have all the
tools at hand to fill it by compiling packages from source.  I am not
interested in bootstrapping from source here since that is not done
very often in a binary distribution.

To speed up build tools, it would be much better to only redirect the
machine code files, and nothing else.


I made a small experiment which should explain better what I have in
mind.

I made a sid armel chroot with debootstrap, copied a x86 qemu-arm into
it together will all the needed libraries, and registered qemu-arm as
a binfmt_misc interpreter.  Then I could chroot into the sid armel
chroot and after some more tweaking I had a system that could compile
Debian source packages.  Every executable was emulated, sh, perl, the
compiler, everything.  That setup was clean but slow, about ten times
slower than compiling natively.

The setup was clean because everything in it was coming from one
Debian distribution.  Apt-get is able to keep everything up to date,
except qemu.

It was slow because every single executable was emulated.  I started
speeding it up by replacing selected executables with x86 versions.
For example, I copied /usr/bin/perl into the chroot, and a matching
POSIX.so.  I left all the interpreted Perl code alone, since the point
was to speed up the execution of the perl binary, not to redirect to
an entirly different version of perl.

This worked nicely and sped up the compilation of my test package by
one minute or so.

Replacing binaries like this breaks the cleanliness of the setup of
course, and needs to be done in a better maintainable way.  I would do
it this way: In addition to the perl-base package which contains the
normal armel binaries there is also a perl-base-accel-i386 package in
the armel architecture that contains the binaries of the perl-base
package, but as i386 machine code and in different locations so that
both perl-base and perl-base-accel-i386 can be installed at the same
time.

Then there would be a way for the user to select whether or not to
make /usr/bin/perl be the armel or i386 version.

The perl-base and perl-base-accel-i386 packages would be produced by
the same maintainer, ideally, and would be forced to be nicely in sync
by package dependencies.  Building the perl-base-accel-i386 package
could be done by fetching the i386 perl-base package and hand picking
the needed binaries.  This all happens within one distribution, so
this is easy to coordinate.

That way, all binaries can be replaced that are used a lot during
compilation and are too slow to be emulated: /bin/sh, m4, awk, make,
perl, gcc, binutils, but not much more.  Packages like autoconf,
automake, libtool are never redirected since they are interpreted
anyway.

Documentation tools like the monster docbook machinery might not even
be needed on armel since documentation is architecture independent and
build on i386.  All documentation tools should of course be available
for armel, and the only consequence of not having a -accel-i386
package is that things are a bit slow.


Assuming that this all works out, people will want to maintain
multiple chroots, and there needs to be a way to mirror the home
directory into these chroots, etc.  In the end, it might look very
much like sbox1 except the horrible redirection into devkits is gone
and the chroots are properly self-contained and upgradeable with
apt-get from a single source.


Soooo, what do you think?  Am I on crack again?