[maemo-developers] Maemo-Optify & Builder Bots = Broken?

[From: Nathan Anderson]

Ed,

	I have actually installed all the packages I have built and
submitted to extras _inside_ my scratchbox from extras(-devel/testing)
without any issues; in fact I needed install a dozen or so packages (some
mine, some not) to make up all the dependancies on the one I was actually
having the issues with and reported.    This is the first package I've built
that has relied upon a optified deb library.   I had to actually "wait" to
package this package up until I could get the optified ICU4.21 into the
extras because I wanted to make sure that my scratchbox environment would
match what the autobuilder would get.   I never did a local install from the
sources, I only did a "apt-get install <package>" to get my newly build
packages back from the extras(-devel/testing) repositories.

	I can also build the source, armel and x86 of this package in my
scratchbox using the dependancies I got from extra(dev/test).   However,
when I submit my nice source.deb to the autobuilder it fails with the /opt
error I listed below.     So even it their was some issue with scratchbox;
there is still an serious issue with the autobuilder since it is not able to
install optified deb packages it built using the "maemo-optify" program.

	Now as to if there is also a issue with scratchbox -- I can't see
how; as I stated above I can (& have) installed the packages that the
autobulder built and stuck into extras-develop for me.   I have no issues
inside my scratchbox.   

	I do not have access to a n900 to test to see what happens on a real
n900.  Anyone who has a n900 want to try "apt-get install clucene-core" (you
will need extras-devel enabled) and see if you get pretty much everything in
/opt/maemo/usr/lib)

	Please remember the autobuilder built and optified all the .deb
packages I am depending on.  If I skipped the maemo-optify step in all my
packages; we would not be having this discussion.  ;-)    

    So right now my choice is to rebuild and turn off the optifing step and
theirby putting about _20_ megs of libraries into the /usr/lib folder or not
do anything until a solution is found on why the autobuilder is having
issues with .deb packages it built and optified.   I'm not particlarly
happy, having to wait, but having these packages optified makes a lot more
sense to me.  ;-)   So I'm currently blocked.  (Please note I am note the
only person who has had this issue.  I know of at least two others as of
this moment.)

	If I do need to change something in my packages, I will!   But I
need to know what.  Afaik, the only "weirdity" to these packages is adding
the "maemo-optify" before the md5 generation step.

Nathan.








-----Original Message-----
From: Ed Bartosh [mailto:bartosh at gmail.com] 
Sent: Wednesday, October 21, 2009 1:26 PM
To: Nathan Anderson
Cc: maemo-developers at maemo.org
Subject: Re: Maemo-Optify & Builder Bots = Broken?

2009/10/21 Nathan Anderson <nathan at andersonsplace.net>:
...
> Selecting previously deselected package clucene-core.
> Unpacking clucene-core (from .../clucene-core_0.9.21b-0maemo1_armel.deb)
...
> dpkg: error processing
> /var/cache/apt/archives/clucene-core_0.9.21b-0maemo1_armel.deb (--unpack):
>  trying to overwrite `/opt', which is also in package base-files
>                      ^^^^
> Uhm, this is not good.   Each of my "optified" packages throws this error.
> That kinda makes optifing the libraries a pointless exercise if I 
> can't use them later in the buildbot.  And Lib-icu is about 16 MEGS of
libraries, so I
> would think it would be a great canidate for optification.     Do I need
to
> resubmit everything w/o optification, or are the build bots broken?
>
Have you tried to reproduce this issue in your build environment by
installing your library and base-files locally inside scratchbox?

Below are my attempts to reproduce and fix the problem:

This is what I see in autobulder environment:
> grep /opt$ /var/lib/dpkg/info/base-files.list
/opt
It means that /opt belongs to base-files package.
And I can easily reproduce the error if I try to install any package with
/opt directory inside it:
 > fakeroot apt-get install clucene-core Reading package lists... Done
Building dependency tree... Done The following NEW packages will be
installed:
  clucene-core
0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
Need to get 319kB of archives.
After unpacking 1085kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  clucene-core
Install these packages without verification [y/N]? y
Get:1 http://osso.stage.dmz fremantle/free clucene-core 0.9.21b-0maemo1
[319kB] Fetched 319kB in 0s (329kB/s)
/scratchbox/tools/bin/sh: line 1: /usr/sbin/dpkg-preconfigure: No such file
or directory Selecting previously deselected package clucene-core.
(Reading database ... 6395 files and directories currently installed.)
Unpacking clucene-core (from .../clucene-core_0.9.21b-0maemo1_armel.deb) ...
dpkg: error processing
/var/cache/apt/archives/clucene-core_0.9.21b-0maemo1_armel.deb
(--unpack):
 trying to overwrite `/opt', which is also in package base-files
dpkg-deb: subprocess paste killed by signal (Broken pipe) Errors were
encountered while processing:
 /var/cache/apt/archives/clucene-core_0.9.21b-0maemo1_armel.deb
E: Sub-process /scratchbox/devkits/debian-etch/bin/dpkg returned an error
code (1)

Hmm. Let's remove /opt from base-files.list and see what happens:
Unpacking clucene-core (from .../clucene-core_0.9.21b-0maemo1_armel.deb) ...
dpkg: error processing
/var/cache/apt/archives/clucene-core_0.9.21b-0maemo1_armel.deb
(--unpack):
 error creating directory `./opt': Permission denied
dpkg-deb: subprocess paste killed by signal (Broken pipe) Errors were
encountered while processing:
 /var/cache/apt/archives/clucene-core_0.9.21b-0maemo1_armel.deb

It didn't help, but we can see that dpkg tries to create directory /opt and
fails because symlink /opt already exists.
OK, then. Let's put /opt back to base-files.list and create directory /opt
instead of symlink:
$ sudo rm /scratchbox/users/ed/opt
$ sudo mkdir /scratchbox/users/ed/opt
$ sudo chown ed /scratchbox/users/ed/opt And try again:
> fakeroot apt-get install clucene-core
Reading package lists... Done
Building dependency tree... Done
The following NEW packages will be installed:
  clucene-core
0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
Need to get 0B/319kB of archives.
After unpacking 1085kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  clucene-core
Install these packages without verification [y/N]? y
/scratchbox/tools/bin/sh: line 1: /usr/sbin/dpkg-preconfigure: No such file
or directory (Reading database ... 6395 files and directories currently
installed.) Unpacking clucene-core (from
.../clucene-core_0.9.21b-0maemo1_armel.deb) ...
Setting up clucene-core (0.9.21b-0maemo1) ...

It works! Hurray! :)

So, my conclusion is this is scratchbox problem, not autobuilder one.
It's not autobuilder, who creates /opt symlinks. They're created by
scratchbox for each user.
May be scratchbox guys can explain how to fix it properly. Of course I can
remove symlinks and create directories instead for all builders, but may be
there is better way to solve the problem?

--
BR,
Ed