[maemo-developers] Issues with Python and maemo-optify (was: Re: Maemo Summit /opt BOF notes...)

[From: Anderson Lizardo]

Hi,

On Sun, Oct 11, 2009 at 7:44 AM, Tim Samoff <samoff at gmail.com> wrote:
> Hi,
>
> Using /opt: Why, How & For How Long?
>
> What are the issues?
>
> • Standards.
> ◦ What about introducing a new 'mobile' Linux standard.
> • Repackaging.
> • Differences in flash performance.

I'd like to raise one very important issue detected during our
experiments with maemo-optifier and Python applications (including
python's core packages, such as python2.5 and python2.5-minimal).

Python in various places relies on paths returned by calls to
readlink(). As some of you might know, the readlink() syscall
"dereferences" symbolic links.

One serious side-effect of optifying python2.5-minimal package is that:

* /opt directories are added to sys.path
* Any modules under /usr/lib/python2.5/site-packages are NOT found.
* Byte-compiled files are put on /opt/maemo/ instead of the usual
/usr/lib/... paths

This happens because Python uses the full dereferenced path to
/usr/bin/python2.5 (as returned by readlink()) to detect the module
path. Therefore it only sees the modules under /opt/maemo, not the
ones under /usr/lib/... (unless you explicitely set PYTHONPATH)

Another issue is that even if python2.5 is not optified, any Python
application which has its main script optified will not be able to
import its own modules. See this experiment (running on a desktop):

$ echo 'import testopt' > main.py
$ echo 'print "hello"' > testopt.py
$ python main.py
hello

The test case works just fine. Now let's "optify" it:

$ rm testopt.pyc
$ mkdir opt
$ mv main.py opt/
$ ln -s opt/main.py .
$ python main.py
Traceback (most recent call last):
  File "main.py", line 1, in <module>
    import testopt
ImportError: No module named testopt
$ ls -lR
.:
total 8
lrwxrwxrwx 1 andgomes andgomes   11 2009-10-13 16:05 main.py -> opt/main.py
drwxr-xr-x 2 andgomes andgomes 4096 2009-10-13 16:06 opt
-rw-r--r-- 1 andgomes andgomes   14 2009-10-13 16:03 testopt.py

./opt:
total 4
-rw-r--r-- 1 andgomes andgomes 15 2009-10-13 16:06 main.py

As you can see, the "testopt" module was not found even though it is
on the same directory as the main.py symlink. This happens because
Python's notion of "current directory" is based on the value returned
by the readlink("main.py", ...) syscall, which in this case is
/tmp/opt/main.py. Therefore, python2.5 looks into /tmp/opt/ for the
testopt.py, but does not find it.

Of course, we can modify python2.5 so that it does not call readlink()
on the main script, but for sure it was put there for some reason, and
can create even more bugs. Besides, it can cause some subtle bugs,
such as modules with the same name but in different directories being
imported by mistake, and causing random exceptions.

Therefore, we would like to ask the community how to better approach
this problem. For sure the PyMaemo packages as is occupy too much
space to be entirely on the root partition of the current size, so we
need a solution to reduce the baseline Python size. Here are the
options I see (in no specific order):

1) hack Python to not rely on paths returned by readlink()
2) Install Python properly under /opt by modifying debian rules (e.g.
using --prefix etc.) and do not run maemo-optify for any python
applications.
3) Work hard to reduce Python footprint

Some remarks of my own:

1) looks dangerous, can create even more bugs than fixing them
2) Should work just fine, as long as ALL Python applications/modules
also install under /opt. Note that creating symlinks back to /usr/...
will NOT work, for the same reasons as maemo-optify currently does not
work for Python. Has implications for all those Python packages, and
will need special handling to make packages work for earlier Maemo
releases.
3) Looks the best solution IMHO, but takes time and is not guaranteed
to have big gains.

Finally, I really like (3) best, but I'm afraid it is not a short term
solution. We will do it anyway in the following releases, and we be
believe we can achieve great gains (although it is hard to tell
whether this good is "good enough").

Comments, suggestions are welcome!

PS: we are about to make a release candidate for PyMaemo and we would
love to see it already installed under /opt to reduce root partition
footprint in short term, but given the various problems we had running
it locally, we will probably have to postpone it :( Suggestions on how
to better approach it are welcome!

Regards,
-- 
Anderson Lizardo
OpenBossa Labs - INdT
Manaus - Brazil