[maemo-developers] Issues with Python and maemo-optify (was: Re: Maemo Summit /opt BOF notes...)

[From: Martin Grimme]

Hi,

I'd suggest to not optify Python, but instead install the modules from
/usr/lib/python to /opt, or ZIP it up. I think I've seen it already
zipped up on some distribution, so this would be a way. Or compile the
whole stuff with --prefix=/opt/python, and just put the links python
and python2.5 into /usr/bin/. The site.py module could still be
modified to support legacy modules that use /usr/ prefix.
Optification looks like an ugly hack to me anyway, and the --prefix
option should be used where possible.


Just my 2 ¢

Martin

2009/10/13, Anderson Lizardo <anderson.lizardo at openbossa.org>:
> Hi,
>
> On Sun, Oct 11, 2009 at 7:44 AM, Tim Samoff <samoff at gmail.com> wrote:
>> Hi,
>>
>> Using /opt: Why, How & For How Long?
>>
>> What are the issues?
>>
>> • Standards.
>> ◦ What about introducing a new 'mobile' Linux standard.
>> • Repackaging.
>> • Differences in flash performance.
>
> I'd like to raise one very important issue detected during our
> experiments with maemo-optifier and Python applications (including
> python's core packages, such as python2.5 and python2.5-minimal).
>
> Python in various places relies on paths returned by calls to
> readlink(). As some of you might know, the readlink() syscall
> "dereferences" symbolic links.
>
> One serious side-effect of optifying python2.5-minimal package is that:
>
> * /opt directories are added to sys.path
> * Any modules under /usr/lib/python2.5/site-packages are NOT found.
> * Byte-compiled files are put on /opt/maemo/ instead of the usual
> /usr/lib/... paths
>
> This happens because Python uses the full dereferenced path to
> /usr/bin/python2.5 (as returned by readlink()) to detect the module
> path. Therefore it only sees the modules under /opt/maemo, not the
> ones under /usr/lib/... (unless you explicitely set PYTHONPATH)
>
> Another issue is that even if python2.5 is not optified, any Python
> application which has its main script optified will not be able to
> import its own modules. See this experiment (running on a desktop):
>
> $ echo 'import testopt' > main.py
> $ echo 'print "hello"' > testopt.py
> $ python main.py
> hello
>
> The test case works just fine. Now let's "optify" it:
>
> $ rm testopt.pyc
> $ mkdir opt
> $ mv main.py opt/
> $ ln -s opt/main.py .
> $ python main.py
> Traceback (most recent call last):
>   File "main.py", line 1, in <module>
>     import testopt
> ImportError: No module named testopt
> $ ls -lR
> .:
> total 8
> lrwxrwxrwx 1 andgomes andgomes   11 2009-10-13 16:05 main.py -> opt/main.py
> drwxr-xr-x 2 andgomes andgomes 4096 2009-10-13 16:06 opt
> -rw-r--r-- 1 andgomes andgomes   14 2009-10-13 16:03 testopt.py
>
> ./opt:
> total 4
> -rw-r--r-- 1 andgomes andgomes 15 2009-10-13 16:06 main.py
>
> As you can see, the "testopt" module was not found even though it is
> on the same directory as the main.py symlink. This happens because
> Python's notion of "current directory" is based on the value returned
> by the readlink("main.py", ...) syscall, which in this case is
> /tmp/opt/main.py. Therefore, python2.5 looks into /tmp/opt/ for the
> testopt.py, but does not find it.
>
> Of course, we can modify python2.5 so that it does not call readlink()
> on the main script, but for sure it was put there for some reason, and
> can create even more bugs. Besides, it can cause some subtle bugs,
> such as modules with the same name but in different directories being
> imported by mistake, and causing random exceptions.
>
> Therefore, we would like to ask the community how to better approach
> this problem. For sure the PyMaemo packages as is occupy too much
> space to be entirely on the root partition of the current size, so we
> need a solution to reduce the baseline Python size. Here are the
> options I see (in no specific order):
>
> 1) hack Python to not rely on paths returned by readlink()
> 2) Install Python properly under /opt by modifying debian rules (e.g.
> using --prefix etc.) and do not run maemo-optify for any python
> applications.
> 3) Work hard to reduce Python footprint
>
> Some remarks of my own:
>
> 1) looks dangerous, can create even more bugs than fixing them
> 2) Should work just fine, as long as ALL Python applications/modules
> also install under /opt. Note that creating symlinks back to /usr/...
> will NOT work, for the same reasons as maemo-optify currently does not
> work for Python. Has implications for all those Python packages, and
> will need special handling to make packages work for earlier Maemo
> releases.
> 3) Looks the best solution IMHO, but takes time and is not guaranteed
> to have big gains.
>
> Finally, I really like (3) best, but I'm afraid it is not a short term
> solution. We will do it anyway in the following releases, and we be
> believe we can achieve great gains (although it is hard to tell
> whether this good is "good enough").
>
> Comments, suggestions are welcome!
>
> PS: we are about to make a release candidate for PyMaemo and we would
> love to see it already installed under /opt to reduce root partition
> footprint in short term, but given the various problems we had running
> it locally, we will probably have to postpone it :( Suggestions on how
> to better approach it are welcome!
>
> Regards,
> --
> Anderson Lizardo
> OpenBossa Labs - INdT
> Manaus - Brazil
> _______________________________________________
> maemo-developers mailing list
> maemo-developers at maemo.org
> https://lists.maemo.org/mailman/listinfo/maemo-developers
>