[maemo-developers] [maemo-developers] Re: The fixes11 mmap patch doesn't respect data alignment on some architectures

[From: Philip Van Hoof]

Hi there everybody who so far assisted me in trying to get the mmap-for-
camel patches to correctly data aligned. Thanks.

I will repeat the situation so that everybody has a clear idea of what
and why about this E-mail.

During the last weeks, I worked on Camel's summary loading. The summary
which Camel (a framework for E-mail access being used by Evolution and
tinymail) loads is typically used by E-mail clients like Evolution and
tinymail for displaying the summary.

The summary is a view with mostly headers like the "date received", the
"from" and the "subject" headers. Most E-mail clients have this and as
you might think, this is indeed one of the most memory and cpu consuming
ui pieces of a E-mail client.

Therefore I decided to mmap() this information rather than the current
fread implementation being used by Camel. You see, tinymail is targeted
for small devices with very few memory availability. Yet tinymail
promises displaying a lot such headers (more than 10,000). Tinymail can
already do this on a device like the Nokia 770. Even with the fread
implementation. But as a passionate software developer, I want more.

You can read some technical information on the mmap stuff here (and on
the Evolution hackers mailing list, and on other GNOME blogs).

http://pvanhoof.be/blog/index.php/the-camel-mmap-summary-stuff-what-is-this-all-about/

I want to display 150,000 headers. And I'm certain this is possible if I
make sure only the visible headers are really in memory. I know doing
that is possible, and I know the shortest path to getting their (going
from A to C without passing B) is mmap.

That is the reason why I did these Camel patches. Surprise, on a x86
architecture I'm already at B. This is actually working stuff on certain
architectures.

On architectures that require data aligned access to memory, however,
this isn't working. An example of a device using such an architecture is
the Nokia 770 which has an ARM cpu. The Nokia 770 *is* definitely a
target device for tinymail.

I have been trying, with very few experience with ARM and such data
aligned memory access, to get the patch to become data aligned.

I've received some very useful pointers and information. As Freddie
Mercury once said: "I thank you all".

I don't have debugger infrastructure for the Nokia 770, nor do I have an
emulator running where I can actually use a debugger on. Nor do I have
good testing infrastructure myself. So this is making it harder for me
to get it right (I'm thinking the problem is a very small one).

Using all these hints and tips, I cooked this version of the patch
(you'll find an URL below).

I will describe how you can get your hands dirty quickly:

It's possible that evolution-data-server is already migrated to GNOME's
Subversion (this migration is happening these days). Nevertheless should
this anonymous CVS infrastructure still work and is this patch based on
the anonymous CVS repository.

cvs -z3 -d :pserver:anoncvs at anoncvs.gnome.org:/cvs/gnome co evolution-data-server
cd evolution-data-server/camel
patch -p0 < thepatch
cd ..
./autogen.sh --prefix=/opt/camel && make && make install

The most important files are camel-file-utils.c and
camel-folder-summary.c. There's also some ./providers/*/*-summary.c
files that are also reading things from the mmap memory (like some
version integers).

You can get yourself a simple test E-mail client using tinymail:

svn co https://svn.tinymail.org/svn/tinymail/
cd tinymail/trunk
export PKG_CONFIG_PATH=/opt/camel/lib/pkgconfig
./autogen.sh --prefix=/opt/tinymail && make && make install

You can configure an account in tinymail using this simple script:
http://pvanhoof.be/files/tny-account-create.sh

If you go "online" and open a folder, it will create "summary" files in
for example:

$HOME/.tinymail/mail/imap/$hostname/folders/INBOX/summary

It's that file (for your INBOX folder) that will be mmap-ed if you close
that folder and re-open it (or if you restart the application).


This is the original patch that didn't care about data alignment at all.
This patch works:

http://pvanhoof.be/files/camel_folder_summary_with_mmap_fixes11.diff

This is my 3th attempt to get data alignment correct. It's not working
on a Nokia 770. It will (when the loading of the mmap stuff happens) get
killed. On the tty, it will print "Killed".

http://pvanhoof.be/files/camel_folder_summary_with_mmap_fixes11_data_alignment03.diff

The camel_folder_summary_load method is where all this starts.


On Sat, 2006-07-15 at 10:15 +0200, Philip Van Hoof wrote:
> On Fri, 2006-07-14 at 17:15 -0400, Jeffrey Stedfast wrote:
> > ah, you're going to have to pad strings and possibly other stuff.
> 
> The pstrings themselves and the 7bit encoded unsigned 32 integer bytes.
> Since data padding on ARM is 4 bytes (32bit), unsigned int32 will be
> aligned correctly? So the header, references and extra information per
> provider is mostly going to work out fine. But for example the count in
> front of the references, the length-bytes of pstrings, the pstrings
> themselves and two or three other 7bit count bytes aren't always going
> to be correctly aligned I fear.
> 
> I guess this means that the summary file format can't be kept backward
> compatible as I will have to take a look at the unit32 encoder, the
> ##type encoder and the string encoder
> 
> Well, if somebody feels brave, feel free to assist me ;-)
> 
> > On Fri, 2006-07-14 at 19:23 +0200, Philip Van Hoof wrote:
> > > While the x86 handles unaligned access, ARM doesn't. The patch will not
> > > work on architectures that don't handle unaligned access.
> > > 
> > > I will try to fix it, but I don't have a lot experience in this field.
> > > 
-- 
Philip Van Hoof, software developer at x-tend 
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
work: vanhoof at x-tend dot be 
http://www.pvanhoof.be - http://www.x-tend.be