[maemo-developers] Maemo-mapper vector data backend follow-up

[From: Simon Pickering]

Hi all,

A few of us sat down with John Costigan (aka Gnuite) after his presentation
at the Maemo Summit and as well as drinking a few beers/vodkas discussed how
to achieve the end goal. I just thought I'd start the conversation going
here and summarise my thoughts (John has probably not even made it home
yet!).

For those who didn't see the presentation, it was basically asking for input
on how to use vector data as a backend to provide tiles and routing
information, without needing to completely re-write Maemo-mapper and change
the way it handles data.

The idea that John presented was that we should store the vector data
locally and act like the OSM/Google server and serve Maemo-mapper with image
tiles and route GPX data in just the same way as already happens. The
advantage here is that the two projects are pretty well separated (though
John said he'd be happy to make alterations to Maemo-mapper if needs be).

The other positive aspect of this split is that if someone were to reverse
engineer the TeleAtlas/Navteq storage format, which might be slightly
controversial, it wouldn't impact on Maemo-mapper or any other backend (e.g.
OSM).

So, considering OpenStreetMap as our vector data set (Garmin is another
option as I believe there's an open source library which can understand
their database format; likewise for British University users Ordnance Survey
data is available; and for those in the US there is Tiger census data;
world-wide NASA Shuttle Radar Topography Mission contour data; i.e. there's
at least a few sets of data which could be used), these are my initial
thoughts.

The OSM data is stored in a MySQL data base (though the data we download as
*.osm is an XML dump I think); there is an API to query the database using
XML. As we'd already be running as a server, and would gain no advantage in
using the API should the database format change (as we access the database
directly, so need to implement the db-side of the API), I would propose we
ignore that, and act on the data directly, should we decide that it doesn't
need pre-processing. The only disadvantage to this, it that if we
implemented the API, we could also work online and talk to the main OSM
database directly. Not sure how much use/interest there would be in that.
Thoughts?

So, we have some data, now we have to be able to do something with it. There
are a number of programs which can render OSM data that we should look at
adapting rather than writing it all from scratch. 

Tile rendering
==============

There's a list of OSM rendering programs here [0].

Mapnik [1] is the program which renders the map tiles currently downloaded
by Maemo-mapper. This requires that the OSM MySQL database be processed and
turned into a PostGIS database. It doesn't look like this is something that
could be done on-device due to the time and memory requirements. PostGIS
rendering is probably a good thing as other map data is stored in this
format. This is supposed to be difficult to setup though, and I don't know
the cpu/memory requirements.

Other applications such as Navit [2], Gosmore [3] also require that the
database be processed to turn it into something other than MySQL. Pyrender
[4] might be a possibility, though it is designed to work from a tile server
(where a tile is a tile of metadata rather than image data) rather than the
MySQL database.

I'm tempted to quickly pull together something using pyrender (it currently
parses XML directly afaik, so will need some tweaking to read from MySQL if
that's the backend of choice). Using a proper db rather than parsing the XML
should hopefully allow it to scale better than it is supposed to do atm.

Obviously C will be faster, but perhaps more people will get involved if
it's Python...? Any thoughts on this?


Navigation/routing
==================

Again, there are a number of programs which do this [5]. Again Gosmore [3]
and Navit [2] and another Python program, Pyroute [6], are on the list. I
don't know much about these, how processor intensive they are or how fast
they are. Does anyone have any ideas?

Pyroute appears to use raw OSM vector data (though not as a MySQL database
as that's not a dep, unless that's assumed for a Python install?); Navit and
Gosmore both use their own converted data formats.

Again I'm tempted to try using Pyroute with a MySQL db. Same arguments as
above.


So to summarise, it looks to be possible. Let discussion commence!

Cheers,


Simon


[0] http://wiki.openstreetmap.org/index.php/Rendering
[1] http://wiki.openstreetmap.org/index.php/Mapnik
[2] http://wiki.navit-project.org/index.php/OpenStreetMaps
[3] http://wiki.openstreetmap.org/index.php/Gosmore
[4] http://wiki.openstreetmap.org/index.php/Pyrender
[5] http://wiki.openstreetmap.org/index.php/Routing
[6] http://wiki.openstreetmap.org/index.php/Pyroute

P.S. Rana is a combination of Pyroute and Pyrender, search for it on the osm
wiki.