[maemo-developers] Qt and Multimedia: the big mess

[From: Benno Senoner]

Buongiorno Andrea,

I agree with you, currently the situation is a bit chaotic, but the
move Qt MultiMedia out of the Qt library
and put it into Qt Mobility was the right move to do since on certain
architectures like symbian you have some API differences
amongst OS versions and combined with the Symbian Qt Smart Installer
(which checks the missing dependencies of a Qt app installs Qt+Qt
Mobility
on the device if they are not already present) I find this a good solution.
For example according to some Nokia engineers in order to achieve low
audio latency you have to resort to proprietary symbian APIs which are
not present
on all phone models. Therefore the smart installer could download the
right version of Qt Mobility which best utilizes all the features of
the phone.
For example if the phone OS comes with the proprietary low latency
audio API then Qt Mobility can use it, otherwise resort to the
standard audio API which
provides higher latency, which far from optimal, at least would make
the app compatible with all phone models.

In general the QAudioInput/QAudioOutput in it's current state is a
mess regarding low latency. It is trumpeted as the API all developers
whould use so that
you can write crossplatform code without messing with the low level
audio API details of each platform (symbian,maemo,meego etc).

But the current state is still bad:
- the ALSA backend is not optimized for low latency and I have the
feeling that the guy that wrote it has in some way misinterpreted the
ALSA API because the backend
 calls several redundand ALSA functions (set buffer size and set
buffer time) which is redundant and might not lead to the desired
buffer size.

- the symbian backend still does not support low latency, a Nokia
engineer wrote on qt-s60-feedback mailing list that he will fix the
issue and that it will appear in the next qt 4.6.x release
  (Qt Mobility I assume)
to date, looking at the repository QAudioOutput::setBufferSize() is
still a stub and does nothing
http://qt.gitorious.org/qt-mobility/contacts/blobs/master/src/multimedia/audio/qaudiooutput_symbian_p.cpp#line242

- the Maemo backend of QAudioInput experiences very high latencies a
few seconds in my case. I had to resort to using pulseaudio and while
it works better it is still not optimal
and loses data when the latency gets too low ( eg <50msec).
Fortunately the application in question did not have to record all the
audio otherwise there would be crackles and clicks in the stream.

- using pulseaudio in Maemo and in Meego is very bad since pulseaudio
was not designed with real time and low latency robustness in mind.
Instead of Nokia tapping into the linux audio talent and it's members
http://www.linuxaudio.org and using and extending the JACK audio
server ( http://www.jackaudio.org ), they did chose
to go with an amateurish project like pulseaudio and now want to use
it in Meego too.
That is simply insane ! The wheel was already invented (jack is
multiprocessing capable and provides single digit latency) but Nokia
did chose to adapt a broken wheel )to their needs.

As promised I will contribute detailed performance numbers and latency
testing tools to ensure that Qt Audio I/O will perform adequately.

I think Meego has a great future, especially now that intel jumped on
board too but with regard to audio, as long it uses the current form
of pulseaudio it will always experience problems
and not provide the same smooth audio experience as the iphone for example.

I think Qt Audio I/O is a good idea but the current backends are all
problematic, some because of incorrect implementation (eg ALSA), some
because the underliying audio server is
broken (pulse audio). And then there is still the whole issue about
all processes and callbacks involved into low latency audio processing
needing to be scheduled with high priority
(SCHED_FIFO under Linux) or like jack does very elegantly, where jack
takes care of the scheduling priorities while the programmer only
needs to write the callback.
The most simple and performant programming model.
LinuxSampler using JACK audio output achieves the same low latencies
and stability as using ALSA directly.
This is what QAudioInput/Output should aim for.

cheers,
Benno
The LinuxSampler project
http://www.linuxsampler.org





2010/7/13 Andrea Grandi <a.grandi at gmail.com>:
> Hi all,
>
> let's make the point about Multimedia libraries available in Qt framework.
> We have: QtMultimedia, QtMobility-Multimedia, Phonon, QSound and
> QtMultimediaKit is going to be released. Did I miss something?
>
> At the moment I'm writing an application that uses
> QtMobility-Multimedia and I had to recompile this library for Qt 4.6.3
> Desktop to be able to test my application for Desktop target (it takes
> to long to test it directly on N900 and it's not a good idea if I've
> just to test networking, xml parsing ecc...).
>
> Why doesn't Nokia try to unify all these multimedia libraries or at
> least try to make them API-consistent? Even from a QtMobility release
> to another there are differences... for example this is what I've to
> do to be able to compile my project on both Desktop and Maemo target:
>
>  // hack to make things work with maemo qt-mobility version
>    #if defined(Q_WS_MAEMO_5)
>        playlist->setMediaObject(player);
>    #else
>        player->setPlaylist(playlist);
>    #endif
>
> Please let me know what do you think about it.
>
> Regards,
>
> --
> Andrea Grandi
> email: a.grandi [AT] gmail [DOT] com
> website: http://www.andreagrandi.it
> PGP Key: http://www.andreagrandi.it/pgp_key.asc
> _______________________________________________
> maemo-developers mailing list
> maemo-developers at maemo.org
> https://lists.maemo.org/mailman/listinfo/maemo-developers
>