[maemo-developers] Low Latency Audio Capture on the N900, QAudioInput BUG: has 5000msec latency, Meego audio future

[From: Benno Senoner]

Hi all,
I'm writing an interactive app on the N900 with PR 1.2. which captures
audio and analyzes it.

I used QAudioInput and while audio capture works, is has an EXTREMELY
high latency, several seconds, (around 3-5secs).

I tried several sample rates, 8000, 44100, 48000 Hz, tried with small
buffer sizes eg 512 samples but QAudioInput always
seems to set the buffer size to about 100msec.
I did take a look at the source of QAudioInput and it confirmed my
findings, QAudioInput::setBufferSize() is not honored and
a fixed buffer size is used instead.
QAudioInput is advertised to provide a low level API to talk directly
to the audio device but in it's present form it's not working as it is
supposed.
And even if 100msec is used as the ALSA buffer size, something fishy
is going on. AFAIK pulseaudio is providing a virtual ALSA device
and it could cause this big delay.
The same code works on Linux where the latency is as QAudioInput sets
it to around 100msecs.
So the problem lies in Maemo/N900.

I'm one of the authors of LinuxSampler, the most widely used open
source audio sampler, http://www.linuxsampler.org
and we can target all common OSes and audio interfaces (we use an
abstraction layer) and we consistently achieve very low latencies
on any platform, including Windows, we are talking about <5-10msec latencies.

So I see no reason why QAudio* cannot achieve those numbers too.  we
can help out with QAudio* by providing advices, bug reports and
working
code implementations.

I think Nokia should take the audio latency issue very seriously since
without low latency a range of apps cannot be implemented and it
diminishes
the value of Nokia's platforms and APIs as a valid multimedia system.

Nokia's losing market share is caused too by things like this, lack of
attention to the detail.
On the Iphone, even the most stupid developer gets this audio
application working with low latency, because the API does what it
advertises and Apple
has paid attention to these things.

I'm an audio developer that programmed on many platforms and embedded
systems but newer experienced something that bad like QAudioInput on
the N900.

So please can you point me to an API on the N900 how I could capture
audio with lower latency ?
since skype runs on the N900, I know it's possible, but the docs on
maemo.org don't clearly say how to do it.

some on maemo irc channel said I should use the gstreamer API. Any
reference what the optimal parameters are ? Some reference code or
pointers to an
existing open source application would be welcome.

Since Nokia's goal is to push Qt as the only API to use because of
it's powerful features and crossplatform capabilities, they should pay
attention to
make it working correctly.

As other professional audio developers said too (ask anyone on the
linux-audio-dev list)  the decision of using pulseaudio on a phone was
very bad since it was not designed with real time in mind.


The most advanced audio server on Linux is currently JACK,
http://www.jackaudio.org/

it has been in use by professional linux audio developers for several
years and IS DESIGNED with low latency in mind.
there are hardware products based on jackd and linuxsampler like the
Lionstracs Mediastation which achieve 5 msec latency running the
software sampler,
audio streaming, video players all in parallel. I contributed to that
keyboard too so I know all the complexities behind the system.
But the fact is that JACK's clean design made this possible.
If we would have chosen to base it on pulseaudio, the product would
have been unsellable.

So it is puzzling why Nokia decides to continue with pulseaudio even in Meego.

Meego is the successor of Maemo so it is natural to inherit features
and code from Maemo, but nowhere is it written in stone that they need
to continue with pulseaudio.

Nokia take a look as JACK's clean design, it was refined over several
iterations, jack2 was developed as a research project at a french
university and is backed by research papers
and many real world apps, and it's even multi processing capable by
partitioning work loads while maintaining excellent low latency. The
Lionstracs keyboard makes use of it
and the quadcore CPU version can process an insane amount ov voices
and audio data in real time. we are talking about around 1000 voices @
24bit 44kHz

So my advice to Nokia, while still in time, cosider switching to JACK
as an audio server for future versions of Meego,

I still not get why big corporations like Nokia don't do their
research to look what's available on the market, why chose an
inferiour product and customize it and put it in their devices
(N900) when there is something better around that is completely free
(LGPL) and is well tested and in use by professional audio users and
hardware.

Sorry for the rants,  and sorry for sounding perhaps a bit arrogant
but I just represent a developer which is frustrated that in 2010
modern hardware cannot provide the application
developer with an easy way to achieve low audio latencies and fast
respose times only due to the companies not doing their homework to
optimize the platform and overlooking  an
already working system (like JACK) and chosing to go with an inferiour product.

juist read the comments on this interview with pulseaudio's developer,
most comments are negative:
http://www.cio.com.au/article/320807/open_source_identity_pulseaudio_creator_lennart_poettering/

I'm sbuscribed to various linux audio mailinglist and other audio
forums and never heard such bad complaints about JACK for example.
This says it all.

I know I'm repetitive, but Nokia made a bad choice by chosing
pulseaudio. I hope they still swtich their audio system in Meego, but
knowing the attitude of big corporations
it will be probably be like that, that since they have chosen to elect
a certain app (pulseaudio) they will not change it and all bugs will
be rooted out.
But if the foundations are not solid, a house is never going to be
stable. Sometimes it makes sense to tear it down and rebuild it from
scratch.

I hope that Meego will become the leading mobile platform of choice
providing great end-user experience and give developers a powerful
platform that is fun easy to develop for.

Those problems like I have experienced on Maemo cannot happen on
Meego. Developers do not tolerate platforms which don't deliver a
smooth user experience with little programming effort.
The mobile world has gotten very competitive and the only way to win
mindshare is providing the things mentioned above, otherwise
developers will switch and the platform will not gain traction..

I hope that the sharing of my experiences, my feedback and opinions ad
a long time audio developer can lead to an improvement of the platform
and help to root out bugs and inefficiencies.

I'm open to any kind of discussion on the matter.

thanks everyone for your advices, tips and contributions,
regards,
Benno
The LinuxSampler project
http://www.linuxsampler.org