[maemo-developers] Low Latency Audio Capture on the N900, QAudioInput BUG: has 5000msec latency, Meego audio future

[From: Benno Senoner]

regards,
Benno
The LinuxSampler project
http://www.linuxsampler.org

2010/6/16 Ville M. Vainio <vivainio at gmail.com>:

>
> I think you are barking at the wrong tree - pulseaudio should get way
> better latency than 5000ms, otherwise it would be useless for pretty
> much everything.
>
> You should take this up at bugreports.qt.nokia.com => Qt => Multimedia.



Yes, pulse audio should, regardless of it's bad implementation achieve
better numbers than 5000msec latency but this is what I am seeing.
from the user space in QAudioInput.

I will fill a bug report but I want to write a testing app first to
back up my claims, but it takes time do do that.
I'm an engineer and I like numbers and diagrams.
I already did such testing tools in the past and they helped to
optimize the scheduler of the linux kernel by providing stable single
digit millisecond
latency for user space apps.
So it seems like my "barking" helped to some extend.
It's not that my main hobby is to shout and whining on developer
mailing lists only because I want some feature implemented which only
I need.
I just represent a multimedia developer wanting to use the platform
and point out the flaws in order to contribute to improve it.

The fact is that the Iphone makes it transparent and easy for
developers to achieve low latency audio while Nokia has still not
achieved this goal,
The day I can pipe audio in and out of QAudioInput/QAudioOutput with
negligible delay and having it work well on both Symbian and Meego I
will shut up
and take all my complaints back!
But at the moment the situation is quite embarassing.
Since Nokia wants us to use only Qt, in order to make the code work on
all their platforms, eliminating system-dependent code, they have to
provide
the developer with a working implementation. Otherwise as said Iphone
and Android will continue to grow and grow, stealing precious market
share
and developer mindshare to Nokia, and how knows if those developers
will come back one day.


2010/6/16 Xavier Bestel <xavier.bestel at free.fr>:
> The typical use case is only one sound played at the same time. In that
> case the sound server needs to:
> - not do mixing
> - use as few interrupts as possible, i.e. having the biggest acceptable
> latency possible. Otherwise the sound hardware keeps waking up the
> processor to fill a tiny buffer again and again.
>
> I don't know for the first, but I'm pretty sure JACK isn't designed to
> do the last.


A hardware audio driver usually exposes an buffer which is composed by fragments
( eg 2 x 1024 samples = 2048 samples buffer) and after each fragments got played
an interrupt is generated and the user space app can read or write
from/to that fragment
while the other(s) are filled,read by the hardware.
Of course the smaller the buffer size the higher the interrupt
frequency and this
can in turn cause higher latencies.
So if power needs to be conserved then as I see it the only way is to
increase the fragment size
and this usually requires closing and reopening the audio device.
AFAIK JACK does not do that, but it's much easier than making
pulseaudio reealtime-safe, efficient
and SMP capable as JACK is.
download the JACK2 source code and compare it to pulseaudio, JACK is a
a complex piece of software very well designed,
it provides glitchfree graph changes etc, things that are far from
trivial to implement.

As said JACK2 in its current form, in order to be useful on mobile
platforms perhaps requires some adaptions but I see no
showstoppers hindering doing that. And the reward will be very big. We
are very satisfied with JACK running on the
Lionstracs MIDI keyboard, and the performance demands are very big in
that cases. On a phone we don't need
1000 of audio voices and dozen of audio apps running. just a few apps
and give the app developer a way to to achieve low audio
latency hasslefree. I'm not expecting a phone doing the same as a
quadcore desktop cpu etc.
but given that the N900 runs  at 600Mhz let's say 50-60 msec round
trip audio latency in user space should be doable.
Not only in the phone application but in user space apps too.
If power consumption is a concern then give the app developer an API
to tell it to the system.
eg a mp3 player can tell the system , low latency not required, and
the system returns an optimal buffersize (on the bigger side)
to minimize power consumption.
If an app needs to work with low audio latencies, for example a
vocoder (voice changer), audio signal analyzer, virtual synth etc,
then tell the system that low latency is required. At this point the
system can change the audio driver's buffer size (eg by closing
and opening it again) and the system continues to work as before,
providing low latency but with increaased battery consumption
due to higher IRQ rate etc. As soon as the app exits it increases the
buffer size again.
I've read a PDF about the audio on the N900 that something like this
is done on Maemo but I think it is still not optimal.
I have so far only used QAudioInput which uses ALSA on the N900 so I
cannot give my final judgement yet.
I'll try all available APIs and run a synthetic benchmarks to see what
kind of latencies are achievable, but I am not very hopeful ATM.

regards,
Benno
The LinuxSampler project
http://www.linuxsampler.org