[maemo-developers] [maemo-developers] maemo mplayer development and its possible future use on Nokia 770

From: Siarhei Siamashka siarhei.siamashka at gmail.com
Date: Tue Dec 12 01:57:02 EET 2006
On Monday 11 December 2006 11:26, Frantisek Dufka wrote:

> Yes, the result would look like video overlay works in windows or linux
> on PC - overlay draws over different windows when it shouldn't :-) We
> can live with that.

I thought it is actually not a problem but quite a good thing :-) Surely for
mplayer as a standalone video player, supporting keboard/initializing 
some window is important.

But if it just outputs video into some rectangular screen area (provided by
some other application) and is controlled via issuing commands through 
a pipe, it makes possible to develop some advanced frontends which use
mplayer as a video rendering engine. For example a twin of the standard 
Nokia 770 video player which simulates all its GUI controls could be created.

My original goal of posting the previous message was an attempt to find a
volunteer who would like to try developing such a frontend.

I don't have that much time to devote to mplayer development myself. Up until
this moment I even could not concentrate on solving some specific task but
tried some bits with MP3 audio output, decoder improvements, GUI and user 
interface, fixing arm specific bugs, and now video output code with hardware
YUV support. Also some kind of management work, integration of useful
patches and support for users in the forums takes some time. I would like to
concentrate on some task such as video decoder optimizations for ARM, but
seeing that other parts are not in a quite good shape, distracts attention
somewhat :-)

> As for framebuffer permissions, it may be better to 
> relax device permissions than to run mplayer as root.

The most right way to solve this issue is probably to add 'user' to 
'video' group. Alternative solutions involve messing with mplayer 
binary ownership and suid/sgid bits. I wonder what is possible to 
do automatically in the least intrusive way when installing mplayer 

> Well, the conversion is done on the fly while the data is transferred to
> internal epson video buffer. I guess it would be hard to do planar YUV
> -> RGB without additional memory. I still don't understand how it is
> done on the fly even in those packed formats since some color
> information (U,V) is common for more lines. Seems like tough task. There
> needs to be additional memory for remembering U,V parts from previous line.

YUV422 is a good format as it matches 16-bit RGB format quite well. Both of
them use 16 bits per pixel, and YUV422 encodes each pair of pixels into a
stride of 4 bytes (16-bit RGB encodes each pixel into 2 bytes, but you can
also treat it as 2 pixels in 4 bytes). So we can mix YUV422 and RGB data in
a framebuffer quite conveniently.

> > Another interesting possibility is to relay video scaling and color
> > conversion (planer -> packed YUV) to DSP.
> I'm not sure, is there some math involved in this or it is just memory
> shuffling? I guess DSP would be really bad for memory shuffling. From
> previous discussions it looks like when you add DSP to the mix all kinds
> of bottlenecks appears. I wonder if gstreamer/dspfbsink could keep up
> with mplayer speed doing just conversion and video output.

Actually DSP may be a good choice for scaling, if you check the same
spru098.pdf you will find "Pixel interpolation Hardware Extension" part :-)
Also looks like dspfbsink uses DSP for scaling as it provides a mapped 
memory for planar YV12 data (or its variant) and accepts a command to 
do the rendering. I looked through xserver sources and gst plugins to dig 
for information and I think I got some impression about how they work, but 
I think this all deserves a separate post along with some additional
inquiries addressed to Nokia developers :-)

ARM can perform YV12->YUV422 conversion quite fast if properly optimized, I
even suspect that it can provide a throughoutput comparable to memcpy (as
memory controller/write buffer performance is a limiting factor here and some
data shuffling will not make much difference). The benchmarks in my previous
message use standard color conversion/scaling code from mplayer which is 
not optimized for ARM. But just color format conversion is a special case,
sometimes scaling is required and mplayer scaler is rather slow. Scaling
performed by mplayer was completely unusable for RGB target colorspace with
x11 driver, that's why maemo build of mplayer had fallback to SDL when
playback for scaled video was required. Now with the target colorspace YUV422,
it is slow but still usable and a bit better than SDL. If we want a fast
scaler for ARM, using JIT is a good option (and I have some experience in
developing JIT translator for x86).

Anyway, I hope that by using DSP for scaling and running it asynchronously, it
is possible to reduce ARM core usage to almost zero and keep all the resources
for video decoding. A related interesting observation is that screen update
ioctl does not seem to affect performance at all (commenting it out does not
improve performace and naturally we get no picture on the screen). I suspect
that screen update is performed asynchronously while ARM core continues to 
execute code after that ioctl call.

With all that said, decoding performance is much more important than 
scaling. If decoder is not fast enough to decode video in time, it is
completely unplayable. If the cpu resources remaining after decoding 
are not enough for scaling and video output, some frames are just 
skipped and the overall effect is not so disasterous.

With a slow scaler and video output code, if we gradually increase 
bitrate and test playback, video will play perfectly first, then it will start
skiping some frames until all the frames get skipped and video becomes
unplayable. With a very fast scaler, video will play perfectly first (and
remain perfect even if with a slow scaler it would get framedrops 
already) but as bitrate increases, it will suddenly become unplayable at 
some point. So the time spent for scaling and video output can be 
considered as some kind of backup resource which can be used to 
keep video watchable at the expense of dropping some frames when 
needed :-)

> Oh BTW, it is off topic but I finally found what that cryptic 'Video
> hardware accelerators for DCT, iDCT, pixel interpolation, and motion
> estimation for video compression' feature on OMAP 1710 page means. Looks
> like the DSP really has some special instructions for performing those
> operations, google for spru098.pdf It is sad that default video player
> is still so bad with such features implemented in hardware.

Thanks for the information, I have found this pdf along with the other docs
from TI quite a long time ago, downloaded their toolchain, but never had
time/motivation to try it in action yet :-)

More information about the maemo-developers mailing list