The document discusses playing with camera preview buffers on BlackBerry 10. It provides an overview of different options for accessing preview buffers, including using autocallback, signals, and filters. It then discusses optimizing the conversion of preview buffer formats like NV12 to RGB formats. Key optimizations discussed include using fixed point arithmetic instead of floating point, precalculating values, improving write operations, and removing unnecessary operations to achieve up to a 500% speed improvement.

1. Playing with camera preview
buffers on BlackBerry10

2. Agenda
- Who am I
- BlackBerry 10 & Camera
- Preview buffers
- Viewfinder raw format

3. Who am I?
- Mobile Software Engineering Manager
at Imagination Technologies (@imgtec)
- BlackBerry Elite
@rrafols
http://blog.rafols.org

4. BlackBerry10 & Camera

5. Multiple options

6. Invoke Camera Card
Trivial

7. InvokeRequest req;
req.setTarget("sys.camera.card");
req.setMimeType("image/jpeg");
req.setAction("bb.action.CAPTURE");
req.setData("photo");
invokeManager->invoke(req);

8. QML
Easy & Simple

9. Page {
onCreationCompleted: {
camera.open(CameraUnit.Rear);
}
Camera {
id: camera
onTouch: {
camera.capturePhoto();
}
onCameraOpened: {
camera.startViewfinder();
}
}
}

10. Qt / Cascades
Slightly more complex
Image work on Qt/C++

11. init
fWin = ForeignWindowControl::create().windowId(QString("cam"));
QObject::connect(fWin,
SIGNAL(windowAttached(screen_window_t, QString&, QString&)),
this,
SLOT(onWindowAttached(screen_window_t, QString&, QString&)));
onWindowAttached
int i = (mCameraUnit == CAMERA_UNIT_FRONT);
screen_set_window_property_iv(win, SCREEN_PROPERTY_MIRROR, &i);
i = -1;
screen_set_window_property_iv(win, SCREEN_PROPERTY_ZORDER, &i);
screen_context_t screen_ctx;
screen_get_window_property_pv(win, SCREEN_PROPERTY_CONTEXT,
(void **)&screen_ctx);
screen_flush_context(screen_ctx, 0);

12. C APIs
With great power comes
great responsibility!

14. And lots of work to do...

15. What are preview buffers and
what can you do with them?

16. PreviewBuffers are
Viewfinder raw buffers

17. Can be used in read only mode
or read/write

18. Multiple options
again

19. Autocallback
Signal
Filter

20. Autocallback
Callback when viewfinder
buffer is available

21. if(camera_start_photo_viewfinder(
cameraHandle,
&previewBufferAvailable,
NULL,
NULL) == CAMERA_EOK)
{
...
}

22. Autocallback
Signal
Filter

23. Signal
Register previewFrameAvailable
signal

24. Multiple buffers can be added
(max 16)

25. Developer is responsible for:
- Allocating buffers
- Adding buffers as available
- Freeing buffers

26. Steps
1 – Register signal

27. cam = root->findChild<Camera*>("cam");
QObject::connect(cam,
SIGNAL(previewFrameAvailable(...)),
this,
SLOT(onPreviewFrameAvailable(...));

28. Steps
2 – Allocate buffers &
add them as available

29. quint64 size = cam->previewBufferSize();
for(int i = 0; i < N_BUFS; i++)
{
buf[i] = malloc(size * sizeof(char));
QSharedPointer<unsigned char>b (buf[i]);
cam->addPreviewBuffer(b, size);
}

30. Steps
3 – Implement slot

31. void onPreviewFrameAvailable(
SharedUCharPointer buffer,
quint64 size,
unsigned int width,
unsigned int height,
unsigned int stride)
{
…
…
cam->addPreviewBuffer(buffer, size)
}

32. Autocallback
Signal
Filter

33. Filter
Applying a filter (r/w)
or processing data (r/o)

34. Filter
Slightly more complex
than other methods

35. Steps
1 – Create image processor
thread

36. chid = ChannelCreate(0);
coid = ConnectAttach(0, 0, chid,
_NTO_SIDE_CHANNEL, 0);
SIGEV_PULSE_INIT(&sigev, coid,
SIGEV_PULSE_PRIO_INHERIT,
FILTER_PULSE_CODE,
0);
pthread_create(&tid, NULL,
processThread, NULL);

37. Steps
2 – Enable viewfinder and
register camera handle

38. if(camera_enable_viewfinder_event(
handle,
CAMERA_EVENTMODE_READWRITE,
&key,
&sigev) != CAMERA_EOK)
{
return NULL;
}
camera_register_resource(handle);

39. Steps
3 – Implement message loop

40. while (!filter_stop) {
rcvid = MsgReceivePulse(
chid, &pulse, sizeof pulse, NULL);
… check right pulse.code …
camera_get_viewfinder_buffers(
handle, key, &inbuf, &outbuf);
apply_filter(&inbuf, &outbuf);
camera_return_buffer(handle, &inbuf);
camera_return_buffer(handle, &outbuf);
}

41. Disclaimer
Those methods are
not mutually exclusive

42. Disclaimer - II
If user callbacks can not keep up,
frames will be dropped

43. Disclaimer - III
Viewfinder not impacted by frame
drops

44. Viewfinder raw format
NV12

46. Ok, but.. what about our RGB?

47. Wikipedia:
int convertYUVtoARGB(int y, int u, int v) {
u = u – 128;
v = v – 128;
int r = y + (int)(1.772f*v);
int g = y - (int)(0.344f*v + 0.714f*u);
int b = y + (int)(1.402f*u);
r = r>255? 255 : r<0 ? 0 : r;
g = g>255? 255 : g<0 ? 0 : g;
b = b>255? 255 : b<0 ? 0 : b;
return 0xff000000 | (r<<16) | (g<<8) | b;
}

48. Qt:
yValue = ((*dataPtr++) - 16) * 1.164;
uValue = ((*uvDataPtr++) - 128);
vValue = ((*uvDataPtr) - 128);
bValue = yValue + 2.018 * uValue;
gValue = yValue - 0.813 * vValue - 0.391
* uValue;
rValue = yValue + 1.596 * vValue;

49. My implementation

50. int r = clip((int) ((y - 16) * 1.164 +
1.596 * (v – 128)));
int g = clip((int) ((y - 16) * 1.164 -
0.391 * (u - 128) - 0.813 * (v – 128)));
int b = clip((int) ((y - 16) * 1.164 +
2.018 * (u - 128)));

51. Optimization
As Y values share UV we can
generate multiple RGB pixels
with a single UV pair.

52. int r0 = clip((int) ((y0 - 16) * 1.164 ...
int g0 = clip((int) ((y0 - 16) * 1.164 ...
int b0 = clip((int) ((y0 - 16) * 1.164 ...
int r1 = clip((int) ((y1 - 16) * 1.164 ...
int g1 = clip((int) ((y1 - 16) * 1.164 ...
int b1 = clip((int) ((y1 - 16) * 1.164 ...

53. Optimization
Do not calculate already
calculated values

54. y0 = y0 – 16; y1 = y1 – 16;
u = u – 128; v = v – 128;
float y0v = y0 * 1.164;
float y1v = y1 * 1.164;
float chromaR = 1.596 * v;
float chromaG = -0.391 * u - 0.813 * v;
float chromaB = 2.018 * u;
r0 = clip((int) y0v + chromaR)
g0 = clip((int) y0v + chromaG)
b0 = clip((int) y0v + chromaB)
r1 = clip((int) y0v + chromaR)
g1 = clip((int) y1v + chromaG)
b1 = clip((int) y1v + chromaB)

55. Optimization
Avoid floating point operations!
Use fixed point arithmetic
(8 bits precision)

56. All floating point values have been
premultiplied by 256
u = u – 128; v = v – 128;
int y0v = (y0 – 16) * 298;
int y1v = (y1 – 16) * 298;
int chromaR = 408 * v;
int chromaG = -100 * u - 208 * v;
int chromaB = 517 * u;
int r0 = clip((y0v + chromaR) >> 8);
int g0 = clip((y0v + chromaG) >> 8);
int b0 = clip((y0v + chromaB) >> 8);
int r1 = clip((y1v + chromaR) >> 8);
int g1 = clip((y1v + chromaG) >> 8);
int b1 = clip((y1v + chromaB) >> 8);

57. Plain integer operations are
usually way faster!

58. Optimization
Precalculate!

59. void precalc() {
for(int i = 0; i < 256; i++) {
factorY[i] = ( 298 * (i - 16)) >> 8;
factorRV[i] = ( 408 * (i - 128)) >> 8;
factorGU[i] = (-100 * (i - 128)) >> 8;
factorGV[i] = (-208 * (i - 128)) >> 8;
factorBU[i] = ( 517 * (i - 128)) >> 8;
}
for(int i = 0; i < 256 + 300; i++) {
clipV[i] = min(max(i - 300, 0), 255);
}
}

60. We add 300 positions to clipping
values to avoid negative indexes

61. int chromaR = factorRV[v];
int chromaG = factorGU[u] + factorGV[v];
int chromaB = factorBU[v];
int r0 = clipV[y0 + chromaR + 300];
int g0 = clipV[y0 + chromaG + 300];
int b0 = clipV[y0 + chromaB + 300];
int r1 = clipV[y1 + chromaR + 300];
int g1 = clipV[y1 + chromaG + 300];
int b1 = clipV[y1 + chromaB + 300];

62. Optimization
Remove the +300

63. int *clipV_ = &clipV[300];
int chromaR = factorRV[v];
int chromaG = factorGU[u] + factorGV[v];
int chromaB = factorBU[v];
int r0 = clipV_[y0 + chromaR];
int g0 = clipV_[y0 + chromaG];
int b0 = clipV_[y0 + chromaB];
int r1 = clipV_[y1 + chromaR];
int g1 = clipV_[y1 + chromaG];
int b1 = clipV_[y1 + chromaB];

64. Optimization
Improve write operations

65. DON'T
out[wpos ] = b0;
out[wpos + 1] = g0;
out[wpos + 2] = r0;
out[wpos + 3] = 0xff;
out[wpos + 4] = b1;
out[wpos + 5] = g1;
out[wpos + 6] = r1;
out[wpos + 7] = 0xff;
wpos += 8;

66. Optimization
Precalc clip values with shift,
mask & alpha value

67. for(i = 0; i < 256 + 300; i++) {
int c = min(max(i - 300, 0), 255);
clipV[i] = c;
clipVR[i] = 0xFF000000 | c << 16);
clipVG[i] = c << 8;
clipVB[i] = c;
}

68. DO
out[wpos ] = clipVR_[y0 + chromaR] |
clipVG_[y0 + chromaG] |
clipVB_[y0 + chromaB];
out[wpos + 1] = clipVR_[y1 + chromaR] |
clipVG_[y1 + chromaG] |
clipVB_[y1 + chromaB];
wpos += 8;

69. Improvement
~500% speed improvement on a
Z10 device
from ~600ms to ~120ms per frame

71. References
http://en.wikipedia.org/wiki/YUV
http://developer.blackberry.com/native/ref
erence/core/com.qnx.doc.camera.lib_ref/top
ic/overview.html
https://qt.gitorious.org/qt/qtmultimedia/c
ommit/31b454b8d6d27dec0fb39987eb315fe93de7
eda1?format=patch
Paul Bernhardt Presentations (@Pbernhardt)
Sean McVeigh Presentations (@sdlmcveigh)

72. Contact
http://blog.rafols.org
twitter: @rrafols