Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Camera 2.0 in Android 4.2


Published on

Android Builder's Summit 2013

Published in: Technology
  • Be the first to comment

Camera 2.0 in Android 4.2

  1. 1. Camera 2.0 The New Camera Hardware Interface in Android 4.2 Balwinder Kaur, Senior Member, Technical Staff, Aptina Imaging Ashutosh Gupta, Principal Software Engineer, Aptina Imaging Android Builder’s Summit, San Francisco, CA 2.18.2013© 2013 Aptina Imaging Corporation. All rights reserved. Products are warranted only to meet Aptina’s production data sheet specifications. Information, products, and/orspecifications are subject to change without notice. All information is provided on an “AS IS” basis without warranties of any kind. Dates are estimates only. Drawings not toscale. Aptina and the Aptina logo are trademarks of Aptina Imaging Corporation. All other trademarks are the property of their respective owners. 1 | © 2013 Aptina Imaging Corporation
  2. 2. Agenda • Camera Use Cases ‣ Prominent Use Cases ‣ Limitations of existing APIs ‣ Overview of android.hardware.Camera • Android Framework : Camera Service ‣ Native Camera service ‣ New - Under the hood – Camera 2.0 • It’s all about the Camera hardware ! ‣ Camera Hardware Abstraction Layer ‣ Camera Device Driver • Challenges in Camera HAL development • Emerging Trends • Q&A2 | © 2013 Aptina Imaging Corporation
  3. 3. Section I Android Camera APIs3 | © 2013 Aptina Imaging Corporation
  4. 4. Prominent Camera Use Cases • Main Use Cases ‣ Live Preview of Camera Stream • Live Preview + copy of the Frame returned to the application ‣ Capture a frame ‣ Video Recording of a Camera Stream • Secondary Use Cases ‣ Configuring the Camera ‣ Snapshot during video recording ‣ Event Callbacks: Shutter Clicked, AutoFocus Achieved • Information Related Use cases • Receiving more than an image back . e.g. face detection data • Meta Data of an Image4 | © 2013 Aptina Imaging Corporation
  5. 5. Limitations in Camera API • No support for Burst Mode Photography • No support for Panoramic Shots • Very limited support for frame metadata • No per-frame control of the camera or the image processing pipeline • No access to control subsystems within the camera e.g. Focus, Flash, Image Sensor5 | © 2013 Aptina Imaging Corporation
  6. 6. Overview of android.hardware.Camera 6 Classes 8 Callback Interfaces • Camera.AutoFocusCallback • Camera • Camera.ErrorCallback • Camera.CameraInfo • Camera.FaceDetectionListener • Camera.Parameters • Camera.OnZoomChangeListener • Camera.Size • Camera.PictureCallback • Camera.Face • Camera.PreviewCallback • Camera.Area • Camera.ShutterCallback • Camera.AutoFocusMoveCallback6 | © 2013 Aptina Imaging Corporation
  7. 7. Camera class Contains all the methods for the Camera Lifecycle • Open & Release • Access to the Camera Controls • Preview ‣ Direct Live Preview to the display or a texture ‣ Get Preview Frame in a Callback • Capture ‣ Callbacks: Shutter, JPEG, RAW, “Postview” • Lock & Unlock • Actions: startAutoFocus, startSmoothZoom & startFaceDetection7 | © 2013 Aptina Imaging Corporation
  8. 8. Camera.Parameters Class for Camera Controls ① Mandatory Feature Set ‣ getSupportedPreviewSizes + set/get ② Optional Feature Set ‣ isVideoStabilizationSupported + set/get ③ Custom Feature Set ‣ Camera.Parameters class provides a “dumb” pipe to the hardware for custom controls ‣ set (“your_param_string”, value); get(“your_param_string”);Auto White Balance, Scene Modes, Focus Modes, Preview Sizes, Picture Sizes,Thumbnail Sizes, Preview Format, Picture Format, Anti-Banding8 | © 2013 Aptina Imaging Corporation
  9. 9. … the rest of the Camera Classes • Camera.CameraInfo ‣ For each camera, front or back facing, orientation of the camera image • Camera.Size ‣ width and height of the image • Camera.Face ‣ face-id, co-ordinates for left eye, right eye, mouth, outer bounds of the face • Camera.Area ‣ Rectangular bounds with a weight ‣ Metering Regions for 3A : Auto Focus, Auto White Balance, Auto Exposure9 | © 2013 Aptina Imaging Corporation
  10. 10. Camera 2 Internals Released as part of Android 4.2 Camera 1 => android.hardware.Camera Camera 2 => android.hardware.ProCamera New Camera HAL implementation based on Camera 2.0 – Samsung exynos5 based No application that actually uses Camera 2.0 Disclaimer: All reference to Camera2 in this slide deck is based on reverse engineering AOSP code !10 | © 2013 Aptina Imaging Corporation
  11. 11. Section II Android Framework - Camera11 | © 2013 Aptina Imaging Corporation
  12. 12. High Level Architecture12 | © 2013 Aptina Imaging Corporation
  13. 13. Android High Level Architecture Source: Android Anatomy and Physiology, Google IO 200813 | © 2013 Aptina Imaging Corporation
  14. 14. Hardware Abstraction LayerCamera Source: Android Anatomy and Physiology, Google IO 2008 14 | © 2013 Aptina Imaging Corporation
  15. 15. Camera Subsystem Application Application framework Hardware Independent Camera Service Camera HAL Implementation Camera Device Driver Hardware Dependent Camera Hardware HAL = Hardware Abstraction Layer15 | © 2013 Aptina Imaging Corporation
  16. 16. Process View App App App Binder IPC ICamera Camera Service Back Facing Front Facing Binder IPCSurfaceflinger ISurface Camera Hardware Camera Hardware Object Object Media server System Call System Call Kernel Driver Kernel Driver16 | © 2013 Aptina Imaging Corporation
  17. 17. Inside the Camera App Application Code android framework code JNI android_hardware_Camera.cpp IBinder Interfaces Camera Service Media server17 | © 2013 Aptina Imaging Corporation
  18. 18. JNI Layer18 | © 2013 Aptina Imaging Corporation
  19. 19. android_hardware_Camera • Allocates Memory from the Java memory heap for JPEG images. • If a Copy of the Preview Frame is requested by the app, then the copy from native to java buffers is done here. • Creates a persistent context for callbacks from native code to Java (JNICameraContext) • Holds references to the Java Camera, Face and Area objects.19 | © 2013 Aptina Imaging Corporation
  20. 20. Camera Service - with Camera 2.020 | © 2013 Aptina Imaging Corporation
  21. 21. Camera Service Application Code android framework code JNI android_hardware_Camera.cpp ICameraClient | ICameraService | ICamera Camera Service camera.h Camera HAL implementation Media server21 | © 2013 Aptina Imaging Corporation
  22. 22. Camera Service with Camera 2 Application Code android framework code JNI android_hardware_Camera.cpp ICameraClient | ICameraService | ICamera Camera2Client Camera Service Camera2Device camera.h camera2.h Camera HAL Camera2 HAL implementation implementation Media server22 | © 2013 Aptina Imaging Corporation
  23. 23. Camera Service • Runs in the media server process • It is a shared library • Queries System for Camera HAL Api version - accordingly creates a CameraClient or Camera2Client for the application • For Camera 1, libcameraservice does: Permission check android.permission.CAMERA Ensures only one Client connects to a Camera Hardware Object Ensures each Process connects to a single Camera Hardware Object Redirects callbacks back to the app layer Accessed over IBinder Interface Number of Cameras Available CameraInfo Details23 | © 2013 Aptina Imaging Corporation
  24. 24. Camera2Client• Implements API on top of camera device HAL version 2• Has processors that run in their own threads RecordingManages Preview & Recording Streams Streaming Processor Preview Stream StreamStill Image capture output processing JPEG Processor Capture StreamProcesses Callback Callback Processor Callback StreamZSL Queue Processing ZSL Processor ZSL StreamMetadata processing for output frames Frame ProcessorManages Capture Sequences for ZSL Capture Sequencerregular and Burst modes24 | © 2013 Aptina Imaging Corporation
  25. 25. Camera2Device • Manages Input and Output Streams between the Camera HAL and the Service • Functions ‣ Retrieves Static Metadata Information of the Camera ‣ Sends requests for Capture and Streaming ‣ Manages input and output streams including re-processing streams and metadata buffers ‣ StreamAdaptor between ANativeWindow Interface and the Camera HAL stream ops • ANativeWindows is Android Native Window interface ‣ Sends Notifications (Error, Shutter, AutoFocus, AutoExposure and AWB)25 | © 2013 Aptina Imaging Corporation
  26. 26. Metadata • Camera2 has a lot of emphasis on retrieving metadata from the camera and making it available to the application • Two kinds of Metadata ‣ Static & Frame-Based ‣ Static does not change and is available without opening the camera device (Camera Info Class) ‣ *_INFO is the static metadata ANDROID_LENS, ANDROID_LENS_INFO, ANDROID_SENSOR, ANDROID_SENSOR_INFO, ANDROID_FLASH, ANDROID_FLASH_INFO, ANDROID_HOT_PIXEL, ANDROID_HOT_PIXEL_INFO, ANDROID_DEMOSAIC, ANDROID_DEMOSAIC_INFO, ANDROID_NOISE, ANDROID_NOISE_INFO, ANDROID_SHADING, ANDROID_SHADING_INFO, ANDROID_GEOMETRIC, ANDROID_GEOMETRIC_INFO, ANDROID_COLOR, ANDROID_COLOR_INFO, ANDROID_TONEMAP , ANDROID_TONEMAP_INFO, ANDROID_EDGE, ANDROID_EDGE_INFO, ANDROID_SCALER, ANDROID_SCALER_INFO, ANDROID_JPEG, ANDROID_JPEG_INFO, ANDROID_STATS, ANDROID_STATS_INFO, ANDROID_CONTROL, ANDROID_CONTROL_INFO, ANDROID_QUIRKS_INFO etc. • Extensible - there is provision for Vendor Specific Tags26 | © 2013 Aptina Imaging Corporation
  27. 27. Android Open Source Project (AOSP) Structure • Android Framework ‣ Java: frameworks/base/core/java/android/hardware ‣ JNI: frameworks/base/core/jni • Camera Service ‣ frameworks/av/services/camera/libcameraservice/ • IBinder Interfaces ‣ frameworks/av/include/camera/ICamera.h etc. • IBinder Implementation ‣ frameworks/av/camera/ICamera.cpp etc. • Camera HAL Interface ‣ hardware/libhardware/include/hardware/camera2.h etc. • Camera HAL Implementation ‣ hardware/<vendor>/camera (typically) • Camera Metadata ‣ system/media/camera/include/system/camera_metadata_tags.h27 | © 2013 Aptina Imaging Corporation
  28. 28. Section III Its all about the Camera Hardware !28 | © 2013 Aptina Imaging Corporation
  29. 29. Camera Hardware Abstraction Layer Review of a Typical Implementation29 | © 2013 Aptina Imaging Corporation
  30. 30. Camera Stack – Camera HAL Upper Camera Stack … Camera Hardware Abstraction Layer SurfaceFlinger (HAL) /Overlay Buffers User Vendor Specific HAL Implementation Camera Driver Kernel Image Sensor Image Sensor Hardware Processer30 | © 2013 Aptina Imaging Corporation
  31. 31. Android CameraHAL Library • The Camera Hardware Abstraction Layer (HAL) is a library that is specific to the camera hardware platform ‣ Written by hardware vendors (Qualcomm, Samsung, TI, others) • CameraHAL maps Android Camera Service calls to driver functions ‣ Ice Cream Sandwich (ICS) uses camera.h ‣ Jelly Bean (and above) use camera2.h • CameraHAL low level interface communicates with the kernel level driver ‣ It can support interfaces including Video for Linux 2 (V4L2) or OpenMax (OMX) ‣ Communicates with the driver through file I/O calls (open, close, input/output controls (IOCTL), etc)31 | © 2013 Aptina Imaging Corporation
  32. 32. Sample ICS CameraHAL Functional Diagram Camera Service I/F Memory Manager Display Surface Manager CameraHAL Event Notification Manager Camera Manager User Camera Driver Kernel Source: TI OMAP4 /dev/videoX git.omapzoom.org32 | © 2013 Aptina Imaging Corporation
  33. 33. Sample JB CameraHAL Functional Diagram Camera Service I/F Stream Manager CameraHAL Operations Handler Metadata Handler Reprocess Stream Management User Camera Driver Kernel Source: Samsung Exynos 5 /dev/videoX33 | © 2013 Aptina Imaging Corporation
  34. 34. JB CameraHAL Block Diagram Discussion • Block Functions ‣ CameraHAL • Initialization, thread creation, function dispatch ‣ Stream Manager • Handle stream events, coordinate buffer and stream usage, manage state machine ‣ Metadata Handler • Acquire per shot metadata, convert to Android format ‣ Reprocess Stream Management • Setup and manage reprocess streams34 | © 2013 Aptina Imaging Corporation
  35. 35. JB CameraHAL Changes • For CameraHAL 2.0, much of the CameraHAL 1.0 functionality is moved to libcameraservice • CameraHAL 2.0 targeted at HW specific functionality • New CameraHAL functionality currently not passed up to applications • Image metadata has more importance • Reprocessing introduced – Process an already captured image stream • Stream based rather than stream function based ‣ CameraHAL 1.0 has methods for specific stream types (e.g., start_preview(), take_picture()) ‣ CameraHAL 2.0 has generalized stream methods (e.g., allocate_stream())35 | © 2013 Aptina Imaging Corporation
  36. 36. Camera Device Driver36 | © 2013 Aptina Imaging Corporation
  37. 37. Camera Stack – Camera Driver Upper Camera Stack … Camera Hardware Abstraction Layer SurfaceFlinger (HAL) /Overlay Buffers User Vendor Specific HAL Implementation Camera Driver Kernel Image Sensor Image Sensor Hardware Processer37 | © 2013 Aptina Imaging Corporation
  38. 38. Android Kernel Camera Driver • The kernel driver presents a standard interface for different types of camera hardware ‣ Camera hardware specific attributes are handled by the low level kernel driver ‣ Image Sensor Processor (ISP) vs. SOC (smart) sensor - differences are handled at the driver level • For Android, Video for Linux 2 (V4L2) is used in many implementations ‣ V4L2 has been in existence for many years ‣ OpenMax (OMX) is also used for a low level driver interface by some vendors.38 | © 2013 Aptina Imaging Corporation
  39. 39. V4L2 Kernel Driver Block Diagram V4L2 Driver Interface - IOCTL support/dispatch - V4L2 driver infrastructure Controlling Interface - Support for different device configurations - Control device flow Buffer/Memory Management Camera HW Management- Memory allocation (if needed) - One of these blocks for each camera- Buffer management type- Buffer queue/de-queue - Device discovery - Device initialization - Power management - Set/get device specific parameter - Enable/disable image streaming39 | © 2013 Aptina Imaging Corporation
  40. 40. V4L2 Kernel Driver Resources • Memory ‣ Memory can be either driver-allocated or user-provided ‣ The image transfers from the camera to memory through hardware Direct Memory Access (DMA) ‣ Hardware memory management may be used to avoid contiguous memory requirement • Interrupts ‣ Camera ports support for interrupts on events such as frame start, finish, focus events, etc. • Camera Control: I2C/SPI ‣ I2C (Inter-Integrated Control) is used for writing or reading camera registers ‣ SPI (Serial Peripheral Interface) is a faster alternative to I2C • Control Signals/GPIO ‣ All controlled by the low level driver • Power ‣ Sensor power management is critical to embedded device operation ‣ Sensors support standby mode where settings are maintained while power usage is reduced40 | © 2013 Aptina Imaging Corporation
  41. 41. V4L2 Driver Buffer Management• One or more buffers are supported• User buffers or kernel-allocated buffers are supported Buff_0• Buffers are treated the same for preview, capture, video (output resolution does not Buff_x Buff_1 matter)• Buffers are queued to a circular list Buff_3 Buff_2• Buffer filling starts when the V4L2 Stream_On command is executed• Once filled, the CameraHAL de-queues a buffer, processes the buffer, then re- queues the buffer• The Stream_Off command causes all buffer to be released41 | © 2013 Aptina Imaging Corporation
  42. 42. Typical V4L2 Preview Sequence (1) • V4L2 preview start up sequence is given below V4L2 Call Driver Events Hardware Events VIDIOC_S_FMT – set format Set image format and size Set both resolution and output pixel format VIDIOC_G_PARM – get Get a camera driver or hardware Read camera parameter parameter parameter VIDIOC_S_PARM – set Set a camera driver or hardware Write camera parameter parameter parameter VIDIOC_CROPCAP – get Return camera cropping None cropping capabilities capabilities VIDIOC_S_CROP – set Set cropping rectangle Set camera cropping rectangle cropping VIDIOC_REQBUFS – request Request buffer support from the None camera buffer driver (user vs. kernel) Loop: VIDIOC_QUERYBUF For kernel allocated buffers, return None – query buffer caps buffer characteristics V4L2_MMAP – map buffers For kernel allocated buffers, None to user space memory map to user space42 | © 2013 Aptina Imaging Corporation
  43. 43. Typical V4L2 Preview Sequence (2) • V4L2 preview start up sequence (cont) V4L2 Call Driver Events Hardware Events Loop: VIDIOC_QBUF – Queue buffers in the circular queue none queue buffers VIDIOC_STREAM_ON – start Start image capture state Enable image output streaming • V4L preview shut-down sequence V4L2 Call Driver Events Hardware Events VIDIOC_STREAM_OFF – stop Stop streaming, deallocate Disable image output streaming buffer43 | © 2013 Aptina Imaging Corporation
  44. 44. V4L2 Media Controller Architecture • Designed to support dynamically re-connectable hardware blocks. • Allows for greater programmer control • Introduces the notion of entities, pads and links44 | © 2013 Aptina Imaging Corporation
  45. 45. V4L2 Media Controller Architecture(Cont.) 45 | © 2013 Aptina Imaging Corporation
  46. 46. Section IV Challenges in Camera HAL Development46 | © 2013 Aptina Imaging Corporation
  47. 47. Challenges in Camera HAL Development • Memory Management • Various Implementation of Camera Driver • Color Format Conversion • Buffer Synchronization between Camera & Display • Support for Advanced Features47 | © 2013 Aptina Imaging Corporation
  48. 48. Are all Camera HALs equal ? No• Supported Features depend on hardware capabilities• Supported Features depend on the implementation in the Camera HAL• Performance• Reliability• Extensions to the standard Android Feature Set48 | © 2013 Aptina Imaging Corporation
  49. 49. A Peek into the Future49 | © 2013 Aptina Imaging Corporation
  50. 50. Emerging Trends • Computer Vision Applications go mainstream ‣ APIs on Object Tracking, Gesture Recognition become more common place • Computation Photography application ‣ Developers get fine grained control of flash and camera • High Dynamic Range ‣ Ability to capture larger exposure range • 3D Imaging ‣ Use of 2 cameras to generate a 3D image50 | © 2013 Aptina Imaging Corporation
  51. 51. What’s coming ?51 | © 2013 Aptina Imaging Corporation
  52. 52. Q&A52 | © 2013 Aptina Imaging Corporation
  53. 53. References • • • • • -sdk-get-raw-preview-camera-image-without- displaying-it • • | © 2013 Aptina Imaging Corporation
  54. 54. About Aptina enables imaging everywhere. A leading innovator of CMOS imaging technology, Aptina delivers excellent pixel performance, sensor functionality and camera system capability to a world going visual. Aptina Camera Software Stack (Access) is Aptina’s implementation of the Android Camera HAL with some custom extensions. Contact Information Balwinder Kaur Ashutosh Gupta ashutosh@aptina.com54 | © 2013 Aptina Imaging Corporation