Camera 2.0 in Android 4.2


Published on

Android Builder's Summit 2013

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • In photography, the metering mode refers to the way in which a camera determines the exposure.Autobracketing is a feature of some more advanced cameras, whether film or digital cameras, particularly single-lens reflex cameras, where the camera will take several successive shots (often three) with slightly different settings. Two modes are used: The images are automatically combined, for example Automatic Exposure Bracketing into oneHigh dynamic range image, or the best-looking separately stored pictures can be picked later from the batch.White balance is a camera setting that adjusts for lighting in order to make white objects appear white in photos. This is more difficult than it might seem due to the fact that light cast from different sources is different in color (technically called temperature). That is to say, light is rarely truly white in nature. The light from an incandescent or halogen bulb, for example, is red/orange in color, while that from the sun is relatively blue. A proper white balance setting in a camera will prevent, for example, a white bed sheet in a photo from appearing orange in color when it is being illuminated by a candle.
  • Provides a common shutter sound and recording sound for all camera clients (using the MediaPlayer) "/system/media/audio/ui/camera_click.ogg");mSoundPlayer[SOUND_RECORDING] = newMediaPlayer("/system/media/audio/ui/VideoRecord.ogg"
  • Uses a MetadataQueue class
  • Camera 2.0 in Android 4.2

    1. 1. Camera 2.0 The New Camera Hardware Interface in Android 4.2 Balwinder Kaur, Senior Member, Technical Staff, Aptina Imaging Ashutosh Gupta, Principal Software Engineer, Aptina Imaging Android Builder’s Summit, San Francisco, CA 2.18.2013© 2013 Aptina Imaging Corporation. All rights reserved. Products are warranted only to meet Aptina’s production data sheet specifications. Information, products, and/orspecifications are subject to change without notice. All information is provided on an “AS IS” basis without warranties of any kind. Dates are estimates only. Drawings not toscale. Aptina and the Aptina logo are trademarks of Aptina Imaging Corporation. All other trademarks are the property of their respective owners. 1 | © 2013 Aptina Imaging Corporation
    2. 2. Agenda • Camera Use Cases ‣ Prominent Use Cases ‣ Limitations of existing APIs ‣ Overview of android.hardware.Camera • Android Framework : Camera Service ‣ Native Camera service ‣ New - Under the hood – Camera 2.0 • It’s all about the Camera hardware ! ‣ Camera Hardware Abstraction Layer ‣ Camera Device Driver • Challenges in Camera HAL development • Emerging Trends • Q&A2 | © 2013 Aptina Imaging Corporation
    3. 3. Section I Android Camera APIs3 | © 2013 Aptina Imaging Corporation
    4. 4. Prominent Camera Use Cases • Main Use Cases ‣ Live Preview of Camera Stream • Live Preview + copy of the Frame returned to the application ‣ Capture a frame ‣ Video Recording of a Camera Stream • Secondary Use Cases ‣ Configuring the Camera ‣ Snapshot during video recording ‣ Event Callbacks: Shutter Clicked, AutoFocus Achieved • Information Related Use cases • Receiving more than an image back . e.g. face detection data • Meta Data of an Image4 | © 2013 Aptina Imaging Corporation
    5. 5. Limitations in Camera API • No support for Burst Mode Photography • No support for Panoramic Shots • Very limited support for frame metadata • No per-frame control of the camera or the image processing pipeline • No access to control subsystems within the camera e.g. Focus, Flash, Image Sensor5 | © 2013 Aptina Imaging Corporation
    6. 6. Overview of android.hardware.Camera 6 Classes 8 Callback Interfaces • Camera.AutoFocusCallback • Camera • Camera.ErrorCallback • Camera.CameraInfo • Camera.FaceDetectionListener • Camera.Parameters • Camera.OnZoomChangeListener • Camera.Size • Camera.PictureCallback • Camera.Face • Camera.PreviewCallback • Camera.Area • Camera.ShutterCallback • Camera.AutoFocusMoveCallback6 | © 2013 Aptina Imaging Corporation
    7. 7. Camera class Contains all the methods for the Camera Lifecycle • Open & Release • Access to the Camera Controls • Preview ‣ Direct Live Preview to the display or a texture ‣ Get Preview Frame in a Callback • Capture ‣ Callbacks: Shutter, JPEG, RAW, “Postview” • Lock & Unlock • Actions: startAutoFocus, startSmoothZoom & startFaceDetection7 | © 2013 Aptina Imaging Corporation
    8. 8. Camera.Parameters Class for Camera Controls ① Mandatory Feature Set ‣ getSupportedPreviewSizes + set/get ② Optional Feature Set ‣ isVideoStabilizationSupported + set/get ③ Custom Feature Set ‣ Camera.Parameters class provides a “dumb” pipe to the hardware for custom controls ‣ set (“your_param_string”, value); get(“your_param_string”);Auto White Balance, Scene Modes, Focus Modes, Preview Sizes, Picture Sizes,Thumbnail Sizes, Preview Format, Picture Format, Anti-Banding8 | © 2013 Aptina Imaging Corporation
    9. 9. … the rest of the Camera Classes • Camera.CameraInfo ‣ For each camera, front or back facing, orientation of the camera image • Camera.Size ‣ width and height of the image • Camera.Face ‣ face-id, co-ordinates for left eye, right eye, mouth, outer bounds of the face • Camera.Area ‣ Rectangular bounds with a weight ‣ Metering Regions for 3A : Auto Focus, Auto White Balance, Auto Exposure9 | © 2013 Aptina Imaging Corporation
    10. 10. Camera 2 Internals Released as part of Android 4.2 Camera 1 => android.hardware.Camera Camera 2 => android.hardware.ProCamera New Camera HAL implementation based on Camera 2.0 – Samsung exynos5 based No application that actually uses Camera 2.0 Disclaimer: All reference to Camera2 in this slide deck is based on reverse engineering AOSP code !10 | © 2013 Aptina Imaging Corporation
    11. 11. Section II Android Framework - Camera11 | © 2013 Aptina Imaging Corporation
    12. 12. High Level Architecture12 | © 2013 Aptina Imaging Corporation
    13. 13. Android High Level Architecture Source: Android Anatomy and Physiology, Google IO 200813 | © 2013 Aptina Imaging Corporation
    14. 14. Hardware Abstraction LayerCamera Source: Android Anatomy and Physiology, Google IO 2008 14 | © 2013 Aptina Imaging Corporation
    15. 15. Camera Subsystem Application Application framework Hardware Independent Camera Service Camera HAL Implementation Camera Device Driver Hardware Dependent Camera Hardware HAL = Hardware Abstraction Layer15 | © 2013 Aptina Imaging Corporation
    16. 16. Process View App App App Binder IPC ICamera Camera Service Back Facing Front Facing Binder IPCSurfaceflinger ISurface Camera Hardware Camera Hardware Object Object Media server System Call System Call Kernel Driver Kernel Driver16 | © 2013 Aptina Imaging Corporation
    17. 17. Inside the Camera App Application Code android framework code JNI android_hardware_Camera.cpp IBinder Interfaces Camera Service Media server17 | © 2013 Aptina Imaging Corporation
    18. 18. JNI Layer18 | © 2013 Aptina Imaging Corporation
    19. 19. android_hardware_Camera • Allocates Memory from the Java memory heap for JPEG images. • If a Copy of the Preview Frame is requested by the app, then the copy from native to java buffers is done here. • Creates a persistent context for callbacks from native code to Java (JNICameraContext) • Holds references to the Java Camera, Face and Area objects.19 | © 2013 Aptina Imaging Corporation
    20. 20. Camera Service - with Camera 2.020 | © 2013 Aptina Imaging Corporation
    21. 21. Camera Service Application Code android framework code JNI android_hardware_Camera.cpp ICameraClient | ICameraService | ICamera Camera Service camera.h Camera HAL implementation Media server21 | © 2013 Aptina Imaging Corporation
    22. 22. Camera Service with Camera 2 Application Code android framework code JNI android_hardware_Camera.cpp ICameraClient | ICameraService | ICamera Camera2Client Camera Service Camera2Device camera.h camera2.h Camera HAL Camera2 HAL implementation implementation Media server22 | © 2013 Aptina Imaging Corporation
    23. 23. Camera Service • Runs in the media server process • It is a shared library • Queries System for Camera HAL Api version - accordingly creates a CameraClient or Camera2Client for the application • For Camera 1, libcameraservice does: Permission check android.permission.CAMERA Ensures only one Client connects to a Camera Hardware Object Ensures each Process connects to a single Camera Hardware Object Redirects callbacks back to the app layer Accessed over IBinder Interface Number of Cameras Available CameraInfo Details23 | © 2013 Aptina Imaging Corporation
    24. 24. Camera2Client• Implements API on top of camera device HAL version 2• Has processors that run in their own threads RecordingManages Preview & Recording Streams Streaming Processor Preview Stream StreamStill Image capture output processing JPEG Processor Capture StreamProcesses Callback Callback Processor Callback StreamZSL Queue Processing ZSL Processor ZSL StreamMetadata processing for output frames Frame ProcessorManages Capture Sequences for ZSL Capture Sequencerregular and Burst modes24 | © 2013 Aptina Imaging Corporation
    25. 25. Camera2Device • Manages Input and Output Streams between the Camera HAL and the Service • Functions ‣ Retrieves Static Metadata Information of the Camera ‣ Sends requests for Capture and Streaming ‣ Manages input and output streams including re-processing streams and metadata buffers ‣ StreamAdaptor between ANativeWindow Interface and the Camera HAL stream ops • ANativeWindows is Android Native Window interface ‣ Sends Notifications (Error, Shutter, AutoFocus, AutoExposure and AWB)25 | © 2013 Aptina Imaging Corporation
    26. 26. Metadata • Camera2 has a lot of emphasis on retrieving metadata from the camera and making it available to the application • Two kinds of Metadata ‣ Static & Frame-Based ‣ Static does not change and is available without opening the camera device (Camera Info Class) ‣ *_INFO is the static metadata ANDROID_LENS, ANDROID_LENS_INFO, ANDROID_SENSOR, ANDROID_SENSOR_INFO, ANDROID_FLASH, ANDROID_FLASH_INFO, ANDROID_HOT_PIXEL, ANDROID_HOT_PIXEL_INFO, ANDROID_DEMOSAIC, ANDROID_DEMOSAIC_INFO, ANDROID_NOISE, ANDROID_NOISE_INFO, ANDROID_SHADING, ANDROID_SHADING_INFO, ANDROID_GEOMETRIC, ANDROID_GEOMETRIC_INFO, ANDROID_COLOR, ANDROID_COLOR_INFO, ANDROID_TONEMAP , ANDROID_TONEMAP_INFO, ANDROID_EDGE, ANDROID_EDGE_INFO, ANDROID_SCALER, ANDROID_SCALER_INFO, ANDROID_JPEG, ANDROID_JPEG_INFO, ANDROID_STATS, ANDROID_STATS_INFO, ANDROID_CONTROL, ANDROID_CONTROL_INFO, ANDROID_QUIRKS_INFO etc. • Extensible - there is provision for Vendor Specific Tags26 | © 2013 Aptina Imaging Corporation
    27. 27. Android Open Source Project (AOSP) Structure • Android Framework ‣ Java: frameworks/base/core/java/android/hardware ‣ JNI: frameworks/base/core/jni • Camera Service ‣ frameworks/av/services/camera/libcameraservice/ • IBinder Interfaces ‣ frameworks/av/include/camera/ICamera.h etc. • IBinder Implementation ‣ frameworks/av/camera/ICamera.cpp etc. • Camera HAL Interface ‣ hardware/libhardware/include/hardware/camera2.h etc. • Camera HAL Implementation ‣ hardware/<vendor>/camera (typically) • Camera Metadata ‣ system/media/camera/include/system/camera_metadata_tags.h27 | © 2013 Aptina Imaging Corporation
    28. 28. Section III Its all about the Camera Hardware !28 | © 2013 Aptina Imaging Corporation
    29. 29. Camera Hardware Abstraction Layer Review of a Typical Implementation29 | © 2013 Aptina Imaging Corporation
    30. 30. Camera Stack – Camera HAL Upper Camera Stack … Camera Hardware Abstraction Layer SurfaceFlinger (HAL) /Overlay Buffers User Vendor Specific HAL Implementation Camera Driver Kernel Image Sensor Image Sensor Hardware Processer30 | © 2013 Aptina Imaging Corporation
    31. 31. Android CameraHAL Library • The Camera Hardware Abstraction Layer (HAL) is a library that is specific to the camera hardware platform ‣ Written by hardware vendors (Qualcomm, Samsung, TI, others) • CameraHAL maps Android Camera Service calls to driver functions ‣ Ice Cream Sandwich (ICS) uses camera.h ‣ Jelly Bean (and above) use camera2.h • CameraHAL low level interface communicates with the kernel level driver ‣ It can support interfaces including Video for Linux 2 (V4L2) or OpenMax (OMX) ‣ Communicates with the driver through file I/O calls (open, close, input/output controls (IOCTL), etc)31 | © 2013 Aptina Imaging Corporation
    32. 32. Sample ICS CameraHAL Functional Diagram Camera Service I/F Memory Manager Display Surface Manager CameraHAL Event Notification Manager Camera Manager User Camera Driver Kernel Source: TI OMAP4 /dev/videoX git.omapzoom.org32 | © 2013 Aptina Imaging Corporation
    33. 33. Sample JB CameraHAL Functional Diagram Camera Service I/F Stream Manager CameraHAL Operations Handler Metadata Handler Reprocess Stream Management User Camera Driver Kernel Source: Samsung Exynos 5 /dev/videoX33 | © 2013 Aptina Imaging Corporation
    34. 34. JB CameraHAL Block Diagram Discussion • Block Functions ‣ CameraHAL • Initialization, thread creation, function dispatch ‣ Stream Manager • Handle stream events, coordinate buffer and stream usage, manage state machine ‣ Metadata Handler • Acquire per shot metadata, convert to Android format ‣ Reprocess Stream Management • Setup and manage reprocess streams34 | © 2013 Aptina Imaging Corporation
    35. 35. JB CameraHAL Changes • For CameraHAL 2.0, much of the CameraHAL 1.0 functionality is moved to libcameraservice • CameraHAL 2.0 targeted at HW specific functionality • New CameraHAL functionality currently not passed up to applications • Image metadata has more importance • Reprocessing introduced – Process an already captured image stream • Stream based rather than stream function based ‣ CameraHAL 1.0 has methods for specific stream types (e.g., start_preview(), take_picture()) ‣ CameraHAL 2.0 has generalized stream methods (e.g., allocate_stream())35 | © 2013 Aptina Imaging Corporation
    36. 36. Camera Device Driver36 | © 2013 Aptina Imaging Corporation
    37. 37. Camera Stack – Camera Driver Upper Camera Stack … Camera Hardware Abstraction Layer SurfaceFlinger (HAL) /Overlay Buffers User Vendor Specific HAL Implementation Camera Driver Kernel Image Sensor Image Sensor Hardware Processer37 | © 2013 Aptina Imaging Corporation
    38. 38. Android Kernel Camera Driver • The kernel driver presents a standard interface for different types of camera hardware ‣ Camera hardware specific attributes are handled by the low level kernel driver ‣ Image Sensor Processor (ISP) vs. SOC (smart) sensor - differences are handled at the driver level • For Android, Video for Linux 2 (V4L2) is used in many implementations ‣ V4L2 has been in existence for many years ‣ OpenMax (OMX) is also used for a low level driver interface by some vendors.38 | © 2013 Aptina Imaging Corporation
    39. 39. V4L2 Kernel Driver Block Diagram V4L2 Driver Interface - IOCTL support/dispatch - V4L2 driver infrastructure Controlling Interface - Support for different device configurations - Control device flow Buffer/Memory Management Camera HW Management- Memory allocation (if needed) - One of these blocks for each camera- Buffer management type- Buffer queue/de-queue - Device discovery - Device initialization - Power management - Set/get device specific parameter - Enable/disable image streaming39 | © 2013 Aptina Imaging Corporation
    40. 40. V4L2 Kernel Driver Resources • Memory ‣ Memory can be either driver-allocated or user-provided ‣ The image transfers from the camera to memory through hardware Direct Memory Access (DMA) ‣ Hardware memory management may be used to avoid contiguous memory requirement • Interrupts ‣ Camera ports support for interrupts on events such as frame start, finish, focus events, etc. • Camera Control: I2C/SPI ‣ I2C (Inter-Integrated Control) is used for writing or reading camera registers ‣ SPI (Serial Peripheral Interface) is a faster alternative to I2C • Control Signals/GPIO ‣ All controlled by the low level driver • Power ‣ Sensor power management is critical to embedded device operation ‣ Sensors support standby mode where settings are maintained while power usage is reduced40 | © 2013 Aptina Imaging Corporation
    41. 41. V4L2 Driver Buffer Management• One or more buffers are supported• User buffers or kernel-allocated buffers are supported Buff_0• Buffers are treated the same for preview, capture, video (output resolution does not Buff_x Buff_1 matter)• Buffers are queued to a circular list Buff_3 Buff_2• Buffer filling starts when the V4L2 Stream_On command is executed• Once filled, the CameraHAL de-queues a buffer, processes the buffer, then re- queues the buffer• The Stream_Off command causes all buffer to be released41 | © 2013 Aptina Imaging Corporation
    42. 42. Typical V4L2 Preview Sequence (1) • V4L2 preview start up sequence is given below V4L2 Call Driver Events Hardware Events VIDIOC_S_FMT – set format Set image format and size Set both resolution and output pixel format VIDIOC_G_PARM – get Get a camera driver or hardware Read camera parameter parameter parameter VIDIOC_S_PARM – set Set a camera driver or hardware Write camera parameter parameter parameter VIDIOC_CROPCAP – get Return camera cropping None cropping capabilities capabilities VIDIOC_S_CROP – set Set cropping rectangle Set camera cropping rectangle cropping VIDIOC_REQBUFS – request Request buffer support from the None camera buffer driver (user vs. kernel) Loop: VIDIOC_QUERYBUF For kernel allocated buffers, return None – query buffer caps buffer characteristics V4L2_MMAP – map buffers For kernel allocated buffers, None to user space memory map to user space42 | © 2013 Aptina Imaging Corporation
    43. 43. Typical V4L2 Preview Sequence (2) • V4L2 preview start up sequence (cont) V4L2 Call Driver Events Hardware Events Loop: VIDIOC_QBUF – Queue buffers in the circular queue none queue buffers VIDIOC_STREAM_ON – start Start image capture state Enable image output streaming • V4L preview shut-down sequence V4L2 Call Driver Events Hardware Events VIDIOC_STREAM_OFF – stop Stop streaming, deallocate Disable image output streaming buffer43 | © 2013 Aptina Imaging Corporation
    44. 44. V4L2 Media Controller Architecture • Designed to support dynamically re-connectable hardware blocks. • Allows for greater programmer control • Introduces the notion of entities, pads and links44 | © 2013 Aptina Imaging Corporation
    45. 45. V4L2 Media Controller Architecture(Cont.) 45 | © 2013 Aptina Imaging Corporation
    46. 46. Section IV Challenges in Camera HAL Development46 | © 2013 Aptina Imaging Corporation
    47. 47. Challenges in Camera HAL Development • Memory Management • Various Implementation of Camera Driver • Color Format Conversion • Buffer Synchronization between Camera & Display • Support for Advanced Features47 | © 2013 Aptina Imaging Corporation
    48. 48. Are all Camera HALs equal ? No• Supported Features depend on hardware capabilities• Supported Features depend on the implementation in the Camera HAL• Performance• Reliability• Extensions to the standard Android Feature Set48 | © 2013 Aptina Imaging Corporation
    49. 49. A Peek into the Future49 | © 2013 Aptina Imaging Corporation
    50. 50. Emerging Trends • Computer Vision Applications go mainstream ‣ APIs on Object Tracking, Gesture Recognition become more common place • Computation Photography application ‣ Developers get fine grained control of flash and camera • High Dynamic Range ‣ Ability to capture larger exposure range • 3D Imaging ‣ Use of 2 cameras to generate a 3D image50 | © 2013 Aptina Imaging Corporation
    51. 51. What’s coming ?51 | © 2013 Aptina Imaging Corporation
    52. 52. Q&A52 | © 2013 Aptina Imaging Corporation
    53. 53. References • • • • • -sdk-get-raw-preview-camera-image-without- displaying-it • • | © 2013 Aptina Imaging Corporation
    54. 54. About Aptina enables imaging everywhere. A leading innovator of CMOS imaging technology, Aptina delivers excellent pixel performance, sensor functionality and camera system capability to a world going visual. Aptina Camera Software Stack (Access) is Aptina’s implementation of the Android Camera HAL with some custom extensions. Contact Information Balwinder Kaur Ashutosh Gupta ashutosh@aptina.com54 | © 2013 Aptina Imaging Corporation