Your SlideShare is downloading. ×
Hidden Camera 3 APIs in Android 4.4 (KitKat)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Hidden Camera 3 APIs in Android 4.4 (KitKat)

12,906
views

Published on

Android 4.4 (KitKat) shipped with some hidden Java APIs for the new Camera 3 APIs. This slide deck was presented at AnDevCon 2013 held in Burlingame, CA. It covers the history of the new Camera …

Android 4.4 (KitKat) shipped with some hidden Java APIs for the new Camera 3 APIs. This slide deck was presented at AnDevCon 2013 held in Burlingame, CA. It covers the history of the new Camera Framework, hidden APIs and some basics of the Camera Platform Framework.

https://github.com/lbk003/Cam3HiddenAPIs

Published in: Technology, Business

1 Comment
16 Likes
Statistics
Notes
  • plzzz plzzz plzzz help me guyzz i need presentation on nestle kitkat in this presentation we need excutive summary ,,,sitution analysis,,,segmentation target positioning ( S.T.P ) ,,, pricing strategy and promotion strategy with recommandations almost 8 plzzzzzzzzzz help me guyzzzz :( :(

    marriam.rouf@yahoo.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
12,906
On Slideshare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
363
Comments
1
Likes
16
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Camera 3 The New Camera Subsystem in Android KitKat Balwinder Kaur, Senior Member, Technical Staff, Aptina Imaging AnDevCon, Burlingame, CA 11.15.13 Slide Deck v 1.2 last updated 12.27.13 © 2013 Aptina Imaging Corporation. All rights reserved. Products are warranted only to meet Aptina’s production data sheet specifications. Information, products, and/or specifications are subject to change without notice. All information is provided on an “AS IS” basis without warranties of any kind. Dates are estimates only. Drawings not to scale. Aptina and the Aptina logo are trademarks of Aptina Imaging Corporation. All other trademarks are the property of their respective owners. 1 | © 2013 Aptina Imaging Corporation
  • 2. Agenda I. Camera 3 ‣ Introduction ‣ History ‣ Computational Photography Use Cases II. The APIs ‣ {@hide} – Sneak Peek into coming Android Pro Camera APIs III. The Platform ‣ Architecture ‣ 3A IV. Emerging Trends V. Demo VI. Q&A 2 | © 2013 Aptina Imaging Corporation
  • 3. Camera 3 – The New Camera Framework 3 | © 2013 Aptina Imaging Corporation
  • 4. Limitations of Camera 1 • 3 Operating Modes ‣ Preview ‣ Capture ‣ Video • Burst Mode Photography – hard to implement ‣ Zero Shutter Lag ‣ Multi-Shot HDR ‣ Panoramic Stitch • Metadata: Practically, only face detection data could be returned to the application 4 | © 2013 Aptina Imaging Corporation
  • 5. Camera 3 • Android 4.3 (Jelly Bean MR2) and Android 4.4 (KitKat) have shipped with a new Camera Hardware Interface ‣ Android 4.3 => Camera 3 Hardware Interface ‣ Android 4.4 => Camera 3.1 Hardware Interface What happened to Camera2 ? 5 | © 2013 Aptina Imaging Corporation
  • 6. Overview of Camera 3 • Camera Framework till Android 4.0 was modeled after a point & shoot camera. • Expanded Camera Subsystem is modeled after a Professional Camera ‣ • 6 But it is more than that It is the basis of a potential paradigm shift in photography | © 2013 Aptina Imaging Corporation
  • 7. A Bit of History FCAM F2 Frankencamera Project Lead - Stanford Professor Marc Levoy Main architects - Eino-Ville (Eddy) Talvala and Andrew Adams Eddy Talvala is the Tech Lead for Camera Framework on the Android Team Industry Partner - Nokia Computational Photography ‣ Out of scope for this talk ‣ Good starting point - A white paper on Computational Photography Platforms by Marc Levoy ‣ Use and relevance of software increases in photography ‣ Processing happens in near real time instead of post-processing ‣ Open platforms facilitate innovation 7 | © 2013 Aptina Imaging Corporation
  • 8. Computational Photography Definition • Computational photography refers to computational image capture, processing, and manipulation techniques that enhance or extend the capabilities of digital photography. • Good examples of image capture and manipulation include panoramas, and high-dynamic-range imaging which is the use of differently exposed pictures of the same scene to extend dynamic range beyond even that of analog film-based media. Source: Wikipedia 8 | © 2013 Aptina Imaging Corporation
  • 9. Multi Shot HDR • HDR = High Dynamic Range • Multiple shots at different exposures and then blended together 9 | © 2013 Aptina Imaging Corporation
  • 10. Panoramic Stitching Source: commons.wikimedia.org 10 | © 2013 Aptina Imaging Corporation
  • 11. Multi-Flash Photography Source: Stanford Graphics Research 11 | © 2013 Aptina Imaging Corporation
  • 12. Flash No-Flash Photography Source: Microsoft Research 12 | © 2013 Aptina Imaging Corporation
  • 13. Flash No-Flash Photography Source: Microsoft Research 13 | © 2013 Aptina Imaging Corporation
  • 14. Focus Bracketing Focus Stacking for macro photography Source: Wikipedia 14 | © 2013 Aptina Imaging Corporation
  • 15. View Finder Editing Today’s smartphones have features like • ‣ Computational Photography can take it to the next level • ‣ • 15 Touch to Expose & Touch to Focus Edit the frame in the view finder by selecting areas to sharpen, blur, brighten and maintain these during capture mode Source: SIGGRAPH 2012 | © 2013 Aptina Imaging Corporation
  • 16. Open Platform Requirements for Computation Photography • Fine grained control of the Sensor, Flash, Lens and Image Signal Processing pipeline • Ability to program ‣ exposure time, auto white balance, focus, frame-rate etc. ‣ metering algorithm, autofocus algorithm etc. • Meta data ‣ Every frame is returned with the settings that it was taken with (verses settings take effect sometime in future) • Platform should have enough compute and memory power. ‣ With multi core devices and GPU capability increasing, Android becomes a very attractive platform as an open computational photography platform. 16 | © 2013 Aptina Imaging Corporation
  • 17. Other use cases • Using Imagers for Information ‣ Gesture Recognition ‣ Face Recognition ‣ Augmented Reality • Requirements differ for this class of applications ‣ High and steady Frame Rate under all light conditions is Important. ‣ Of less importance • • 17 | High Resolution Image Quality & Color Accuracy © 2013 Aptina Imaging Corporation
  • 18. The APIs 18 | © 2013 Aptina Imaging Corporation
  • 19. Camera API Camera APIs Package - android.hardware.camera Android Version Additions to android.hardware.camera Kit Kat (4.4) API level 19 None Jelly Bean MR2 (4.3) API level 18 None Jelly Bean MR1 (4.2) API level 17 Minor (HDR, shutterSound) Jelly Bean (4.1) API level 16 Minor (AutoFocus Move Callback) Ice-cream Sandwich API level 15 Minor (Video Stabilization) Ice-cream Sandwich 19 API Level API level 14 Major (Face Detection) | © 2013 Aptina Imaging Corporation
  • 20. A Sneak Peek of the New Camera APIs • Kitkat shipped with a set of Experimental Java camera APIs with the {@hide} tag. ‣ https://android.googlesource.com/platform/frameworks/base /+/kitkat-release/core/java/android/hardware/camera2/ • Package name is android.hardware.camera2 ‣ But in other parts of AOSP many references to ProCamera (Professional Camera) are found. 20 | © 2013 Aptina Imaging Corporation
  • 21. Pro Camera APIs ? • They are experimental and not part of the SDK, but available in the platform. • Likely to change before released in the SDK. • Camera 3 Hardware Interface is fairly stable now. • It’s a good time to start thinking about new Camera applications 21 | © 2013 Aptina Imaging Corporation
  • 22. android.hardware.camera2 Interface Camera Device Exception CameraAccessException 22 | © 2013 Aptina Imaging Corporation Classes CameraCharacteristics CameraDevice.CaptureListener CameraDevice.StateListener CameraManager CameraManager.AvailabilityListener CameraMetadata CameraMetadata.Key<t> CaptureFailure CaptureRequest CaptureRequest.Builder CaptureResult Face Rational Size
  • 23. Camera Manager Interface for iterating, listing and accessing Camera Devices CameraManager manager = (CameraManager) getSystemService(Context.CAMERA_SERVICE); //Iterating through Camera Devices manager.getCameraIdList(); // Query a device for Capabilities manager.getCameraCharacteristics(cameraId); // Open a Camera manager.openCamera ( cameraId,...); // add an Availability Listener as Cameras become available or unavailable manager.addAvailabilityListener(CameraManager.AvailabilityListener); 23 | © 2013 Aptina Imaging Corporation
  • 24. Camera Device • Device is modeled as a pipeline ‣ Input : CaptureRequest for a single Frame ‣ Output: CaptureResult Metadata packet + output image buffers for the request. These buffers can go to one or multiple output targets • SurfaceView • SurfaceTexture • MediaCodec • MediaRecorder • Renderscript Allocation et al • ImageReader (Use this for Still Images) ‣ ‣ 24 Requests are processed in order, multiple requests are in flight to maintain full frame rate ( generally means 30 frames per second ) Android 4.4 added the flush() command to drop all inflight requests. | © 2013 Aptina Imaging Corporation
  • 25. Capture Request // Step 1 : Configure Output Surfaces camera.configureOutputs (List<Surfaces> outputs) // Set the size and format. // Step 2: Build a CaptureRequest // Use default Templates – Manual, Preview, Still_Capture, // Record, Video_Snapshot, Zero_Shutter_Lag CaptureRequest.Builder bldr = camera.createCaptureRequest(CameraDevice.TEMPLATE_VIDEO_SNAPSHOT); //no chaining of calls available bldr.addTarget(); bldr.setTag(); // to identify the result when it is returned bldr.set (Metadata.Key …); CaptureRequest req = bldr.build(); 25 | © 2013 Aptina Imaging Corporation
  • 26. Capture Modes //Step 3: Submit a request //Pick 1 of the 4 // capture, captureBurst, setRepeatingRequest, setRepeatingBurst camera.capture(CaptureRequest, CameraDevice.CaptureListener) //one time capture camera.captureBurst(…) // Bracketing Modes. Guaranteed that no other request will be interspersed in this queue camera.setRepeatingRequest (…) // Preview. Lower priority. Cancelled with stopRepeating() or flush() call. videoRates – needs clarification camera.setRepeatingBurst (…) // same as setRepeatingBurst but at full frame rates 26 | © 2013 Aptina Imaging Corporation
  • 27. CaptureListener //Step 4: Process the progress of the capture //4 callback methods available onCaptureStarted(CameraDevice, CaptureRequest, long timestamp); onCaptureCompleted(CameraDevice, CaptureRequest, CaptureResult); //or onCaptureFailed(CameraDevice, CaptureRequest, CaptureFailure); // once all the capture requests are processed onCaptureSequenceCompleted(CameraDevice, int seqid, int framenumber) 27 | © 2013 Aptina Imaging Corporation
  • 28. Capture Result //Step 5: Investigate the CaptureResult or CaptureFailure Object Received // The results of a single image capture from the //image sensor. Contains the final configuration for //the capture hardware (sensor, lens, flash), the //processing pipeline, the control algorithms, //and the output buffers. get(CameraMetadata.Key<T> key) ; getRequest() ; 28 | © 2013 Aptina Imaging Corporation
  • 29. ImageReader • Introduced in KitKat API Level 19 • Android.media.ImageReader • Allows direct application access to image data rendered into a Surface • maxImages determines the maximum number of Image objects that can be be acquired from the ImageReader simultaneously. Use cautiously ! • ImageFormat is defined by ImageFormat or PixelFormat • ImageReader.OnImageAvailableListener provides callback for when the next Image is available • Use acquireLatestImage, rather than acquireNextImage. It removes old images ImageReader imgreader = ImageReader.newInstance(width, height, format, maxImages); imgreader.getSurface() ; // Pass this into the CameraRequest Class 29 | © 2013 Aptina Imaging Corporation
  • 30. Camera Metadata • Base class for Camera Controls and Information • Metadata is of 2 types ‣ Static and Dynamic • Sensor Control • Flash Control • Lens Control • Pipeline Control • 3 Subclasses ‣ CameraCharacteristics ‣ CaptureRequest ‣ CaptureResult 30 | © 2013 Aptina Imaging Corporation
  • 31. Camera Metadata – Static • Characteristics for an Individual Camera Device • Static – does not change for a device, so no need to open a Camera to get this information. ‣ Can be queried from Camera Manager or Camera Device • Keys are of the format - “Available” or “Info” • Some interesting Metadata Keys ‣ INFO_SUPPORTED_HARDWARE_LEVEL : Limited or Full ‣ 3A : CONTROL_AE_*, CONTROL_AWB_*, CONTROL_AF_* ‣ LENS_INFO_*, FLASH_INFO_*, SENSOR_INFO_* 31 | © 2013 Aptina Imaging Corporation
  • 32. Camera Metadata - Dynamic • Dynamic Metadata is “per frame” • Set in CaptureRequest via the Builder to specify settings while capturing a frame. • Returned as part of the CaptureResult • One has to be a “pro” with Image Sensors and Image Sensing Pipelines to use this. • Very long list of options available 32 | © 2013 Aptina Imaging Corporation
  • 33. How do we use these APIs? • To query whether the APIs are available or not on a platform, use reflection. • {@hide} only means not available in SDK. • Steps to setup development environment ‣ Connect KitKat Device or start KitKat AVD adb pull /system/framework/core.jar . adb pull /system/framework/framework.jar . ‣ Convert from Dex to Jar using tool such as dex2jar dex2jar core.jar dex2jar framework.jar ‣ In Eclipse, Project → Properties → Java Build Path → Libraries → Add External JARs... → core-dex2jar.jar & framework-dex2jar.jar Source : stackoverflow • 33 APIs don’t work on AVD because the HAL is a Version 1 Implementation | © 2013 Aptina Imaging Corporation
  • 34. • https://github.com/lbk003/Cam3HiddenAPIs 34 | © 2013 Aptina Imaging Corporation
  • 35. The Platform 35 | © 2013 Aptina Imaging Corporation
  • 36. AOSP Architecture Application Framework/SDK Java Classes Exposed to the application for interacting with the hardware frameworks/base/core/java/and roid/hardware JNI Glue Code between Java Classes and Native Classes frameworks/base/core/jni/andro id_hardware_Camera.cpp Native Framework Native counterpart of the Java Camera Classes. Manages all Binder Interactions frameworks/av/camera/Camera. cpp IPC Binder 3 Binder Interfaces ICameraService and Icamera from Application to Framework. ICameraClient for callbacks into the Application frameworks/av/camera Camera Service The Camera Service that manages permissions and lifecycle of the Camera Devices frameworks/av/services/camera /libcameraservice/CameraServic e.cpp HAL Interface Hardware Interface. Multiple Versions platform/hardware/libhardware/ include HAL Implementation Hardware specific implementations. Depends on host processor, ISP and Sensor platform/hardware/<vendor>/<pl atform> Kernel drivers http://source.android.com/devices/camera/camera.html 36 | © 2013 Aptina Imaging Corporation
  • 37. Camera Version Saga Camera HAL Version Description 1.0 Android 4.0 (Icecream Sandwich) Converted from C++ CameraHardwareInterface abstraction layer to camera.h 2.0 Android 4.2 (Jelly Bean MR1) Initial Release of expanded capability HAL. camera2.h Allowed ZSL in the service layer. 3.0 Android 4.3 (Jelly Bean MR2) First revision of expanded-capability HAL camera3.h. Operational mode same as Camera 2, but ABI different, and optimized to support multiple host architectures 3.1 37 Android Version Android 4.4 (Kit Kat) Minor Revision. Added flush() call to drop all inflight requests immediately | © 2013 Aptina Imaging Corporation
  • 38. Camera Version Saga (contd) • platform/hardware/libhardware/include/camera_common.h • Camera HAL version = CAMERA_DEVICE_API_VERSION • CAMERA_MODULE_API_VERSION is different CAMERA_MODULE_API_ VERSION CAMERA_DEVICE_A PI_VERSION Android Version (Camera HAL Version) 1.0 1.0 Android 4.0 (Ice-cream Sandwich) 2.0 2.0 Android 4.2 (Jelly Bean MR1) 2.1 3.0 Android 4.3 (Jelly Bean MR2) 2.2 3.1 Android 4.4 (Kit Kat) • CAMERA_DEVICE_API_VERSION 2.x is outdated. 38 | © 2013 Aptina Imaging Corporation
  • 39. Camera 3 HAL • Supports 2 modes – Limited and Full • Limited just means Camera version 1 • Camera HAL version 3 and 1 can co-exist. For example, a front facing camera may be a version 1 device, whereas the rear facing one could be a version 3 device. 39 | © 2013 Aptina Imaging Corporation
  • 40. Camera 1 HAL http://source.android.com/devices/camera/camera.html 40 | © 2013 Aptina Imaging Corporation
  • 41. Camera 3 Core Operation Model http://source.android.com/devices/camera/camera.html 41 | © 2013 Aptina Imaging Corporation
  • 42. Camera 3 HAL Model http://source.android.com/devices/camera/camera.html 42 | © 2013 Aptina Imaging Corporation
  • 43. Camera Pipeline http://source.android.com/devices/camera/camera.html 43 | © 2013 Aptina Imaging Corporation
  • 44. 3A • 3A = Auto Focus, Auto Exposure, Auto White Balance • Along with demosaicing and denoising, forms a crucial part of the image pipeline. • The 3A algorithms are typically the “secret sauce” of the ISP • The State Machine for each of the 3A is defined by the HAL. • When a device is opened the 3A is INACTIVE • 3A algorithms at the top level are controlled by ANDROID_CONTROL_MODE key 1. ANDROID_CONTROL_MODE_OFF 2. ANDROID_CONTROL_MODE_AUTO 3. ANDROID_CONTROL_USE_SCENE_MODE • 44 | SCENE_MODE_FACE_PRIORITY -the AE/AWB/AFMODE controls work as in ANDROID_CONTROL_MODE_AUTO, but the 3A routines must bias toward metering and focusing on any detected faces in the scene. © 2013 Aptina Imaging Corporation
  • 45. Open Source Implementation of HAL3 Nexus 5 • https://android.googlesource.com/platform/hardware/ qcom/camera/+/kitkat-release/QCamera2/HAL3/ • Released for the Qualcomm MSM8974 Snapdragon 800 based device 45 | © 2013 Aptina Imaging Corporation
  • 46. Implementing Camera3 HALs • Overall with Camera3 specification, the Camera HAL implementations have become simpler, since most of the functionality like stream management, etc. are defined and have moved into the Service Layer • Most Camera HAL implementations have had required considerable amount of cleanup and refactoring. • Most of the work is getting the Hardware vendors (AP, ISPs) ready to support these features • Devices with Camera 3 implementations - Nexus 7 (4.3) and Nexus 5 • Major progress but there are areas of investigations - like processing of non-Bayer sensors, multi-stream, multi – array sensors etc. 46 | © 2013 Aptina Imaging Corporation
  • 47. A Peek into the Future 47 | © 2013 Aptina Imaging Corporation
  • 48. Emerging Trends • Photography is going to become a lot about the software as processing from offline computers to smart devices • Imagers for information ‣ Computer Vision, Gesture Recognition, Augmented Reality • Non-Bayer Sensors ‣ Clarity+ • High Dynamic Range ‣ HDR in Preview, Video • Multi Sensor Imagers – Array imagers, Stereo 48 | © 2013 Aptina Imaging Corporation
  • 49. DEMO 49 | © 2013 Aptina Imaging Corporation
  • 50. Q&A 50 | © 2013 Aptina Imaging Corporation
  • 51. References • Computational Photography ‣ ‣ http://graphics.stanford.edu/projects/lightfield/ ‣ http://fcam.garage.maemo.org/ ‣ http://graphics.stanford.edu/courses/cs178/applets/ ‣ https://graphics.stanford.edu/papers/fcam/fcam-sigtalk.pdf ‣ • http://graphics.stanford.edu/projects/camera-2.0/ http://graphics.stanford.edu/papers/camera20/levoy-platforms-cga10.pdf Android ‣ ‣ http://source.android.com/devices/camera/camera.html ‣ • http://source.android.com http://android.googlesource.com Clarity+ ‣ 51 http://www.aptina.com/Aptina_ClarityPlus_WhitePaper.pdf | © 2013 Aptina Imaging Corporation
  • 52. About Aptina is a global provider of CMOS imaging solutions that enable Imaging Everywhere™. Using performance enhancing technologies like Aptina A-Pix™, Aptina A-PixHS™ , DR-Pix™ and Clarity+™ technology, Aptina has created a market-leading portfolio of image sensor products found in leading consumer electronics like smart phones, tablets, laptops, digital and video cameras, as well as applications in surveillance, automotive, medical, video conferencing, and gaming. Clarity+ is Aptina’s breakthrough clear pixel technology that provides 2x improvement in image quality under low light conditions. Contact Information Balwinder Kaur bkaur@aptina.com 52 | © 2013 Aptina Imaging Corporation