Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Camera2 API: Overview

68 views

Published on

Overviewing Android's advanced Camera API and sharing my attempt to migrate an existing camera app to the new API.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Camera2 API: Overview

  1. 1. Camera2 API: Overview Shift (Suhyun Park)
  2. 2. Android Camera API existed since version 1.0 (API 1)
  3. 3. * only ‘major’ feature added in
  4. 4. Camera2 API (android.hardware.camera2) - Introduced in Lollipop 5.0 (API 21) / Nov. 12, 2014: almost 4 years ago - New devices can use both Camera2 API (android.hardware.camera2) and deprecated Camera API (android.hardware.camera) - To reduce confusion, we’ll call the deprecated one as ‘Camera1’
  5. 5. What Camera2 can do while Camera1 can’t - Accessing to 3+ cameras (hopefully at once, for P 9.0 (API 28) or higher) - DSLR-like manual controls (ISO, exposure time, lens focus distance, white balance, etc.) © LGE © LGE
  6. 6. What Camera2 can do while Camera1 can’t - Burst mode - RAW support - etc. © Google
  7. 7. Example use of Camera2 - Portrait mode Open 2 cameras simultaneously, outfocus one of them and process image © Google
  8. 8. … but do they?
  9. 9. Vendors They don’t really care about Camera2
  10. 10. Vendors … or they care so much about Camera2 that they don’t care about the deprecated one
  11. 11. Fragmentation - Some vendors even has their own SDKs (it’s usable on other vendors’ devices, hopefully) © Samsung
  12. 12. So is it worth using? - Camera1 is deprecated - Camera2 has some serious professional features for a camera app, and it’s generally way faster compared to Camera1 when implemented right - API 21+ (>87%, as of July 2018) can benefit from it - But we have to make 2 logics (for devices not fully supporting and/or acting strange on Camera2)
  13. 13. Dealing with it : Technical Aspect
  14. 14. © Google
  15. 15. © Google
  16. 16. The pipeline - It takes a whole new pipeline (“completely reworked” according to Google) - Everything is asynchronous © Google
  17. 17. The pipeline CameraManager - is a system service - is used to query available cameras and features - is a starting point of Camera2 © Google
  18. 18. The pipeline CameraCharacteristics - is a map of the camera’s characteristics - Used like this: characteristics[CameraChar acteristics.LENS_FACING] © Google
  19. 19. The pipeline CameraCharacteristics - is NOT equivalent to Camera1’s properties © Google
  20. 20. The pipeline CameraDevice - is the camera device - unlike Camera1, it is got asynchronously using CameraManager and a callback © Google
  21. 21. The pipeline CaptureRequest - is used to control camera parameters, and request for preview or image capture © Google
  22. 22. The pipeline CameraCaptureSession - is used to send requests and receive results from/to the hardware © Google
  23. 23. © Google
  24. 24. The pipeline Notice this? we can specify multiple surface outputs! - this was not possible in Camera1 © Google
  25. 25. The pipeline But the aspect ratio has to be same across all sizes (at least to support Galaxy devices)
  26. 26. The pipeline Example CaptureRequest for camera previewing // Preview request is built with builder pattern previewRequestBuilder = cameraDevice?.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW) ?: return // We can set multiple targets, i.e. show multiple previews previewRequestBuilder.addTarget(Surface(imageTexture)) // Setting camera preferences previewRequestBuilder[CaptureRequest.CONTROL_AF_MODE] = CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE previewRequestBuilder[CaptureRequest.FLASH_MODE] = CaptureRequest.FLASH_MODE_OFF previewRequest = previewRequestBuilder.build() // Sending a PreviewRequest to current camera session // The session will handle given PreviewRequest appropriately captureSession?.setRepeatingRequest(previewRequest, captureCallback, backgroundHandler)
  27. 27. The pipeline Notice the backgroundHandler? the API manages all the background stuff for us // Preview request is built with builder pattern previewRequestBuilder = cameraDevice?.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW) ?: return // We can set multiple targets, i.e. show multiple previews previewRequestBuilder.addTarget(Surface(imageTexture)) // Setting camera preferences previewRequestBuilder[CaptureRequest.CONTROL_AF_MODE] = CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE previewRequestBuilder[CaptureRequest.FLASH_MODE] = CaptureRequest.FLASH_MODE_OFF previewRequest = previewRequestBuilder.build() // Sending a PreviewRequest to current camera session // The session will handle given PreviewRequest appropriately captureSession?.setRepeatingRequest(previewRequest, captureCallback, backgroundHandler)
  28. 28. Find the Difference : Differences Between Camera1 and 2
  29. 29. Setting Up Preview / Connecting to Surface camera = Camera.open(id) // This is the camera val callback = object: CameraDevice.StateCallback() { override fun onOpened(camera: CameraDevice?) { cameraDevice = camera // This is the cameraDevice } ... } cameraManager.openCamera(id, callback, backgroundHandler) Straightforward, is blocking current thread Gets device by callback, does its job by backgroundHandler
  30. 30. Setting Up Preview / Connecting to Surface (camera ?: return).let { it.setPreviewTexture(imageTexture) // or you can do // it.setPreviewDisplay(surfaceHolder) it.startPreview() } val surface = Surface(imageTexture) previewRequestBuilder = cameraDevice!! .createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW) previewRequestBuilder.addTarget(surface) cameraDevice!!.createCaptureSession( listOf(surface, imageReader!!.surface), object : CameraCaptureSession.StateCallback() { override fun onConfigured( session: CameraCaptureSession?) { captureSession = session // This is the session previewRequest = previewRequestBuilder.build() captureSession? .setRepeatingRequest(previewRequest, captureCallback, backgroundHandler) } ... } }, backgroundHandler) Again, straightforward, is blocking current thread. We have to manually set preview size to match the surface’s size. Gets session by callback, and we can start preview by calling setRepeatingRequest. Preview size is automatically chosen by the API to match the surface size.
  31. 31. Properties vs. CameraCharacteristics Properties (Camera1) - Mixed mutable/immutable properties - things like previewSize are mutable; things like supportedPreviewSizes are not - You CAN call camera.setProperties() to update camera settings - You NEED to open a camera to use this CameraCharacteristics (Camera2) - Immutable properties only - like the name says, you can only get the characteristics of a CameraDevice - You CAN’T update camera settings with this class; it’s used only for checking device features - You DON’T NEED to open a camera
  32. 32. Querying Features camera!!.parameters!!.supportedFlashModes cameraManager .getCameraCharacteristics(cameraId) .get(CameraCharacteristics.FLASH_INFO_AVAILABLE) Can query features by accessing Camera.Parameters Need to open camera before querying Can query features by CameraCharacteristics, using cameraId (not cameraDevice) so no need to open cameraDevice
  33. 33. Setting Parameters camera!!.parameters = camera!!.parameters.apply { flashMode = Camera.Parameters.FLASH_MODE_ON zoom = 125 // Sets zoom to x1.25, if supported } previewRequestBuilder[CaptureRequest.LENS_FOCUS_DISTANCE] = 9.50f previewRequestBuilder[CaptureRequest.SENSOR_EXPOSURE_TIME] = 1000000000L / 30 previewRequest = previewRequestBuilder.build() captureSession?.stopRepeating() captureSession?.setRepeatingRequest(previewRequest, captureCallback, backgroundHandler) Sets parameters directly When given unsupported values, it’ll throw an error Sets parameters by builder pattern When given unsupported values, it’ll try its best to fit into supported values, by rounding or clamping, etc
  34. 34. Setting Parameters camera!!.parameters = camera!!.parameters.apply { flashMode = Camera.Parameters.FLASH_MODE_ON zoom = 125 // Sets zoom to x1.25, if supported } private fun <T> setPreviewOptions( vararg options: Pair<CaptureRequest.Key<in T>, T>, postProcess: () -> Unit = {}) { for (option in options) { previewRequestBuilder[option.first] = option.second } postProcess() previewRequest = previewRequestBuilder.build() captureSession?.stopRepeating() captureSession?.setRepeatingRequest( previewRequest, captureCallback, backgroundHandler) } ... setPreivewOptions( CaptureRequest.LENS_FOCUS_DISTANCE to 9.50f, CaptureRequest.SENSOR_EXPOSURE_TIME to 1000000000L / 30 ) Sets parameters directly When given unsupported values, it’ll throw an error We can make an extension function to simplify this
  35. 35. Setting Parameters camera!!.parameters = camera!!.parameters.apply { flashMode = Camera.Parameters.FLASH_MODE_ON zoom = 125 // Sets zoom to x1.25, if supported } It can be rather frustrating because only some specific values are supported by the camera, so if I request x1.25, the app will likely to crash on Galaxy S9+ But Camera2 handles similar cases appropriately - like what would we expect Example supported zoom ratios (by Samsung Galaxy S9+ Camera) Check out my Camera1-based OSP here : https://github.com/shiftpsh/sensor-tester/releases
  36. 36. Setting parameters in Camera1 vs. 2 Camera1 - Parameters are global - If we want to make a change, then it’s changed globally. But since image processing takes time, the API waits until last image finishes processing and finally change the settings Camera2 - Parameters are set per request - Image processing stages gets parameters per request - the API don’t have to care about the last image or whatsoever, resulting improved frame rates
  37. 37. Setting parameters in Camera1 1234567 processing stages request params A params B current params: A result
  38. 38. Setting parameters in Camera1 1234567 processing stages request params A params B If we change settings... It get mixed current params: B result
  39. 39. Setting parameters in Camera1 1222223 processing stages request params A params B current params: A So Camera1 does this way … one by one result
  40. 40. Setting parameters in Camera2 1 / A2 / A3 / A4 / B5 / B6 / B7 / B processing stages request params A params B But in Camera2, parameters are baked into the requests result
  41. 41. Setting parameters in Camera1 vs. 2 Camera1 Camera2 result … resulting much higher speed
  42. 42. © Google
  43. 43. © Google
  44. 44. Taking Pictures camera!!.takePicture(null, null) { bytes, _ -> process(bytes) } imageReader = ImageReader.newInstance(largestSize.width, largestSize.height, ImageFormat.JPEG, 2) val onImageAvailableListener = ImageReader.OnImageAvailableListener { reader -> process(image) } imageReader!!.setOnImageAvailableListener( onImageAvailableListener, backgroundHandler) val captureBuilder = cameraDevice!! .createCaptureRequest(CameraDevice.TEMPLATE_STILL_CAPTURE) captureBuilder.addTarget(imageReader!!.surface) ... val captureCallback = object : CameraCaptureSession.CaptureCallback() { override fun onCaptureCompleted(...) {...} } captureSession?.stopRepeating() captureSession?.abortCaptures() captureSession?.capture( captureBuilder.build(), captureCallback, backgroundHandler) One of the few functions that uses a callback. Returns a ByteArray - RGB format It uses an imageReader to capture image. Also uses a callback. Returns an Image - YUV format (typically)
  45. 45. Implementing it in existing apps
  46. 46. API may be different but the flow is same Open Camera Connect to Surface Start Preview / Repeating Request Apply Filter, etc.. Capture Image Close Camera
  47. 47. Speed comparison Device Camera1 Camera2 Improvement Google Pixel 2 (9) 87.7ms 35.1ms 150% Samsung Galaxy S9+ (8.0) 71.7ms 27.9ms 157% LG G7 ThinQ (8.0) 37.3ms 11.5ms 224% Opening the camera * Averaged result of 10 runs
  48. 48. Speed comparison Device Camera1 Camera2 Improvement Google Pixel 2 (9) 810.3ms 406.2ms 99% Samsung Galaxy S9+ (8.0) 477.2ms 351.1ms 36% LG G7 ThinQ (8.0) 616.4ms 382.9ms 61% Taking a single full resolution picture * Averaged result of 10 runs, right after callback
  49. 49. Migration Strategy - Not all devices work well with Camera2, even if its API level ≥ 21 - It’s up to vendors - if they wrote good enough HAL(hardware abstraction layer) for Camera2 then it’ll work well, otherwise not
  50. 50. Migration Strategy - We can test some devices if they supports Camera2 well and make a whitelist - Or we can ask our users to opt-in, like putting an option to enable advanced camera in ‘experimental’ section like many other apps do
  51. 51. Q&A

×