Overviewing Android's advanced Camera API and sharing my attempt to migrate an existing camera app to the new API.

1. Camera2 API: Overview
Shift (Suhyun Park)

2. Android Camera API existed
since version 1.0 (API 1)

4. * only ‘major’ feature added in

5. Camera2 API (android.hardware.camera2)
- Introduced in Lollipop 5.0 (API 21) / Nov. 12, 2014: almost 4 years ago
- New devices can use both Camera2 API (android.hardware.camera2) and
deprecated Camera API (android.hardware.camera)
- To reduce confusion, we’ll call the deprecated one as ‘Camera1’

6. What Camera2 can do while Camera1 can’t
- Accessing to 3+ cameras
(hopefully at once, for P 9.0 (API
28) or higher)
- DSLR-like manual controls (ISO,
exposure time, lens focus
distance, white balance, etc.)
© LGE
© LGE

8. What Camera2 can do while Camera1 can’t
- Burst mode
- RAW support
- etc.
© Google

9. Example use of Camera2
- Portrait mode
Open 2 cameras
simultaneously, outfocus
one of them and process
image
© Google

10. … but do they?

11. Vendors
They don’t really care about
Camera2

12. Vendors
… or they care so much about Camera2 that they don’t care about the
deprecated one

13. Fragmentation
- Some vendors even has their own
SDKs (it’s usable on other
vendors’ devices, hopefully)
© Samsung

15. So is it worth using?
- Camera1 is deprecated
- Camera2 has some serious professional features for a camera app, and it’s
generally way faster compared to Camera1 when implemented right
- API 21+ (>87%, as of July 2018) can benefit from it
- But we have to make 2 logics (for devices not fully supporting and/or
acting strange on Camera2)

16. Dealing with it
: Technical Aspect

17. © Google

18. © Google

19. The pipeline
- It takes a whole new
pipeline (“completely
reworked” according to
Google)
- Everything is asynchronous
© Google

20. The pipeline
CameraManager
- is a system service
- is used to query available
cameras and features
- is a starting point of
Camera2
© Google

21. The pipeline
CameraCharacteristics
- is a map of the camera’s
characteristics
- Used like this:
characteristics[CameraChar
acteristics.LENS_FACING]
© Google

22. The pipeline
CameraCharacteristics
- is NOT equivalent to
Camera1’s properties
© Google

23. The pipeline
CameraDevice
- is the camera device
- unlike Camera1, it is got
asynchronously using
CameraManager and a
callback
© Google

24. The pipeline
CaptureRequest
- is used to control camera
parameters, and request for
preview or image capture
© Google

25. The pipeline
CameraCaptureSession
- is used to send requests
and receive results from/to
the hardware
© Google

26. © Google

27. The pipeline
Notice this? we can specify
multiple surface outputs! - this
was not possible in Camera1
© Google

28. The pipeline
But the aspect ratio has to be
same across all sizes
(at least to support Galaxy
devices)

29. The pipeline
Example CaptureRequest for camera previewing
// Preview request is built with builder pattern
previewRequestBuilder = cameraDevice?.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW) ?: return
// We can set multiple targets, i.e. show multiple previews
previewRequestBuilder.addTarget(Surface(imageTexture))
// Setting camera preferences
previewRequestBuilder[CaptureRequest.CONTROL_AF_MODE] = CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE
previewRequestBuilder[CaptureRequest.FLASH_MODE] = CaptureRequest.FLASH_MODE_OFF
previewRequest = previewRequestBuilder.build()
// Sending a PreviewRequest to current camera session
// The session will handle given PreviewRequest appropriately
captureSession?.setRepeatingRequest(previewRequest, captureCallback, backgroundHandler)

30. The pipeline
Notice the backgroundHandler?
the API manages all the background stuff for us
// Preview request is built with builder pattern
previewRequestBuilder = cameraDevice?.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW) ?: return
// We can set multiple targets, i.e. show multiple previews
previewRequestBuilder.addTarget(Surface(imageTexture))
// Setting camera preferences
previewRequestBuilder[CaptureRequest.CONTROL_AF_MODE] = CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE
previewRequestBuilder[CaptureRequest.FLASH_MODE] = CaptureRequest.FLASH_MODE_OFF
previewRequest = previewRequestBuilder.build()
// Sending a PreviewRequest to current camera session
// The session will handle given PreviewRequest appropriately
captureSession?.setRepeatingRequest(previewRequest, captureCallback, backgroundHandler)

31. Find the Difference
: Differences Between Camera1 and 2

32. Setting Up Preview / Connecting to Surface
camera = Camera.open(id) // This is the camera val callback = object: CameraDevice.StateCallback() {
override fun onOpened(camera: CameraDevice?) {
cameraDevice = camera // This is the cameraDevice
}
...
}
cameraManager.openCamera(id, callback, backgroundHandler)
Straightforward, is blocking current thread Gets device by callback, does its job by backgroundHandler

33. Setting Up Preview / Connecting to Surface
(camera ?: return).let {
it.setPreviewTexture(imageTexture)
// or you can do
// it.setPreviewDisplay(surfaceHolder)
it.startPreview()
}
val surface = Surface(imageTexture)
previewRequestBuilder = cameraDevice!!
.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW)
previewRequestBuilder.addTarget(surface)
cameraDevice!!.createCaptureSession(
listOf(surface, imageReader!!.surface),
object : CameraCaptureSession.StateCallback() {
override fun onConfigured(
session: CameraCaptureSession?) {
captureSession = session // This is the session
previewRequest = previewRequestBuilder.build()
captureSession?
.setRepeatingRequest(previewRequest, captureCallback,
backgroundHandler)
}
...
}
}, backgroundHandler)
Again, straightforward, is blocking current thread.
We have to manually set preview size to match
the surface’s size.
Gets session by callback, and we can start preview by calling
setRepeatingRequest.
Preview size is automatically chosen by the API to match the surface size.

34. Properties vs. CameraCharacteristics
Properties (Camera1)
- Mixed mutable/immutable properties
- things like previewSize are mutable;
things like supportedPreviewSizes are not
- You CAN call camera.setProperties() to
update camera settings
- You NEED to open a camera to use this
CameraCharacteristics (Camera2)
- Immutable properties only
- like the name says, you can only get the
characteristics of a CameraDevice
- You CAN’T update camera settings with
this class; it’s used only for checking
device features
- You DON’T NEED to open a camera

35. Querying Features
camera!!.parameters!!.supportedFlashModes cameraManager
.getCameraCharacteristics(cameraId)
.get(CameraCharacteristics.FLASH_INFO_AVAILABLE)
Can query features by accessing
Camera.Parameters
Need to open camera before querying
Can query features by CameraCharacteristics, using cameraId (not
cameraDevice) so no need to open cameraDevice

36. Setting Parameters
camera!!.parameters = camera!!.parameters.apply {
flashMode = Camera.Parameters.FLASH_MODE_ON
zoom = 125 // Sets zoom to x1.25, if supported
}
previewRequestBuilder[CaptureRequest.LENS_FOCUS_DISTANCE] = 9.50f
previewRequestBuilder[CaptureRequest.SENSOR_EXPOSURE_TIME] = 1000000000L / 30
previewRequest = previewRequestBuilder.build()
captureSession?.stopRepeating()
captureSession?.setRepeatingRequest(previewRequest, captureCallback,
backgroundHandler)
Sets parameters directly
When given unsupported values, it’ll throw an
error
Sets parameters by builder pattern
When given unsupported values, it’ll try its best to fit into supported values,
by rounding or clamping, etc

37. Setting Parameters
camera!!.parameters = camera!!.parameters.apply {
flashMode = Camera.Parameters.FLASH_MODE_ON
zoom = 125 // Sets zoom to x1.25, if supported
}
private fun <T> setPreviewOptions(
vararg options: Pair<CaptureRequest.Key<in T>, T>,
postProcess: () -> Unit = {}) {
for (option in options) {
previewRequestBuilder[option.first] = option.second
}
postProcess()
previewRequest = previewRequestBuilder.build()
captureSession?.stopRepeating()
captureSession?.setRepeatingRequest(
previewRequest, captureCallback, backgroundHandler)
}
...
setPreivewOptions(
CaptureRequest.LENS_FOCUS_DISTANCE to 9.50f,
CaptureRequest.SENSOR_EXPOSURE_TIME to 1000000000L / 30
)
Sets parameters directly
When given unsupported values, it’ll throw an
error
We can make an extension function to simplify this

38. Setting Parameters
camera!!.parameters = camera!!.parameters.apply {
flashMode = Camera.Parameters.FLASH_MODE_ON
zoom = 125 // Sets zoom to x1.25, if supported
}
It can be rather frustrating because only some
specific values are supported by the camera, so if I
request x1.25, the app will likely to crash on
Galaxy S9+
But Camera2 handles similar cases appropriately -
like what would we expect
Example supported zoom ratios (by Samsung Galaxy S9+ Camera)
Check out my Camera1-based OSP here :
https://github.com/shiftpsh/sensor-tester/releases

39. Setting parameters in Camera1 vs. 2
Camera1
- Parameters are global
- If we want to make a change, then it’s
changed globally. But since image
processing takes time, the API waits until
last image finishes processing and finally
change the settings
Camera2
- Parameters are set per request
- Image processing stages gets parameters
per request - the API don’t have to care
about the last image or whatsoever,
resulting improved frame rates

40. Setting parameters in Camera1
1234567
processing stages
request
params A
params B
current params: A
result

41. Setting parameters in Camera1
1234567
processing stages
request
params A
params B
If we change settings... It get mixed
current params: B
result

42. Setting parameters in Camera1
1222223
processing stages
request
params A
params B
current params: A
So Camera1 does this way … one by one
result

43. Setting parameters in Camera2
1 / A2 / A3 / A4 / B5 / B6 / B7 / B
processing stages
request
params A
params B
But in Camera2, parameters are baked into the requests
result

44. Setting parameters in Camera1 vs. 2
Camera1
Camera2
result
… resulting much higher speed

45. © Google

46. © Google

47. Taking Pictures
camera!!.takePicture(null, null) { bytes, _ ->
process(bytes)
}
imageReader = ImageReader.newInstance(largestSize.width, largestSize.height,
ImageFormat.JPEG, 2)
val onImageAvailableListener = ImageReader.OnImageAvailableListener { reader ->
process(image)
}
imageReader!!.setOnImageAvailableListener(
onImageAvailableListener, backgroundHandler)
val captureBuilder = cameraDevice!!
.createCaptureRequest(CameraDevice.TEMPLATE_STILL_CAPTURE)
captureBuilder.addTarget(imageReader!!.surface)
...
val captureCallback = object : CameraCaptureSession.CaptureCallback() {
override fun onCaptureCompleted(...) {...}
}
captureSession?.stopRepeating()
captureSession?.abortCaptures()
captureSession?.capture(
captureBuilder.build(), captureCallback, backgroundHandler)
One of the few functions that uses a callback.
Returns a ByteArray - RGB format
It uses an imageReader to capture image. Also uses a callback.
Returns an Image - YUV format (typically)

48. Implementing it in existing apps

49. API may be different but the flow is same
Open Camera Connect to Surface
Start Preview /
Repeating Request
Apply Filter, etc..
Capture Image
Close Camera

50. Speed comparison
Device Camera1 Camera2 Improvement
Google Pixel 2 (9) 87.7ms 35.1ms 150%
Samsung Galaxy S9+ (8.0) 71.7ms 27.9ms 157%
LG G7 ThinQ (8.0) 37.3ms 11.5ms 224%
Opening the camera
* Averaged result of 10 runs

51. Speed comparison
Device Camera1 Camera2 Improvement
Google Pixel 2 (9) 810.3ms 406.2ms 99%
Samsung Galaxy S9+ (8.0) 477.2ms 351.1ms 36%
LG G7 ThinQ (8.0) 616.4ms 382.9ms 61%
Taking a single full resolution picture
* Averaged result of 10 runs, right after callback

52. Migration Strategy
- Not all devices work well with Camera2, even if its API level ≥ 21
- It’s up to vendors - if they wrote good enough HAL(hardware abstraction
layer) for Camera2 then it’ll work well, otherwise not

53. Migration Strategy
- We can test some devices if they supports
Camera2 well and make a whitelist
- Or we can ask our users to opt-in, like
putting an option to enable advanced
camera in ‘experimental’ section like many
other apps do

54. Q&A