CameraX is a new camera library that provides a simpler way to access camera features on Android. It consolidates the camera APIs and works on older Android versions back to L. CameraX includes use cases like camera preview, image analysis and image capture. It can also be used with ML Kit for on-device image labeling with just a few lines of code. While still in alpha/beta stage, CameraX and ML Kit make integrating machine learning into camera apps on Android much easier.

1. CameraX, meets ML
Wonyoung Choi
Singapore Tourism Board
@TORU_0239
GDG DevFest Songdo 2019, at 30 Nov, 2019

2. I am…
Wonyoung Choi, a.k.a Toru
Android Advocate,
Singapore Tourism Board
@TORU_0239
GitHub: TORU239

3. CameraX

4. Introduction
• Announced in Google I/O 2019
• Jetpack Support Library
• Easier Approach to leverage Camera2

5. Consolidation
• Lifecycle-aware component
• Backward compatibility till L
• Use-case based approaches
• Resolving device-compatibility issues

6. Use Cases
• Preview
• Image Analysis
• Image Capture

7. Preview

8. <TextureView
android:id="@+id/view_finder"
android:layout_width="match_parent"
android:layout_height="match_parent"/>

9. val previewConfig = PreviewConfig.Builder().apply {
setLensFacing(CameraX.LensFacing.BACK)
setTargetAspectRatio(AspectRatio.RATIO_4_3)
setTargetRotation(viewFinder.display.rotation)
}.build()

10. val previewConfig = PreviewConfig.Builder().apply {
setLensFacing(CameraX.LensFacing.BACK)
setTargetResolution(
Size(metrics.widthPixels,metrics.heightPixels)
)
setTargetRotation(viewFinder.display.rotation)
}.build()

11. val previewConfig = PreviewConfig.Builder().apply {
setLensFacing(CameraX.LensFacing.BACK)
setTargetAspectRatio(AspectRatio.RATIO_4_3)
setTargetResolution(
Size(metrics.widthPixels,metrics.heightPixels)
)
setTargetRotation(viewFinder.display.rotation)
}.build()
How about this?

12. val previewConfig = PreviewConfig.Builder().apply {
setLensFacing(CameraX.LensFacing.BACK)
setTargetAspectRatio(AspectRatio.RATIO_4_3)
setTargetResolution(
Size(metrics.widthPixels,metrics.heightPixels)
)
setTargetRotation(viewFinder.display.rotation)
}.build()
Crash!

13. Image Analysis

14. val imageAnalysisConfig = ImageAnalysisConfig.Builder().apply {
setLensFacing(CameraX.LensFacing.BACK)
setImageReaderMode(ImageAnalysis.ImageReaderMode
.ACQUIRE_LATEST_IMAGE)
setTargetRotation(viewFinder.display.rotation)
}.build()

15. val executor = Executors.newSingleThreadExecutor()
imageAnalyzer = ImageAnalysis(imageAnalysisConfig).apply {
setAnalyzer(
executor,
ImageAnalysis.Analyzer { image, rotationDegrees ->
// Do whatever you want.
})
}

16. Analyzer
• Interface for analysing images
• Implement analyze() for use

17. Image Proxy
• Analogous interface as Image
• Informations on image contained

18. Image Capture

19. val captureConfig = ImageCaptureConfig.Builder().apply {
setLensFacing(CameraX.LensFacing.BACK)
setCaptureMode(ImageCapture.CaptureMode.MIN_LATENCY)
setTargetResolution(Size(metrics.widthPixels, metrics.heightPixels))
setTargetRotation(viewFinder.display.rotation)
}
.build()
val imageCapture = ImageCapture(captureConfig)

20. Binding to lifecycle
• Only few codes required to bind.

21. CameraX.bindToLifecycle(
viewLifecycleOwner,
preview,
imageCapture,
imageAnalyzer
)

22. CameraX.unbindAll()

23. Interact
• Taking picture by an api

24. imageCapture.takePicture(,,,)

25. Use Cases
• Preview
• Image Analysis
• Image Capture

26. Beware
• Alpha Stage
• Particular things on particular devices

27. ML Kit

28. What is ML Kit?
• Google’s Machine Learning SDK for Mobile
• Approachable by few lines of code

29. Vision API
• Image Labelling
• Text Recognition
• Barcode Scanning
• Face Detection
• Landmark Detection
• Object Detection and Tracking
• AutoML Vision Edge

30. Image Labelling
• Recognising image without additional metadata
• Deriving entities with confidence level

31. Image Labelling
On-Device Model
Cloud Model
ML Kit
Camera

32. Image Labelling
On-Device Model
Cloud Model
ML Kit
Camera
400+ categories
10000+ categories

33. Beware
• Beta Stage
• Changes on API

34. AutoML

35. Introduction
• Suites of ML Products
• Fitted to environment with less expertises

36. AutoML Vision Edge
• Custom Image Labelling
• Narrower domain of concepts

37. AutoML Vision Edge

38. val firebaseAutoMLLocalModel = FirebaseAutoMLLocalModel.Builder()
.setAssetFilePath("manifest.json")
.build()

39. val labeller = FirebaseVision.getInstance()
.getOnDeviceAutoMLImageLabeler(
FirebaseVisionOnDeviceAutoMLImageLabelerOptions
.Builder(firebaseAutoMLLocalModel)
.setConfidenceThreshold(0.5F)
.build()
)

40. labeller.processImage(image)
.addOnSuccessListener { labels ->
// Task completed successfully
// ...
}
.addOnFailureListener { e ->
// Task failed with an exception
// ...
}

41. val image: FirebaseVisionImage
try {
image = FirebaseVisionImage.fromFilePath(context, uri)
} catch (e: IOException) {
e.printStackTrace()
}

42. val image = FirebaseVisionImage.fromBitmap(bitmap)

43. dependencies {
// ...
implementation 'com.google.firebase:firebase-ml-vision:24.0.1'
implementation 'com.google.firebase:firebase-ml-vision-image-label-model:19.0.0'
}
apply plugin: 'com.google.gms.google-services'
build.gradle in app level

44. android {
...
aaptOptions {
noCompress "tflite"
}
}
Don’t forget to put this code!

45. Sample Project -
Food Scanner!

46. Further Studies

47. New capabilities
• Tap to focus
• Zoom
• Device Rotation Info
• Flash Availability
• And so on!

48. Extensions
• Night mode
• HDR
• Few lines of code required only!

49. Thank You!
Wonyoung Choi
@TORU_0239