CameraX with MLKit and AutoML

2. CameraX, meets MLKit and AutoML
Wonyoung Choi
Singapore Tourism Board

3. Contents
1. Biography
2. CameraX
3. MLKit
4. AutoML
5. Example

4. Biography

5. Wonyoung Choi, a.k.a Toru
Android Advocate, Singapore Tourism Board
@TORU_0239
GitHub: TORU239

6. CameraX

7. Introduction
• Announced in Google I/O 2019
• Jetpack Support Library
• Easier Approach to leverage Camera2

8. • Lifecycle-aware component
• Backward compatibility till L
• Use-case based approaches
• Resolving device-compatibility
issues
Features

9. User cases
• Preview
• Image Analysis
• Image Capture

10. Preview Image
Analysis
Capture

11. <TextureView
android:id="@+id/view_finder"
android:layout_width="match_parent"
android:layout_height="match_parent"/>

12. val previewConfig = PreviewConfig.Builder().apply {
setLensFacing(CameraX.LensFacing.BACK)
setTargetAspectRatio(AspectRatio.RATIO_4_3)
setTargetRotation(viewFinder.display.rotation)
}.build()

13. val previewConfig = PreviewConfig.Builder().apply {
setLensFacing(CameraX.LensFacing.BACK)
setTargetResolution(
Size(metrics.widthPixels,metrics.heightPixels)
)
setTargetRotation(viewFinder.display.rotation)
}.build()

14. val previewConfig = PreviewConfig.Builder().apply {
setLensFacing(CameraX.LensFacing.BACK)
setTargetAspectRatio(AspectRatio.RATIO_4_3)
setTargetResolution(
Size(metrics.widthPixels,metrics.heightPixels)
)
setTargetRotation(viewFinder.display.rotation)
}.build()
How about this?

15. val previewConfig = PreviewConfig.Builder().apply {
setLensFacing(CameraX.LensFacing.BACK)
setTargetAspectRatio(AspectRatio.RATIO_4_3)
setTargetResolution(
Size(metrics.widthPixels,metrics.heightPixels)
)
setTargetRotation(viewFinder.display.rotation)
}.build()
Crash!

16. Preview Image
Analysis
Capture

17. val imageAnalysisConfig = ImageAnalysisConfig.Builder().apply {
setLensFacing(CameraX.LensFacing.BACK)
setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE)
setTargetRotation(viewFinder.display.rotation)
}.build()
val executor = Executors.newSingleThreadExecutor()
imageAnalyzer = ImageAnalysis(imageAnalysisConfig).apply {
setAnalyzer(
executor,
ImageAnalysis.Analyzer { image, rotationDegrees ->
// Do whatever you want.
})
}

18. ImageProxy?
• Analogous Interface as Image
• Contains informationsAPI

19. Analyzer?
• Interfaces for analysing images
• Implement analyze() to useAPI

20. Preview Image
Analysis
Capture

21. val captureConfig = ImageCaptureConfig.Builder().apply {
setLensFacing(CameraX.LensFacing.BACK)
setCaptureMode(ImageCapture.CaptureMode.MIN_LATENCY)
setTargetResolution(Size(metrics.widthPixels, metrics.heightPixels))
setTargetRotation(viewFinder.display.rotation)
}
.build()
val imageCapture = ImageCapture(captureConfig)

22. Binding to Lifecycle
• Few codes required to bind.
API

23. CameraX.bindToLifecycle(
viewLifecycleOwner,
preview,
imageCapture,
imageAnalyzer
)

24. CameraX.unbindAll()

25. Interact
• Taking picture using one api
API

26. imageCapture.takePicture(,,,)

27. `
Configure Bind Interact!

28. Caution
API
• Alpha stage
• Particular thing on particular
Devices

29. MLKit

30. Introduction
• Mobile SDK for both platform
• Easy-to-use,
with few lines of code
• Approachable for everyone

31. Vision API • Image Labelling
• Face Recognition
• Barcode Scanning
• Face Detection
• Landmark Recognition

32. Image Labelling • Enables user to recognise
image without additional
metadata
• Deriving entities with
confidence level
• On-device or Cloud API

33. Image Labelling
• 400+ categories for On-
Device
• 10,000+ for Cloud API

34. Image Labelling
• What if we want to use our
custom model?

35. Please wait for a while 🤣

36. Caution
API
• Beta stage
• Changes in the future

37. AutoML

38. AutoML
• Suites of ML Products
• Fitting to environment with less
expertises

39. AutoML • AutoML Vision
• AutoML Video Intelligence
• AutoML Natural Languages
• AutoML Translation
• AutoML Table

40. AutoML Vision Edge
• Custom Image Labelling
• Narrower domain of concepts
• Follows policies of Google
Cloud

41. AutoML Vision Edge
• Able to be found and linked
from Firebase Console

42. Go along together!

43. val firebaseAutoMLLocalModel = FirebaseAutoMLLocalModel.Builder()
.setAssetFilePath("manifest.json")
.build()

44. val labeller = FirebaseVision.getInstance().getOnDeviceAutoMLImageLabeler(
FirebaseVisionOnDeviceAutoMLImageLabelerOptions
.Builder(firebaseAutoMLLocalModel)
.setConfidenceThreshold(0.5F)
.build()
)

45. labeller.processImage(image)
.addOnSuccessListener { labels ->
// Task completed successfully
// ...
}
.addOnFailureListener { e ->
// Task failed with an exception
// ...
}

46. val image: FirebaseVisionImage
try {
image = FirebaseVisionImage.fromFilePath(context, uri)
} catch (e: IOException) {
e.printStackTrace()
}

47. val image = FirebaseVisionImage.fromBitmap(bitmap)

48. aaptOptions {
noCompress “tflite”
}
Don’t forget to put into build.gradle!

49. `
Capture Image
Labelling
Detect!

50. Example Project:
Cebu Food Detector!

51. Let’s go and find LECHON!

52. WE HAVE,
ONE MORE THING! 😆

53. Furthermore
• Plan to release in beta in Dec
• More apis not showcased hereAPI

54. New capabilities • Tap to focus
• Zoom
• Device rotation Info
• Flash Availability
• And so on!

55. Extensions
• Night mode
• HDR
• Few lines needed only!
• Restriction for API

56. https://developer.android.com/training/camerax
https://firebase.google.com/docs/ml-kit
https://cloud.google.com/automl
https://github.com/android/camera-samples/tree/master/CameraXBasic
https://github.com/TORU0239/CebuDevfest19

57. Thank you!