Recommended
Recommended
More Related Content
Similar to Fusing AI With AR for Android - 9/28/2019
Similar to Fusing AI With AR for Android - 9/28/2019 (20)
Recently uploaded
Recently uploaded (20)
Fusing AI With AR for Android - 9/28/2019
- 1. Stephen Wylie GDG Dallas @SWebCEO Fusing AI with AR NH GDG DevFest 9/28/2019
- 2. NH GDG DevFest ●GDE in Machine Learning ●Software Engineer @ rewardStyle ●Fix old arcade machines & computers, and pinballs of all eras ●Love fancy men’s clothing & vintage TV game shows ●Lots of patents in this area About Me
- 3. AI + AR: Does anyone do this themselves?
- 4. NH GDG DevFest Sports ● Homecourt- Practice basketball, get coaching ● MLB - ID players, show stats AI + AR: Does anyone do this themselves? Yes, of course! But with help... APIs & Frameworks Are Your Friends! Retail ● Burberry - Customize items, detect counterfeits ● Cup Noodles - Turn cups into musical instruments ● Wayfair, IKEA - Preview items in your own house Vehicle Recognition & Auto Loans ● Blippar - AR/VR company ● Capital One - AR feature of Auto Navigator ● USAA - Loans & insurance
- 5. Key Terms
- 6. Key Terms In AI ● MobileNet, SqueezeNet, Inception, ResNet Neural networks optimized for computer vision tasks ● Tensorflow Lite Inference engine optimized to run .tflite models on mobile devices ● MLKit APIs that enable specific Machine Learning use cases on devices or cloud In AR ● ARCore Cross-platform augmented reality SDK using OpenGL for motion tracking, scene building ● Sceneform AR SDK for Android saving you from learning OpenGL
- 8. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFra gment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
- 9. NH GDG DevFest ●Latest: ●Java JDK 10 (not 11, not 8) ●Building in Windows? You’ll need ~5GB worth of: ○Visual C++ Build Tools - visualcppbuildtools_full.exe ■Windows 10 SDK 10.0.10240 ■.NET Framework SDK ○A real shell like MSYS2 ●Pick your poison! Download Tensorflow source from GitHub or Docker Install Prerequisites ●Android Studio ●Android API level ●Android NDK ●Gradle ●Bazel
- 10. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFra gment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
- 11. NH GDG DevFest Code root: tensorflow/tensorflow/lite/examples/android/app ●Consider copying to a different folder ●Add Sceneform dependencies to Gradle build ●Update Gradle build to latest Tools, API >= 26 ●Add AR permissions to Android Manifest ●Make new Layout incorporating ArFragment ●Use this Layout in your CameraActivity ○Or make it a separate class ○Update Manifest with new Activity ○Ensure DetectorActivity extends it Gradle downloads ML model assets Start Modifying Code
- 12. Code Thus Far
- 13. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFra gment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
- 14. NH GDG DevFest ●Define androidsdk and androidndk path in WORKSPACE ●Omit API level & build tools version ●cd to tensorflow/ ●Run bazel build //tensorflow/lite/examples/android:tflite_demo ●Funky Windows business ●Pray hard! Build with Bazel
- 16. Code Thus Far
- 17. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFra gment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
- 18. NH GDG DevFest ●Can’t use Bazel anymore ○Incorporate object detection model into src/main/jniLibs ○Set parameter for CPU architecture in Gradle ●Sceneform Plugin or Gradle Instruction ○Sceneform codelab calls for 1.4; Android Studio plugin is 1.6 ○Best to just use Gradle instructions to create SFB assets ●Add augmented reality code to MultiBoxTracker Add Augmented Reality Code
- 19. Final Product
- 20. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFra gment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
- 21. Pitfalls of AI + AR
- 22. Battery Life A Pitfall of AI + AR “Turn your phone into an incendiary device! Run AI & AR at the same time.” Throttling AI is key.
- 23. NH GDG DevFest Depth & size of object Another Pitfall of AI + AR Projecting 3D object in AR requires good understanding of 3D space: ●Size of object, in pixels, proportional to the camera ●Size of object, in real life, to position 3D object correctly ●Positioning 3D object visible to user without maneuvering ●Improvement: Correlate bounding box with hit detection
- 24. NH GDG DevFest Sensor drift Yet Another Pitfall of AI + AR Object out of the viewport? ●AR relies on point clouds, hardware sensors ●Hardware sensors likely to drift with lots of motion ●Objects anchored to world space likely to wander off Solution? ●Ensure scene has good contrast, enough corners to see every frame ●Attach objects to anchors ●Scan enough of the scene ahead of time before drawing AR
- 25. NH GDG DevFest Tracking Multiple Objects And Yet Another Pitfall of AI + AR ●Object detections could be missed, false, or switch IDs ●Discover correlations between objects moving on camera ○Pairwise dependencies ○Uniform or non-uniform hypergraph ●Multiple object tracking is easier with the tensorflowlite_demo library ●No library? Gotta code it yourself! Mitigating This ●Consumption issues? Minimize anchors as you instantiate objects ●Take the time to set up NDK and/or libraries
- 27. NH GDG DevFest ●Garbage in, garbage out ● Need at least 100 samples per category ○ Kaggle ○ Record your own videos ● Keras imagedatagenerator ● Label your own bounding boxes Collecting Data
- 28. NH GDG DevFest Building Your Model Tensorflow Cloud AutoML + Firebase
- 29. NH GDG DevFest ●tflite_convert binary available in Github code ○ Convert a GraphDef ○ Convert a SavedModel ○ Convert a Keras model ● Or, use AutoML Vision Object Detection Beta ○ AutoML Vision Edge in ML Kit - Export to Android using Firebase ○ Models in this form are quantized - i.e. ints, not floats ○ Small, edge device compatible; may lose accuracy Converting Your Model
- 30. Thank You! ML Summit ‘19 #MLSummit @SWebCEO goshtastic.blogspot.com github.com/mrcity/mlworkshop Slideshare: StephenWylie3