Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fusing Artificial Intelligence with Augmented Reality on Android - 1 Feb. 2019

493 views

Published on

Here is an overview of how to use open-source libraries and existing Google codelabs to synthesize augmented reality and artificial intelligence into a single application. The app performs object detection to guess a 2D bounding box around a detected object. The center of the box is provided to a hit test that, if successful, will be used as the coordinate to place a 3D model in the AR environment at that location.

The technologies involved are native Android in Java, Tensorflow Lite, Sceneform AR library, Bazel build, Gradle build, and contains extra instructions for Windows users.

Besides just wiring up the libraries to work together, there are other pitfalls and gotchas that must be considered, and those are touched upon near the end of the deck.

YouTube video from 1 Feb. 2019: TBD
GitHub repo of final project: TBD

Published in: Software
  • How Bookies CHEAT and How to beat them?■■■ http://scamcb.com/zcodesys/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Fusing Artificial Intelligence with Augmented Reality on Android - 1 Feb. 2019

  1. 1. Fusing AI with AR By Stephen Wylie Windy City DevFest - Chicago Feb. 1, 2019
  2. 2. About Me ● Google Developer Expert in Machine Learning ● Fix old arcade machines & computers, and pinballs of all eras ● Love fancy men’s clothing & vintage TV game shows ● Lots of patents in this area
  3. 3. AI + AR: Does anyone do this themselves? Yes, of course! But with help… Vehicle Recognition & Auto Loans ● Blippar - AR/VR company ● Capital One - AR feature of Auto Navigator ● USAA - Loans & insurance APIs & Frameworks Are Your Friends! Retail ● Burberry - Customize items, detect counterfeits ● Cup Noodles - Turn cups into musical instruments Sports ● Homecourt- Practice basketball, get coaching ● MLB - ID players, show stats
  4. 4. Key Terms ● MobileNet, SqueezeNet, Inception, ResNet Neural networks optimized for computer vision tasks ● Tensorflow Lite Inference engine optimized to run .tflite models on mobile devices ● MLKit APIs that enable specific Machine Learning use cases on devices or cloud ● ARCore Cross-platform augmented reality SDK using OpenGL for motion tracking, scene building ● Sceneform AR SDK for Android saving you from learning OpenGL
  5. 5. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFragment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
  6. 6. Install Prerequisites ● Latest: ● Java JDK 10 (not 11, not 8) ● Building in Windows? You’ll need ~5GB worth of: ○ Visual C++ Build Tools - visualcppbuildtools_full.exe ■ Windows 10 SDK 10.0.10240 ■ .NET Framework SDK ○ A real shell like MSYS2 ● Pick your poison! Download Tensorflow source from GitHub or Docker - Android Studio - Android API level - Android NDK - Gradle - Bazel
  7. 7. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFragment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
  8. 8. Start Modifying Code Code root: tensorflow/tensorflow/lite/examples/android/app ● Consider copying to a different folder ● Add Sceneform dependencies to Gradle build ● Update Gradle build to latest Tools, API >= 26 ● Add AR permissions to Android Manifest ● Make new Layout incorporating ArFragment ● Use this Layout in your CameraActivity ○ Or make it a separate class ○ Update Manifest with new Activity ○ Ensure DetectorActivity extends it Gradle downloads ML model assets
  9. 9. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFragment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
  10. 10. Build with Bazel ● Define androidsdk and androidndk path in WORKSPACE ● Omit API level & build tools version ● cd to tensorflow/ ● Run bazel build //tensorflow/lite/examples/android:tflite_demo ● Funky Windows business in “Windows Kits” directory ● Pray hard!
  11. 11. What Success Looks Like
  12. 12. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFragment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
  13. 13. Add Augmented Reality Code ● Can’t use Bazel anymore ○ Incorporate object detection model into src/main/jniLibs ○ Set parameter for CPU architecture in Gradle ● Sceneform Plugin or Gradle Instruction ○ Sceneform codelab calls for 1.4; Android Studio plugin is 1.6 ○ Best to just use Gradle instructions to create SFB assets ● Add augmented reality code to MultiBoxTracker
  14. 14. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFragment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
  15. 15. Pitfalls of AI + AR: Battery life “Turn your phone into an incendiary device! Run AI & AR at the same time.” Throttling AI is key.
  16. 16. Pitfalls of AI + AR: Depth & size of object Projecting 3D object in AR requires good understanding of 3D space: ● Size of object, in pixels, proportional to the camera ● Size of object, in real life, to position 3D object correctly ● Positioning 3D object visible to user without maneuvering ● Improvement: Correlate bounding box with hit detection
  17. 17. Pitfalls of AI + AR: Sensor drift Object out of the viewport? ● AR relies on point clouds, hardware sensors ● Hardware sensors likely to drift with lots of motion ● Objects anchored to world space likely to wander off Solution? ● Ensure scene has good contrast, enough corners to see every frame ● Attach objects to anchors ● Scan enough of the scene ahead of time before drawing AR
  18. 18. Pitfalls of AI + AR: Tracking multiple objects ● Object detections could be missed, false, or switch IDs ● Discover correlations between objects moving on camera ○ Pairwise dependencies ○ Uniform or non-uniform hypergraph ● Multiple object tracking is easier with the tensorflowlite_demo library ● No library? Gotta code it yourself! Mitigating This ● Consumption issues? Minimize anchors as you instantiate objects ● Take the time to set up NDK and/or libraries
  19. 19. Thank You @SWebCEO goshtastic.blogspot.com github.com/mrcity/mlworkshop Slideshare: StephenWylie3

×