Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deeper into ARKit with CoreML and Turi Create


Published on

Have you ever tried to make something cool and fun with ARKit, only to find out there is a missing piece? Then this talk is for you. I struggled to make my first AR app (Notable Me), but CoreML and Turi Create was there for me. This framework and tool allowed me to create something I never knew I could make.

I will share all the lessons I learned from developing this app, focusing on how to utilize machine learning into an ARKit app. Also how to unlock hidden features of Turi Create, Appleโ€™s Open Source tool for easily creating custom ML models, to drastically improve the quality.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Deeper into ARKit with CoreML and Turi Create

  1. 1. Deeper into ARKit with CoreML and Turi Create Soojin Ro
  2. 2. Notable Me
  3. 3. AR Image Tracking
  4. 4. Machine Learning Face Realtime Detection Face Contour Detection Artistic Style Transfer AVFoundation
  5. 5. Face Realtime Detection AVMetadataFaceObject let session = AVCaptureSession() let metadataOutput = AVCaptureMetadataOutput() if session.canAddOutput(metadataOutput) { session.addOutput(metadataOutput) if metadataOutput.availableMetadataObjectTypes.contains(.face) { metadataOutput.metadataObjectTypes = [.face] } } else { setupResult = .configurationFailed session.commitConfiguration() return } var outputSynchronizer: AVCaptureDataOutputSynchronizer? outputSynchronizer = AVCaptureDataOutputSynchronizer(dataOutputs: [videoDataOutput, metadataOutput]) outputSynchronizer?.setDelegate(self, queue: dataOutputQueue)
  6. 6. Face Realtime Detection AVCaptureDataOutputSynchronizerDelegate { func dataOutputSynchronizer(_ synchronizer: AVCaptureDataOutputSynchronizer, didOutput synchronizedDataCollection: AVCaptureSynchronizedDataCollection) { let syncedMetadata = synchronizedDataCollection.synchronizedData(for: metadataOutput) as? AVCaptureSynchronizedMetadataObjectData if let firstObject = syncedMetadata?.metadataObjects.first, let videoConnection = videoDataOutput.connection(with: .video), let faceObject = videoDataOutput.transformedMetadataObject(for: firstObject, connection: videoConnection) { print(faceObject.bounds) // This is the frame of the face in the video buffer coordinate } }
  7. 7. Face Realtime Detection Enhancing User Experience
  8. 8. Face off
  9. 9. Face Contour Detection MLKit for Firebase Vision Framework
  10. 10. Speed? Accuracy?
  11. 11. Artistic Style Transfer @DmitryUlyanovML + =
  12. 12. Model-based Approach Crafting machine learning models from scratch Requires intermediate to expert knowledge
  13. 13. Image Classifier Text Classifier Sound Classifier Activity Classifier Recommender Style Transfer Object Detection Image Similarity Create ML Task-based Approach Easily create a model from predefined tasks For ML beginners (like me !)
  14. 14. Swift Python macOS macOS, Linux, Win10 external GPU Linux + nvidia GPU Full GUI tool Open Source Style Transfer! Language OS GPU Support Platform Create ML
  15. 15. Preparing Data Style Images Content Images Test Images
  16. 16. Hmm... ๐Ÿค” + =
  17. 17. Hmm... ๐Ÿค” + = ,
  18. 18. src/python/turicreate/toolkits/style_transfer/ style_loss_mult [float, float, float, float] finetune_all_params Boolean use_augmentation Boolean input_shape Boolean the coefficients of convolution layers whether to train feature extractor the size of content & style images augment training data set (selfie images) Hyperparameters! Undocumented Detail information below
  19. 19. Gist!
  20. 20. style_1: [0, 1], style_2: [0, 1], โ€ฆ GPU GPUGPU --style_1=0.1 --style_2=0.3 --style_1=0.6 --style_2=0.8 --style_1=0.7 --style_2=0.4 Training โ˜• Amazon SageMaker, Google Cloud ML Engine
  21. 21. + =
  22. 22. + =
  23. 23. NotableMeStyle Transfer.mlmodel
  24. 24. do { let model = NotableMeStyleTransfer() let output = try model.prediction(image: imageBuffer, index: style.mlIndex) return CIImage(cvImageBuffer: output.stylizedImage) } catch { return nil } Core
  25. 25. func makeOutlineNode(from bill: Bill) -> SCNNode { let geometry = SCNPlane(size: bill.outlineSize) geometry.firstMaterial?.diffuse.contents = bill.outlineImage let node = SCNNode(geometry: geometry) = PortraitNode.PortraitOutlineNodeName node.eulerAngles.x = -.pi / 2 return node } ARAnchor SCNNode extension MainViewController: ARSCNViewDelegate { func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? { if let imageAnchor = anchor as? ARImageAnchor { //return an appropriate SCNNode for your anchor } return nil } }
  26. 26. Face Realtime Detection Face Contour Detection Artistic Style Transfer AR Image Tracking ๐Ÿฅณ
  27. 27. Thank you!