Swift & Fika talk discussing an overview of machine learning tools for Apple platforms, covering short examples using Vision, Create ML, Turi Create, & Core ML.

1. Navigating the
!
ML Landscape
Meghan Kane, @meghafon
developer @novoda, Berlin
Swift & Fika, 9/2018

4. !
before we pack our bags..

5. why use machine learning .. at all?

6. ML tasks
→ detection: rectangles, face, barcode, text
→ classification: image, text, activity, column data
→ style transfer
→ image similarity
→ object detection
→ natural language processin
→ regression

8. Advantages of on device inference
1. user privacy
2. low latency user experience

9. Let's start our road trip

11. Classify
board game
pieces

12. ML approach
Label: "Border"
1. Detect game box (rectangle
detection)
2. Classify Agricola piece within
rectangle (image classification)

13. Capabilities
... built right into the Vision
framework, no model training
needed
→ detection: rectangles, face,
barcode, text
→ object tracking
→ image alignment

14. How?
1. create handler
2. create task specific request
3. send request to handler
4. handle results

15. // 1. Create handler
let handler = VNImageRequestHandler(cgImage: image,
orientation: orientation,
options: [:])

16. // 2. Create request
let request = VNDetectRectanglesRequest(completionHandler: self.handleDetectedRectangles)

17. // 3. Send request to handler
do {
try handler.perform([request])
} catch let error as NSError {
// handle error
return
}

18. // 4. Handle results
func handleDetectedRectangles(request: VNRequest?, error: Error?) {
if let results = request?.results as? [VNRectangleObservation] {
// Do something with results [*bounding box coordinates*]
}
}

19. ML approach
Label: "Border"
1. Detect game box (rectangle
detection)
2. Classify Agricola piece within
rectangle (image classification)

20. Why restrict input image to the box?
→ easier to train an accurate model
→ faster to collect image data

21. Capabilities
In Xcode playground, train custom
model for:
→ image classification
→ text classification
→ classification & regression of
column data

22. Collect Data
→ Collect images representative of real world use
cases
→ Vary angle & lighting
→ >10 images per label, but ideally more
→ Equal # images for each label
→ Recommended: >299x299 pixels

23. Collecting Data Quickly
// Extract .jpg frames from .mov @ 5 frames/sec
ffmpeg -i stone.mov -r 5 data/stone/stone_%04d.jpg

24. Prepare Data
→ Split data: 80% train / 20% test

28. what's happening behind the scenes?

29. Transfer
Learning

31. Evaluate

33. Save model

34. Capabilities
→ perform predictions using
model
→ quantized weights (32 bit -> 16,
8, 4... bit)
→ perform batch predictions
→ create custom model layer

35. Vision + Core ML
1. create Vision Core ML model
2. create handler
3. create task specific request
4. send request to handler
5. handle results

36. // 1. Create Vision Core ML model
let model = AgricolaPieceClassifier()
guard let visionCoreMLModel = try? VNCoreMLModel(for: model.model) else { return }

37. // 2. Create handler
let handler = VNImageRequestHandler(cgImage: cgImage, orientation: cgImageOrientation)

38. // 3. Create request
let request = VNCoreMLRequest(model: visionCoreMLModel,
completionHandler: self.handleClassificationResults)

39. // 4. Send request to handler
do {
try handler.perform([request])
} catch let error as NSError {
// handle error
return
}

40. // 5. Handle results
func handleClassificationResults(request: VNRequest?, error: Error?) {
if let results = request?.results as? [VNClassificationObservation] {
// Do something with results
}
}

41. Problem solved
✅
Label: "Border"
1. Detect game box (rectangle
detection)
2. Classify Agricola piece within
rectangle (image classification)

42. Next
challenge...

44. Capabilities
...robust Python training
framework
→ style transfer
→ activity classification
→ image similarity
→ recommmendations
→ object detection

45. Setting up python
virtualenv venv
source venv/bin/activate
pip install requests==2.18.4 turicreate==5.0b2 jupyter

46. Train style transfer
jupyter notebook