An overview of using AI for positive impact on the Azure Cognitive Services team

1. AI for Good @
Deep Art Generation, Unsupervised Object Detection,
and Distributed Intelligent Microservices
Mark Hamilton
marhamil@microsoft.com

2. Overview
 AI for Cultural Institutions: The MET
 GAN image synthesis
 Reverse Image Search
 Intelligent Image Search
 AI for Earth: The Snow Leopard Trust
 Unsupervised Object Detection
 LIME
 AI for Accessibility: Seeing AI
 Currency Detection
 The Tech Stack
 Spark, MMLSpark, Kubernetes, Microservices
Seeing AI

3. AI for Cultural Institutions

4. Celebrating 2 years of Open Access at The MET
4
https://gen.studio
 In 2016 The MET Released 400k
Images under Open Access
 This past Winter the MET released a
new subject-keyword dataset of
image annotations
 MIT, The MET, and Microsoft
participated in a 3-day hackathon to
create intelligent experiences using
the new collection

5. Goals:
Create new
works of art
Use new work
to explore
existing art
Explore further
with intelligent
search
Generative
Adversarial
Networks
Reverse Image
Search
Elasticsearch
with Cognitive
Services
Needed Technologies:

6. Generative Adversarial Art
https://gen.studio
Real or
Generated?
Noise
Vector
Generator
Generated
Image
Training Data
Discriminator
Real or
Generated?

7. Custom Reverse Image Search
Filters from Zeiler + Fergus 2013
Query
Image
ResNet
Featurizer
Deep
Features
Closest
Match
Fast Nearest
Neighbor
Lookup
MMLSpark SparkML LSH or Annoy

8. Example Nearest NeighborsQueryImages
Nearest
Neighbors

9. Intelligent Search Index
 Pipe images through
Computer Vision API
to annotate image for
searching
 Stream images and
intelligent
annotations to Azure
Search
A picture
containing a
person
A picture
containing a
glass, cup
A fish
swimming
underwater
Query
Image:
Describe
Image
Output:
Deep
Feature
Nearest
Neighbors:

10. End Application: Gen Studio
10
https://gen.studio

11. AI for Earth

12. Endangered Status Matters

13. Classifying all 1.3
million images will
take 20k hours
Remote Camera Traps

15. Creating a labelled Training Dataset

16. Creating a labelled Training Dataset

17. Input Images
Low Level
Features
Mid Level
Features
Logistic
Regression
Output Classes
Filters from Zeiler + Fergus 2013
Any SparkML
Model
Other
Leopard
Predictions
Transfer Learning with ResNet 50

18. Performance
With Deep Featurization,
Augmentation, and
Temporal Ensembling
Without Deep
Featurization
Accuracy 65.6% Accuracy 94.7%

19. Goal: Identify Individual Leopards
Source: HotSpotter - Patterned Species Instance Recognition

20. Automating Detection with LIME on Spark

21. LIME on Spark

22. End to End Architecture

23. Results
Human Labels
Unsupervised
FRCNN Outputs Human Labels
Unsupervised
FRCNN Outputs

24. AI for Accessibility
Seeing AI

25. Currency Identification

26. A Familiar Architecture…
Union +
Distinct
MobileNet
Logistic
Regression
Bing
Image
Search
Deep Features
Azure Machine
Learning +
Spark Serving
Queries
1 Dollar
5 Dollars
10 Dollars
20 Dollars
Labelled Images
$1
$5
$2
0
Prep Data Train Deploy

27. The Tech Stack

28.  A fault-tolerant distributed
computing framework
 Map Reduce + SQL
 Whole program optimization +
query pushdown
 Elastic
 Scala, Python, R, Java, Julia
 ML, Graph Processing, Streaming
Driver
Worker
110
0.9
0
20
40
60
80
100
120
Running Time(s)
Hadoop Spark
Worker Worker
Data Source (ADLS, Blob, Local, Cosmos, etc)
Age:
Int
Name:
String
15 Bob
25 Alice
33 Mary
Age:
Int
Name:
String
1 Sam
2 Claire
53 Tina
Age:
Int
Name:
String
25 Bob
25 Tom
66 Ted
● Code
● Data

29. ML
 High level library for distributed
machine learning
 More general than SciKit-Learn
 All models have a uniform
interface
Can compose models into
complex pipelines
Can save, load, and transport
models
Load Data
Tokenizer
Term Hashing
Logistic Reg
Evaluate
Pipeline
Evaluate
Load Data
● Estimator
● Transformer

30. 30#UnifiedAnalytics #SparkAISummit
Microsoft Machine Learning for
Apache Spark v0.17
Microsoft’s Open Source
Contributions to Apache Spark
www.aka.ms/spark Azure/mmlspark
Cognitive
Services
Spark Serving Model
Interpretability
LightGBM Gradient
Boosting
Deep Networks
with CNTK
HTTP on
Spark

31. CNTK on Spark
 GPU/CPU acceleration
 Automated gradient calculations
 Flexible language for defining
huge space of models
 State of the art performance and
accuracy
 ONNX Runtime
 Large scale parallelism
 Fault tolerance
 Auto-scaling / elasticity
 High throughput streaming
 Data source + orchestrator
agnostic

32. Azure Cognitive Services on Spark
 Easy to use integration
between Spark and the
Azure Cognitive Services
 Composable and
pipelinable with all other
SparkML models!
 Python, Scala, R (Beta) val df = new TextSentiment()
.setTextCol(“text”)
.setOutputCol(“sentiment”)
.transform(inputs)
Spark Worker
Partition Partition Partition
Client Client Client
Web Service

33. Spark Serving
 Sub-millisecond RESTful Model
Deployment on Spark Clusters
Spark Worker
Partition Partition Partition
Server
Spark Worker
Partition Partition Partition
Server
Spark Master
Users / Apps
Load Balancer HTTP Requests and
Responses
spark.read.parquet.load(…)
.select(…)
spark.readStream.kafka.load(…)
.select(…)
Batch API:
Streaming API:
Serving API:
spark.readStream.server(“0.0.0.0”, 5000).load(…)
.select(…)

34. Azure Kubernetes Service + Helm
• Works on any k8s cluster
• Helm: Package Manager
for Kubernetes
34
Kubernetes (AKS, ACS, GKE, On-Prem etc)
K8s workerK8s worker
Spark
Worker
Spark
Worker
K8s worker
Cognitive
Service
Container
HTTP on Spark
Spark
Worker
Cognitive
Service
Container
HTTP on Spark
Spark
Worker
Cognitive
Service
Container
HTTP on Spark
Spark
Serving Load
Balancer
Jupyter,
Zepplin,
LIVY, or
Spark
Submit LB
Zepplin
Jupyter
Storage or
other
Databases
Cloud
Cognitive
Services
Spark Serving Hotpath
HTTP on Spark
Spark Readers
REST Requests to
Deployed Models
Submit Jobs, Run Notebooks,
Manage Cluster, etc
Users / Apps
helm repo add mmlspark
https://dbanda.github.io/charts
helm install mmlspark/spark
--set localTextApi=true
Dalitso Banda, dbanda@microsoft.com
Microsoft AI Development Acceleration Program

35. Thanks to
 You all!
 Joseph Sirosh, Sudarshan Raghunathan, Ilya Matiach, Eli Barzilay, Tong
Wen
 Microsoft NERD Garage Team + MIT Externship Program
 Microsoft Development Acceleration Team:
 Dalitso Banda, Casey Hong, Karthik Rajendran, Manon Knoertzer, Tayo
Amuneke, Alejandro Buendia
 Pablo Castro, Chris Hoder, Ryan Gaspar, Henrik Neilsen, Andrew
Schonhoffer, Daniel Ciborowski, Markus Cosowicz
 Azure CAT, AzureML, and Azure Search Teams