The document provides an overview of machine learning and how to use Azure Cognitive Services for facial recognition and emotion detection. It discusses collecting data, finding patterns in the data, building models, training models, and applying models. It then introduces Azure Cognitive Services as an easier way to perform these tasks using pre-built APIs and provides a step-by-step example of using the Face API to detect emotions from a selfie photo taken via webcam.

1. Mood analyzer
@Sherrrylst

2. Hello!
I’m Sherry List
Azure Developer Engagement Lead, Microsoft
You can find me at @SherrryLst

3. http://bit.ly/mood-analyzer

4. Machine
Learning 101

5. @Sherrrylst
Data
Machine Learning 101

6. @Sherrrylst
Data
Patterns
Machine Learning 101

7. @Sherrrylst
Data
Patterns
Algorithm
(Deep learning, clustering, …)
Machine Learning 101

8. @Sherrrylst
Data
Patterns
Algorithm
(Deep learning, clustering, …)
Find patterns
Machine Learning 101

9. @Sherrrylst
Data
Patterns
Model
Algorithm
(Deep learning, clustering, …)
Find patterns
Machine Learning 101

10. @Sherrrylst
Data
Patterns
Model
Algorithm
(Deep learning, clustering, …)
Find patterns Recognizes patterns
Machine Learning 101

11. @Sherrrylst
Data
Patterns
Model
Algorithm
(Deep learning, clustering, …)
Find patterns Recognizes patterns
Training
Machine Learning 101

12. @Sherrrylst
Data
Patterns
Model
Algorithm
(Deep learning, clustering, …)
Find patterns Recognizes patterns Application
Training
Machine Learning 101

13. Challenges

14. @Sherrrylst
Data
Patterns
Model
Algorithm
(Deep learning, clustering, …)
Find patterns Recognizes patterns Application
Challenges
Data preparation

15. @Sherrrylst
Data
Patterns
Model
Algorithm
(Deep learning, clustering, …)
Find patterns Recognizes patterns Application
Challenges
Data preparation Create and test algorithms

16. @Sherrrylst
Algorithm
(Deep learning, clustering, …)
Find patterns
Challenges
Create and test algorithms
Chihuahua or muffin?

17. @Sherrrylst
Data
Patterns
Model
Algorithm
(Deep learning, clustering, …)
Find patterns Recognizes patterns Application
Challenges
Data preparation Create and test algorithms Expose the model to app

18. Is there an
easier way?

19. Azure
Cognitive
Services

20. @Sherrrylst
Application
The easier way
Azure Cognitive
Services
REST API

21. @Sherrrylst
Application
The easier way
Azure Cognitive
Services
REST API

22. Cognitive Services
Decision
Speech
Language
Vision
Web search
@Sherrrylst

23. Cognitive Services
Decision
Speech
Language
Vision
Web search
@Sherrrylst

24. Cognitive Services - Vision
• Computer vision
• Video Indexer
• Face
• Form Recognizer
• Custom Vision
@Sherrrylst

25. Cognitive Services - Vision
• Computer vision
• Video Indexer
• Face
• Form Recognizer
• Custom Vision
@Sherrrylst

26. @Sherrrylst
Face Detection
Face API

27. Let’s do it!
@Sherrrylst

28. Create a Cognitive Services account in the
Azure portal
@Sherrrylst

29. @Sherrrylst
Cognitive Services account

30. 💻</code>
@Sherrrylst

31. @Sherrrylst
Capture the picture
<div class="p-1">
<video #video autoplay></video>
</div>
<div class="pb-2">
<button (click)="capture()">🤳</button>
</div>
<div>
<canvas #canvas id="flatten"></canvas>
</div>
Source code

32. @Sherrrylst
Start the camera
export class CameraComponent implements OnInit {
@ViewChild("video", { static: true }) videoElement: ElementRef;
@ViewChild("canvas", { static: true }) canvas: ElementRef;
constraints = {
video: {
facingMode: "user",
width: { ideal: 400 },
height: { ideal: 400 }
}
};
constructor(private renderer: Renderer2) { }
ngOnInit(){
this.startCamera();
}
Source code

33. @Sherrrylst
Start the camera
export class CameraComponent implements OnInit {
@ViewChild("video", { static: true }) videoElement: ElementRef;
@ViewChild("canvas", { static: true }) canvas: ElementRef;
constraints = {
video: {
facingMode: "user",
width: { ideal: 400 },
height: { ideal: 400 }
}
};
constructor(private renderer: Renderer2) { }
ngOnInit(){
this.startCamera();
}
Source code

34. @Sherrrylst
Start the camera
export class CameraComponent implements OnInit {
@ViewChild("video", { static: true }) videoElement: ElementRef;
@ViewChild("canvas", { static: true }) canvas: ElementRef;
constraints = {
video: {
facingMode: "user",
width: { ideal: 400 },
height: { ideal: 400 }
}
};
constructor(private renderer: Renderer2) { }
ngOnInit(){
this.startCamera();
}
Source code

35. @Sherrrylst
Start the camera
startCamera() {
if (!!(navigator.mediaDevices && navigator.mediaDevices.getUserMedia)) {
navigator.mediaDevices
.getUserMedia(this.constraints)
.then(this.attachVideo.bind(this))
.catch(this.handleError);
} else {
alert("Sorry, camera not available.");
}
}
Source code

36. @Sherrrylst
Start the camera
startCamera() {
if (!!(navigator.mediaDevices && navigator.mediaDevices.getUserMedia)) {
navigator.mediaDevices
.getUserMedia(this.constraints)
.then(this.attachVideo.bind(this))
.catch(this.handleError);
} else {
alert("Sorry, camera not available.");
}
}
Source code

37. @Sherrrylst
Start the camera
attachVideo(stream) {
this.renderer.setProperty(
this.videoElement.nativeElement,
"srcObject",
stream
);
this.renderer.listen(this.videoElement.nativeElement, "play", event => {
this.videoHeight = this.videoElement.nativeElement.videoHeight;
this.videoWidth = this.videoElement.nativeElement.videoWidth;
});
}
Source code

38. @Sherrrylst
Start the camera
Source code

39. @Sherrrylst
Start the camera
attachVideo(stream) {
this.renderer.setProperty(
this.videoElement.nativeElement,
"srcObject",
stream
);
this.renderer.listen(this.videoElement.nativeElement, "play", event => {
this.videoHeight = this.videoElement.nativeElement.videoHeight;
this.videoWidth = this.videoElement.nativeElement.videoWidth;
});
}
Source code

40. 🤳 Snap
@Sherrrylst

41. @Sherrrylst
Capture the picture
capture() {
this.renderer.setProperty(
this.canvas.nativeElement,
"width",
this.videoWidth
);
this.renderer.setProperty(
this.canvas.nativeElement,
"height",
this.videoHeight
);
this.canvas.nativeElement
.getContext("2d")
.drawImage(this.videoElement.nativeElement, 0, 0);
...
}
Source code

42. @Sherrrylst
Capture the picture
capture() {
this.renderer.setProperty(
this.canvas.nativeElement,
"width",
this.videoWidth
);
this.renderer.setProperty(
this.canvas.nativeElement,
"height",
this.videoHeight
);
this.canvas.nativeElement
.getContext("2d")
.drawImage(this.videoElement.nativeElement, 0, 0);
...
}
Source code

43. @Sherrrylst
Capture the picture
capture() {
...
this.selfie = this.canvas.nativeElement.toDataURL("img/png");
let file = this.cameraService.dataURItoBlob(this.selfie);
// Mojify this
this.detectEmotions(file);
}
Source code

44. @Sherrrylst
Capture the picture
capture() {
...
this.selfie = this.canvas.nativeElement.toDataURL("img/png");
let file = this.cameraService.dataURItoBlob(this.selfie);
// Mojify this
this.detectEmotions(file);
}
Source code

45. @Sherrrylst
Capture the picture
capture() {
...
this.selfie = this.canvas.nativeElement.toDataURL("img/png");
let file = this.cameraService.dataURItoBlob(this.selfie);
// Mojify this
this.detectEmotions(file);
}
Source code

46. @Sherrrylst
Capture the picture
async detectEmotions(file: Blob) {
const headerDict = {
"Content-Type": "application/octet-stream",
"Ocp-Apim-Subscription-Key": this.apiKey
};
const requestOptions = {
headers: new HttpHeaders(headerDict)
};
this.http
.post(this.faceAPI, file, requestOptions)
.subscribe((res: FaceData[]) => {
this.emotions = this.cameraService.getEmotionData(res);
// let's drwa the emoji on canvas
this.draw(this.emotions);
});
}
Source code

47. @Sherrrylst
Capture the picture
async detectEmotions(file: Blob) {
const headerDict = {
"Content-Type": "application/octet-stream",
"Ocp-Apim-Subscription-Key": this.apiKey
};
const requestOptions = {
headers: new HttpHeaders(headerDict)
};
this.http
.post(this.faceAPI, file, requestOptions)
.subscribe((res: FaceData[]) => {
this.emotions = this.cameraService.getEmotionData(res);
// let's drwa the emoji on canvas
this.draw(this.emotions);
});
}
Source code

48. @Sherrrylst
Capture the picture
async detectEmotions(file: Blob) {
const headerDict = {
"Content-Type": "application/octet-stream",
"Ocp-Apim-Subscription-Key": this.apiKey
};
const requestOptions = {
headers: new HttpHeaders(headerDict)
};
this.http
.post(this.faceAPI, file, requestOptions)
.subscribe((res: FaceData[]) => {
this.emotions = this.cameraService.getEmotionData(res);
// let's drwa the emoji on canvas
this.draw(this.emotions);
});
}
Source code

49. 📚 Service
@Sherrrylst

50. @Sherrrylst
Capture the picture
async detectEmotions(file: Blob) {
const headerDict = {
"Content-Type": "application/octet-stream",
"Ocp-Apim-Subscription-Key": this.apiKey
};
const requestOptions = {
headers: new HttpHeaders(headerDict)
};
this.http
.post(this.faceAPI, file, requestOptions)
.subscribe((res: FaceData[]) => {
this.emotions = this.cameraService.getEmotionData(res);
// let's drwa the emoji on canvas
this.draw(this.emotions);
});
}
Source code

51. @Sherrrylst
Capture the picture
async detectEmotions(file: Blob) {
const headerDict = {
"Content-Type": "application/octet-stream",
"Ocp-Apim-Subscription-Key": this.apiKey
};
const requestOptions = {
headers: new HttpHeaders(headerDict)
};
this.http
.post(this.faceAPI, file, requestOptions)
.subscribe((res: FaceData[]) => {
this.emotions = this.cameraService.getEmotionData(res);
// let's drwa the emoji on canvas
this.draw(this.emotions);
});
}
Source code

52. @Sherrrylst
🤳 Draw

53. @Sherrrylst
Capture the picture
draw(faces: Face[]) {
const cameraCanvas = this.canvas.nativeElement.getContext("2d");
for (let face of faces) {
let multiplier = face.faceRectangle.height / face.faceRectangle.width;
let fontSize = face.faceRectangle.width * multiplier;
let stringFont = fontSize + "px Arial";
cameraCanvas.font = stringFont;
cameraCanvas.fillText(
face.mojiIcon,
face.faceRectangle.left * multiplier - 20,
face.faceRectangle.top * multiplier + face.faceRectangle.height / 2 + 20
);
}
}
Source code

54. @Sherrrylst
Capture the picture
draw(faces: Face[]) {
const cameraCanvas = this.canvas.nativeElement.getContext("2d");
for (let face of faces) {
let multiplier = face.faceRectangle.height / face.faceRectangle.width;
let fontSize = face.faceRectangle.width * multiplier;
let stringFont = fontSize + "px Arial";
cameraCanvas.font = stringFont;
cameraCanvas.fillText(
face.mojiIcon,
face.faceRectangle.left * multiplier - 20,
face.faceRectangle.top * multiplier + face.faceRectangle.height / 2 + 20
);
}
}
Source code

55. @Sherrrylst
Capture the picture
draw(faces: Face[]) {
const cameraCanvas = this.canvas.nativeElement.getContext("2d");
for (let face of faces) {
let multiplier = face.faceRectangle.height / face.faceRectangle.width;
let fontSize = face.faceRectangle.width * multiplier;
let stringFont = fontSize + "px Arial";
cameraCanvas.font = stringFont;
cameraCanvas.fillText(
face.mojiIcon,
face.faceRectangle.left * multiplier - 20,
face.faceRectangle.top * multiplier + face.faceRectangle.height / 2 + 20
);
}
}
Source code

56. @Sherrrylst
Capture the picture
draw(faces: Face[]) {
const cameraCanvas = this.canvas.nativeElement.getContext("2d");
for (let face of faces) {
let multiplier = face.faceRectangle.height / face.faceRectangle.width;
let fontSize = face.faceRectangle.width * multiplier;
let stringFont = fontSize + "px Arial";
cameraCanvas.font = stringFont;
cameraCanvas.fillText(
face.mojiIcon,
face.faceRectangle.left * multiplier - 20,
face.faceRectangle.top * multiplier + face.faceRectangle.height / 2 + 20
);
}
}
Source code

57. @Sherrrylst
Capture the picture
draw(faces: Face[]) {
const cameraCanvas = this.canvas.nativeElement.getContext("2d");
for (let face of faces) {
let multiplier = face.faceRectangle.height / face.faceRectangle.width;
let fontSize = face.faceRectangle.width * multiplier;
let stringFont = fontSize + "px Arial";
cameraCanvas.font = stringFont;
cameraCanvas.fillText(
face.mojiIcon,
face.faceRectangle.left * multiplier - 20,
face.faceRectangle.top * multiplier + face.faceRectangle.height / 2 + 20
);
}
}
Source code

58. Tell me, how are you?
http://bit.ly/mood-analyzer
# ngPoland
@Sherrrylst

59. AI.lab@Sherrrylst

60. The Learner

61. aka.ms/azure.heroes
#azureHeroes

62. Join the
community!
azureheroes.community

63. @Sherrrylst
Thank you!
https://aka.ms/mood-analyzer

64. Resources
• Microsoft Azure Cognitive Services: The Big Picture
• The Mojifier (MS Learn)
• The Mojifier (Github)
• Where's Chewie? Object detection with Azure Custom Vision by Goran Vuksic
• What does the Computer Vision see? Analyse a local image with JavaScript by Goran Vuksic
• Azure Cognitive Services API — I need your clothes, boots and your motorcycle by Chris Noring
• Add conversational intelligence to your apps by using LUIS (MS Learn)
• Discover sentiment in text with the Text Analytics API (MS Learn)
• Create Intelligent Bots with the Azure Bot Service (MS Learn)
• Capturing Camera Images with Angular by Chad Upton

65. Artificial Intelligence
AI is the simulation of human intelligence
processes by machines. These processes
include, reasoning, remembering, learning
and self-correction.
@Sherrrylst

66. Machine Learning
Machine learning is a field of computer
science that gives computers the ability to
learn without being explicitly programmed.
Arthur Samuel, 1959
@Sherrrylst

67. Machine learning
techniques
• Artificial Neural Networks
• Deep Learning
• Bayesian Networks
• Clustering
@Sherrrylst

68. Artificial Intelligence (AI)
The bigger picture
Machine
Learning
Artificial Neural Networks
Deep Learning
Bayesian Networks
Clustering
@Sherrrylst

69. AI Services
Cognitive Services Bot Service Azure ML, Databricks, HDInsight
Pre-build AI Conversational AI Custom AI
Azure Infrastructure
CPU, FPGA, GPU
Cosmos
DB
SQL DB Data Lake IOT EdgeDSVMSparkBatch AISQL DW

70. Deep learning Frameworks
Cognitive Toolkits Tensorflow Azure ML, Databricks, HDInsight
Pre-build AI Conversational AI Custom AI
Deep learning Frameworks
CPU, FPGA, GPU
Cosmos
DB
SQL DB Data Lake IOT EdgeDSVMSparkBatch AISQL DW

71. • Speech to Text
• Text to Speech
• Speaker recognition (Preview)
• Speech Translation
@Sherrrylst
Cognitive Services - Speech

72. • Speech to Text
• Text to Speech
• Speaker recognition (Preview)
• Speech Translation
@Sherrrylst
Cognitive Services - Speech

73. • Language Understanding
• Bing Spell Check
• Translator Text
• Content Moderator
• Text Analytics
@Sherrrylst
Cognitive Services - Language

74. • Language Understanding
• Bing Spell Check
• Translator Text
• Content Moderator
• Text Analytics
@Sherrrylst
Cognitive Services - Language

75. Cognitive Services - Search
@Sherrrylst
• Bing Custom Search
• Bing Web Search
• Bing Video Search
• Bing Image Search
• Bing Local Business Search (Preview)
• Bing Visual Search
• Bing Entity Search
• Bing News Search
• Bing Auto Suggest