The document discusses Sherry List's presentation on mood analysis and machine learning. It begins with introductions and provides a demo link. It then discusses key concepts like artificial intelligence, machine learning, and machine learning techniques. The remainder discusses Azure Cognitive Services and how to use them, including an example of using the Face API to detect emotions by analyzing a captured photo. Code snippets are provided for capturing a photo from the camera, calling the Face API to detect emotions, and drawing the emotions on the canvas.

1. Mood analyzer
@Sherrrylst

2. Hello!
I’m Sherry List
Azure Developer Engagement Lead, Microsoft
You can find me at @SherrryLst

3. Demo
http://bit.ly/mood-analyzer
#devconmu
@Sherrrylst

4. AI, ML, DL, ANN?
@Sherrrylst

5. Artificial Intelligence
AI is the simulation of human intelligence
processes by machines. These
processes include, reasoning,
remembering, learning and self-
correction.
@Sherrrylst

6. Machine Learning
@Sherrrylst
Machine learning is a field of computer
science that gives computers the ability
to learn without being explicitly
programmed.
Arthur Samuel, 1959

7. Machine learning techniques
@Sherrrylst
• Artificial Neural Networks
• Deep Learning
• Bayesian Networks
• Clustering

8. Artificial Intelligence (AI)
The bigger picture
Machine
Learning
Artificial Neural Networks
Deep Learning
Bayesian Networks
Clustering
@Sherrrylst

9. Machine
Learning 101
@Sherrrylst

10. Machine Learning 101
@Sherrrylst
Data

11. @Sherrrylst
Data
Patterns
Machine Learning 101

12. @Sherrrylst
Data
Patterns
Algorithm
(Deep learning, clustering, …)
Machine Learning 101

13. @Sherrrylst
Data
Patterns
Algorithm
(Deep learning, clustering, …)
Find patterns
Machine Learning 101

14. @Sherrrylst
Data
Patterns
Model
Algorithm
(Deep learning, clustering, …)
Find patterns
Machine Learning 101

15. @Sherrrylst
Data
Patterns
Model
Algorithm
(Deep learning, clustering, …)
Find patterns Recognizes patterns
Machine Learning 101

16. @Sherrrylst
Data
Patterns
Model
Algorithm
(Deep learning, clustering, …)
Find patterns Recognizes patterns
Training
Machine Learning 101

17. @Sherrrylst
Data
Patterns
Model
Algorithm
(Deep learning, clustering, …)
Find patterns Recognizes patterns Application
Training
Machine Learning 101

18. @Sherrrylst
Data
Patterns
Model
Algorithm
(Deep learning, clustering, …)
Find patterns Recognizes patterns Application
Challenges
Data preparation

19. @Sherrrylst
Data
Patterns
Model
Algorithm
(Deep learning, clustering, …)
Find patterns Recognizes patterns Application
Challenges
Data preparation Create and test algorithms

20. @Sherrrylst
Algorithm
(Deep learning, clustering, …)
Find patterns
Challenges
Create and test algorithms
Chihuahua or muffin?

21. @Sherrrylst
Data
Patterns
Model
Algorithm
(Deep learning, clustering, …)
Find patterns Recognizes patterns Application
Challenges
Data preparation Create and test algorithms Expose the model to app

22. Is there a better
way?
@Sherrrylst

23. Azure
Cognitive
Services
@Sherrrylst

24. @Sherrrylst
Application
The easier way
Azure Cognitive
Services
REST API

25. Azure Cognitive Services === machine-learning-as-a-service
@Sherrrylst

26. @Sherrrylst
Application
The easier way
Azure Cognitive
Services
REST API

27. Cognitive Services
Decision
Speech
Language
Vision
Web search
@Sherrrylst

28. Cognitive Services
Decision
Speech
Language
Vision
Web search
@Sherrrylst

29. • Computer vision
@Sherrrylst
Cognitive Services - Vision

30. • Computer vision
• Video Indexer
@Sherrrylst
Cognitive Services - Vision

31. • Computer vision
• Video Indexer
• Face
@Sherrrylst
Cognitive Services - Vision

32. • Computer vision
• Video Indexer
• Face
• Ink Recognizer (Preview)
@Sherrrylst
Cognitive Services - Vision

33. • Computer vision
• Video Indexer
• Face
• Ink Recognizer (Preview)
• Form Recognizer (Preview)
@Sherrrylst
Cognitive Services - Vision

34. • Computer vision
• Video Indexer
• Face
• Ink Recognizer (Preview)
• Form Recognizer (Preview)
• Custom Vision
@Sherrrylst
Cognitive Services - Vision

35. How to use it?
@Sherrrylst

36. How to use Cognitive Services?
1. Create service in Azure Portal
(https://portal.azure.com/)
@Sherrrylst

37. How to use Cognitive Services?
1. Create service in Azure Portal
(https://portal.azure.com/)
2. Customize and train model in the
portal (Optional)
@Sherrrylst

38. How to use Cognitive Services?
1. Create service in Azure Portal
(https://portal.azure.com/)
2. Customize and train model in the
portal (Optional)
3. Call the REST API from your app
@Sherrrylst

39. How to use Cognitive Services?
1. Create service in Azure Portal
(https://portal.azure.com/)
2. Customize and train model in the
portal (Optional)
3. Call the REST API from your app
4. Parse the JSON response
@Sherrrylst

40. Let’s do it!
@Sherrrylst

41. @Sherrrylst
Cognitive Services account - Face
Face API

42. Create a Cognitive Services account in the
Azure portal
@Sherrrylst

43. @Sherrrylst
Cognitive Services account

44. @Sherrrylst
• URL : {Endpoint}/face/v1.0/detect
• URI Parameters
• returnFaceLandmarks (Boolean)
• returnFaceAttributes
• age,gender,smile,glasses,emotion,facialHair
• Headers
• Content-Type: application/octet-stream | application/json
• Ocp-Apim-Subscription-Key
Face API - Detect
Face API Documentation

45. @Sherrrylst
Face API - Detect
Face API Documentation

46. @Sherrrylst
💻</code>

47. @Sherrrylst
Capture the picture
<div class="p-1">
<video #video autoplay></video>
</div>
<div class="pb-2">
<button class="btn btn-primary" (click)="capture()">🤳</button>
</div>
<div>
<canvas #canvas id="result"></canvas>
</div>
Source code

48. @Sherrrylst
Start the camera
export class CameraComponent {
@ViewChild("video", { static: true }) videoElement: ElementRef;
@ViewChild("canvas", { static: true }) canvas: ElementRef;
constraints = {
video: {
facingMode: ”user",
width: { ideal: 400 },
height: { ideal: 400 }
}
};
constructor( private renderer: Renderer2 ) {}
ngOnInit() {
this.startCamera();
}
Source code

49. @Sherrrylst
Start the camera
export class CameraComponent {
@ViewChild("video", { static: true }) videoElement: ElementRef;
@ViewChild("canvas", { static: true }) canvas: ElementRef;
constraints = {
video: {
facingMode: "environment",
width: { ideal: 400 },
height: { ideal: 400 }
}
};
constructor( private renderer: Renderer2 ) {}
ngOnInit() {
this.startCamera();
}
Source code

50. @Sherrrylst
Start the camera
export class CameraComponent {
@ViewChild("video", { static: true }) videoElement: ElementRef;
@ViewChild("canvas", { static: true }) canvas: ElementRef;
constraints = {
video: {
facingMode: "environment",
width: { ideal: 400 },
height: { ideal: 400 }
}
};
constructor( private renderer: Renderer2 ) {}
ngOnInit() {
this.startCamera();
}
Source code

51. @Sherrrylst
Start the camera
export class CameraComponent {
@ViewChild("video", { static: true }) videoElement: ElementRef;
@ViewChild("canvas", { static: true }) canvas: ElementRef;
constraints = {
video: {
facingMode: "environment",
width: { ideal: 400 },
height: { ideal: 400 }
}
};
constructor( private renderer: Renderer2 ) {}
ngOnInit() {
this.startCamera();
}
Source code

52. @Sherrrylst
Start the camera
startCamera() {
if (!!(navigator.mediaDevices && navigator.mediaDevices.getUserMedia)) {
navigator.mediaDevices
.getUserMedia(this.constraints)
.then(this.attachVideo.bind(this))
.catch(this.handleError);
} else {
alert("Sorry, camera not available.");
}
}
Source code

53. @Sherrrylst
Start the camera
attachVideo(stream) {
this.renderer.setProperty(
this.videoElement.nativeElement,
"srcObject",
stream
);
}
Source code

54. @Sherrrylst
Start the camera
Source code

55. @Sherrrylst
Start the camera
attachVideo(stream) {
this.renderer.setProperty(
this.videoElement.nativeElement, "srcObject", stream
);
this.renderer.listen(this.videoElement.nativeElement, "play", event =
> {
this.videoHeight = this.videoElement.nativeElement.videoHeight;
this.videoWidth = this.videoElement.nativeElement.videoWidth;
});
}
Source code

56. @Sherrrylst
🤳 Snap

57. @Sherrrylst
Capture the picture
capture() {
this.renderer.setProperty( this.canvas.nativeElement, "width",
this.videoWidth);
this.renderer.setProperty( this.canvas.nativeElement,"height",
this.videoHeight);
this.canvas.nativeElement
.getContext("2d")
.drawImage(this.videoElement.nativeElement, 0, 0);
...
}
Source code

58. @Sherrrylst
Capture the picture
capture() {
this.renderer.setProperty( this.canvas.nativeElement, "width",
this.videoWidth);
this.renderer.setProperty( this.canvas.nativeElement,"height",
this.videoHeight);
this.canvas.nativeElement
.getContext("2d")
.drawImage(this.videoElement.nativeElement, 0, 0);
...
}
Source code

59. @Sherrrylst
Capture the picture
capture() {
...
this.selfie = this.canvas.nativeElement.toDataURL("img/png");
let file = this.cameraService.dataURItoBlob(this.selfie);
// Mojify this
this.detectEmotions(file);
}
Source code

60. @Sherrrylst
Capture the picture
capture() {
...
this.selfie = this.canvas.nativeElement.toDataURL("img/png");
let file = this.cameraService.dataURItoBlob(this.selfie);
// Mojify this
this.detectEmotions(file);
}
Source code

61. @Sherrrylst
Capture the picture
async detectEmotions(file: Blob) {
const headerDict = {
"Content-Type": "application/octet-stream","Ocp-Apim-Subscription-Key": this.apiKey
};
const requestOptions = {
headers: new HttpHeaders(headerDict)
};
this.http
.post(this.faceAPI, file, requestOptions)
.subscribe((res: FaceData[]) => {
this.emotions = this.cameraService.getEmotionData(res);
this.draw(this.emotions);
});
}
Source code

62. @Sherrrylst
Capture the picture
async detectEmotions(file: Blob) {
const headerDict = {
"Content-Type": "application/octet-stream","Ocp-Apim-Subscription-Key": this.apiKey
};
const requestOptions = {
headers: new HttpHeaders(headerDict)
};
this.http
.post(this.faceAPI, file, requestOptions)
.subscribe((res: FaceData[]) => {
this.emotions = this.cameraService.getEmotionData(res);
this.draw(this.emotions);
});
}
Source code

63. @Sherrrylst
Capture the picture
async detectEmotions(file: Blob) {
const headerDict = {
"Content-Type": "application/octet-stream","Ocp-Apim-Subscription-Key": this.apiKey
};
const requestOptions = {
headers: new HttpHeaders(headerDict)
};
this.http
.post(this.faceAPI, file, requestOptions)
.subscribe((res: FaceData[]) => {
this.emotions = this.cameraService.getEmotionData(res);
this.draw(this.emotions);
});
}
Source code

64. @Sherrrylst
Capture the picture
async detectEmotions(file: Blob) {
const headerDict = {
"Content-Type": "application/octet-stream","Ocp-Apim-Subscription-Key": this.apiKey
};
const requestOptions = {
headers: new HttpHeaders(headerDict)
};
this.http
.post(this.faceAPI, file, requestOptions)
.subscribe((res: FaceData[]) => {
this.emotions = this.cameraService.getEmotionData(res);
this.draw(this.emotions);
});
}
Source code

65. @Sherrrylst
Capture the picture
async detectEmotions(file: Blob) {
const headerDict = {
"Content-Type": "application/octet-stream","Ocp-Apim-Subscription-Key": this.apiKey
};
const requestOptions = {
headers: new HttpHeaders(headerDict)
};
this.http
.post(this.faceAPI, file, requestOptions)
.subscribe((res: FaceData[]) => {
this.emotions = this.cameraService.getEmotionData(res);
this.draw(this.emotions);
});
}
Source code

66. @Sherrrylst
🤳Draw

67. @Sherrrylst
Draw the Emoji
draw(faces: Face[]) {
const cameraCanvas = this.canvas.nativeElement.getContext("2d");
for (let face of faces) {
let multiplier = face.faceRectangle.height / face.faceRectangle.width;
let fontSize = face.faceRectangle.width * multiplier;
cameraCanvas.font = fontSize + "px Arial";
cameraCanvas.fillText(
face.mojiIcon,
(face.faceRectangle.left * multiplier) - 20,
(face.faceRectangle.top * multiplier) + (face.faceRectangle.height / 2) + 20
);
}
}
Source code

68. @Sherrrylst
Capture the picture
draw(faces: Face[]) {
const cameraCanvas = this.canvas.nativeElement.getContext("2d");
for (let face of faces) {
let multiplier = face.faceRectangle.height / face.faceRectangle.width;
let fontSize = face.faceRectangle.width * multiplier;
cameraCanvas.font = fontSize + "px Arial";
cameraCanvas.fillText(
face.mojiIcon,
(face.faceRectangle.left * multiplier) - 20,
(face.faceRectangle.top * multiplier) + (face.faceRectangle.height / 2) + 20
);
}
}
Source code

69. @Sherrrylst
Capture the picture
draw(faces: Face[]) {
const cameraCanvas = this.canvas.nativeElement.getContext("2d");
for (let face of faces) {
let multiplier = face.faceRectangle.height / face.faceRectangle.width;
let fontSize = face.faceRectangle.width * multiplier;
cameraCanvas.font = fontSize + "px Arial";
cameraCanvas.fillText(
face.mojiIcon,
(face.faceRectangle.left * multiplier) - 20,
(face.faceRectangle.top * multiplier) + (face.faceRectangle.height / 2) + 20
);
}
}
Source code

70. Tell me, how are you?
http://bit.ly/mood-analyzer
# devconmu
@Sherrrylst

71. AI.lab@Sherrrylst

72. The Learner

73. aka.ms/azure.heroes
#azureHeroes

74. @Sherrrylst
Thank you!
https://aka.ms/mood-analyzer

75. Resources
• Microsoft Azure Cognitive Services: The Big Picture
• The Mojifier (MS Learn)
• The Mojifier (Github)
• Where's Chewie? Object detection with Azure Custom Vision by Goran Vuksic
• What does the Computer Vision see? Analyse a local image with JavaScript by Goran Vuksic
• Azure Cognitive Services API — I need your clothes, boots and your motorcycle by Chris Noring
• Add conversational intelligence to your apps by using LUIS (MS Learn)
• Discover sentiment in text with the Text Analytics API (MS Learn)
• Create Intelligent Bots with the Azure Bot Service (MS Learn)
• Capturing Camera Images with Angular by Chad Upton

76. AI Services
Cognitive Services Bot Service Azure ML, Databricks, HDInsight
Pre-build AI Conversational AI Custom AI
Azure Infrastructure
CPU, FPGA, GPU
Cosmos
DB
SQL DB Data Lake IOT EdgeDSVMSparkBatch AISQL DW

77. Deep learning Frameworks
Cognitive Toolkits Tensorflow Azure ML, Databricks, HDInsight
Pre-build AI Conversational AI Custom AI
Deep learning Frameworks
CPU, FPGA, GPU
Cosmos
DB
SQL DB Data Lake IOT EdgeDSVMSparkBatch AISQL DW

78. • Speech to Text
• Text to Speech
• Speaker recognition (Preview)
• Speech Translation
@Sherrrylst
Cognitive Services - Speech

79. • Speech to Text
• Text to Speech
• Speaker recognition (Preview)
• Speech Translation
@Sherrrylst
Cognitive Services - Speech

80. • Language Understanding
• Bing Spell Check
• Translator Text
• Content Moderator
• Text Analytics
@Sherrrylst
Cognitive Services - Language

81. • Language Understanding
• Bing Spell Check
• Translator Text
• Content Moderator
• Text Analytics
@Sherrrylst
Cognitive Services - Language

82. Cognitive Services - Search
@Sherrrylst
• Bing Custom Search
• Bing Web Search
• Bing Video Search
• Bing Image Search
• Bing Local Business Search (Preview)
• Bing Visual Search
• Bing Entity Search
• Bing News Search
• Bing Auto Suggest