Artificial Intelligence is not something exotic anymore. With Siri, Cortana & Google Assistant on virtually any device, our users have come to expect our applications to understand them in more meaningful ways than ever before. Luckily for us, various platforms have emerged which enable us to integrate this kind of functionality into our applications and services without having to do the heavy lifting by ourselves. In this session we will take a full-fledged cross platform Xamarin application built around a public image API, and enhance it with intelligence using Azure Cognitive Services. We'll see how we can add vision, language and speech into our applications, making them smarter than ever before. Stand out in the crowd - bring intelligence into your apps!

1. Why Don’t You Understand Me?
Build Intelligence into Your Apps
1
Eran Stiller
@eranstiller
Chief Technology Officer
erans@codevalue.net
http://stiller.blog

2. 2

3. 3

4. 4

5. Agenda
Microsoft Cognitive Services
Add Intelligence to Sample App
Vision
Language
Search
Speech
5

6. About Me
Eran Stiller (@eranstiller)
CTO & Founder at CodeValue
Software architect, consultant and instructor
Microsoft Azure MVP
Many years of hands-on experience
Expert in large-scale, server-side, highly-concurrent
systems
Founder of Azure Israel Meetup
6

7. About CodeValue
Awesome software company!
About 200 employees, most of which technology experts
High quality software development solutions
7

8. 8
OzCode – Debug Like a WizardQuit debugging, spend more time writing brilliant software
With our Visual Studio extension for C#, follow the road to a bug-free world
oz-code.com | @oz_code
Magic Glance / Figure out complex expressionsLINQ Debugging / Know the flow of your LINQ queries
Reveal / Focus on data that actually matterSearch/ Find that needle in a haystack of data

9. 10
Microsoft Cognitive Services

10. 11

11. Sample Application
IntelligentPX
Xamarin Mobile Application
Cross-Platform
Android, iOS & Windows
Based on the 500px REST API
https://github.com/500px/api-documentation
12

12. Sample Application
13
All content in this
presentation is live!
Viewer discretion advised….

13. Demo
14
IntelligentPX

14. 15
Vision

15. Computer Vision API
General Purpose Vision API
Description
Tags
Adult/Racy content classification
Face detection
Dominant colors
16

16. Computer Vision API
17
Description { "tags": [ "person", "indoor", "woman", "sitting", "holding", "grass", "looking",
"young", "hair", "man", "smiling", "black", "laptop", "wearing", "standing",
"room", "phone", "shirt", "water", "red", "living", "white", "blue", "cat", "night",
"umbrella" ], "captions": [ { "text": "Mona Lisa looking at the camera",
"confidence": 0.8883329 } ] }
Tags [ { "name": "person", "confidence": 0.98936075 } ]
Image format "Jpeg"
Image dimensions 1797 x 1191
Clip art type 0
Line drawing type 0
Black and white false
Adult content false
Adult score 0.0318996236
Racy false
Racy score 0.0194663126
Categories null
Faces [ { "age": 22, "gender": "Female", "faceRectangle": { "top": 335, "left": 389,
"width": 290, "height": 290 } } ]
Dominant color
background "Black"
Dominant color
foreground "Black"
Accent Color
#516936

17. Demo
18
Computer Vision API

18. Custom Vision Service
Build your own classification model!
https://customvision.ai/
19
Flexibility
Complexity
Computer Vision API
Custom AI
Custom Vision Service

19. Custom Vision Service
Simple cycle
Upload
Train
Evaluate
Active learning
20

20. Demo
21
Custom Vision Service

21. Emotion API
Recognize emotions in images
React to the emotions of your users
22

22. Emotion API
[
{
"faceRectangle": {
"top": 335,
"left": 389,
"width": 290,
"height": 290
},
"scores": {
"anger": 6.88452974E-06,
"contempt": 0.00261615869,
"disgust": 1.97378677E-06,
"fear": 1.35348373E-07,
"happiness": 0.214705735,
"neutral": 0.7825558,
"sadness": 8.04257361E-05,
"surprise": 3.28936549E-05
}
}
]
23

23. Face Detection API
Detect faces in an image
Highly detailed information
Face attributes
Emotions
Pose
Gender
Age
Glasses
Makeup
Etc.
24

24. Demo
25
Emotion API
Face Detection API

25. Other Vision Services
Computer Vision API
Read text
Read handwritten text
Recognize celebrities and landmarks
Content Moderator
Image, text & video
Human review tool
Video API
Stabilize shaky videos
Detect and track faces
Detect motion
Generate video thumbnails
Video Indexer
Unlock video insights
26

26. 27
Language

27. Text Analytics API
Analyze various features of text
Language
Sentiment
Key Phrases
Topic
Varying number of supported languages
https://docs.microsoft.com/en-us/azure/cognitive-services/text-
analytics/overview#supported-languages
28

28. Text Analytics API
Eran Stiller was awesome.
He had the best lecture.
I just wish he would have shown
more code on-stage.
29

29. Demo
30
Text Analytics API

30. Other Language Services
Language Understanding Intelligent Service (LUIS)
Bing Spell Check API
Translator Text API
Web Language Model API
Linguistic Analysis API
31

31. 32
Search

32. Bing Autosuggest API
Empower users to type less and do more
33

33. Demo
34
Bing Autosuggest API

34. Other Search Services
Bing Image Search API
Bing News Search API
Bing Video Search API
Bing Web Search API
Bing Custom Search
Bing Entity Search API
35

35. 36
Speech

36. Bing Speech API
Convert audio to text
Understand intent
Convert text back to speech for natural responsiveness
37

37. Demo
38
Bing Speech API

38. Other Speech Services
Translator Speech API
Speaker Recognition API
Custom Speech Service
39

39. Other Knowledge Services
Recommendations API
Academic Knowledge API
Knowledge Exploration Service
QnA Maker API
Entity Linking Intelligence Service API
Custom Decision Service
40

40. Takeaways
41
Microsoft Cognitive Services are easy to use
Intelligence as a Service
Features which were very hard before are suddenly easy to add
The next generation of applications will take this to the extreme

41. Takeaways
42

42. Resources
Sample Code
https://github.com/estiller/intelligent-px-sample
Cognitive Services
https://azure.microsoft.com/en-us/services/cognitive-services/
https://docs.microsoft.com/en-us/azure/cognitive-services/
43

43. Thank You!
44
Eran Stiller
@eranstiller
Chief Technology Officer
erans@codevalue.net
http://stiller.blog