The document discusses Microsoft Cognitive Services, which includes a wide range of APIs for vision, speech, language, knowledge, and search applications. It outlines setup procedures, usage examples, and comparisons with other platforms like Google and AWS. The future of AI is perceived to trend towards consumerization and specialized services that integrate seamlessly with applications.

1. Computers have feelings
too.
COGNITIVE SERVICES – PAUL GLAVICH
@GLAV
PAUL.GLAVICH@READIFY.NET

2. And you are?
Paul Glavich
Principal Consultant @ readify
Author
Former MVP of 13 years
International Speaker (if you count NZ)
Industry veteran > 20 years

3. Agenda
Cognitive services – Overview
Setting up
API Examination, Analytics, Emotion, Vision
Flailing while demo fails 
Future?
Wrap up

4. Microsoft AI Overview
Cognitive
Services
• Prebuilt/Pre-trained
Models
• Dev / Human
friendly
Machine
learning,
Bots
• Cortana Intelligence
Suite
• Bot framework
• Build/train your own
models
Cognitive
toolkit
• Deep learning
• Neural networks
• Script driven

5. Cognitive Services Overview
Microsoft Cognitive Services - an API platform
That’s right – It is all about the “Apey Eyes”
Approx. 30 Artificial intelligence API’s

6. Current List
Vision
◦ Computer Vision API, Emotion API PREVIEW, Video Indexer PREVIEW, Face API, Video API PREVIEW, Content moderator, Custom Vision
Service PREVIEW
Speech
◦ Custom Speech Service PREVIEW, Translator Speech API, Bing Speech API, Speaker Recognition API PREVIEW
Language
◦ Language Understanding Intelligent Service PREVIEW, Bing Spell Check API, Web Language Model API PREVIEW, Text Analytics API
PREVIEW, Translator Text API, Linguistic Analysis API PREVIEW
Knowledge
◦ Recommendations API PREVIEW, Academic Knowledge API PREVIEW, Knowledge Exploration Service PREVIEW, QnA Maker API
PREVIEW, Entity Linking Intelligence Service API PREVIEW, Custom Decision Service PREVIEW
Search
◦ Bing Autosuggest API, Bing Video Search API, Bing Image Search API, Bing Web Search API, Bing News Search API, Bing Custom Search
Labs
◦ Project Prague, Gesture-based controls, Project Johannesburg, Route logistics, Project Abu Dhabi, Distance matrix, Project Nanjing,
Isochrones calculations, Project Cuzco, Event associated with Wikipedia entries, Project Wollongong

7. Current List
Vision
◦ Computer Vision API, Emotion API PREVIEW, Video Indexer PREVIEW, Face API, Video API PREVIEW, Content moderator, Custom Vision
Service PREVIEW
Speech
◦ Custom Speech Service PREVIEW, Translator Speech API, Bing Speech API, Speaker Recognition API PREVIEW
Language
◦ Language Understanding Intelligent Service PREVIEW, Bing Spell Check API, Web Language Model API PREVIEW, Text Analytics API
PREVIEW, Translator Text API, Linguistic Analysis API PREVIEW
Knowledge
◦ Recommendations API PREVIEW, Academic Knowledge API PREVIEW, Knowledge Exploration Service PREVIEW, QnA Maker API PREVIEW, Entity Linking
Intelligence Service API PREVIEW, Custom Decision Service PREVIEW
Search
◦ Bing Autosuggest API, Bing Video Search API, Bing Image Search API, Bing Web Search API, Bing News Search API, Bing Custom Search
Labs
◦ Project Prague, Gesture-based controls, Project Johannesburg, Route logistics, Project Abu Dhabi, Distance matrix, Project Nanjing, Isochrones calculations,
Project Cuzco, Event associated with Wikipedia entries, Project Wollongong

8. The favourite child
LUIS
◦ Language Understanding Intelligent Service
Conversational API (Apey Eye  )
https://azure.microsoft.com/en-us/services/cognitive-
services/language-understanding-intelligent-service/
We wont be looking at this

9. What we will be looking at
TextAnalytic API
◦ Extract information such as sentiment, key phrases, topics and language from your text
Emotion API
◦ Examines facial expression in an image as an input, and returns the confidence across a set of emotions
for each face in the image, as well as bounding box for the face.

10. But before we dive in
What about some of the other players?
Google
◦ Machine Learning, Cloud Natural Language API, Cloud Speech API, Cloud Translation API, Cloud Vision API,
Cloud Video Intelligence API BETA
◦ Tensorflow, Kaggle, API.AI
◦ Deepmind – acquired in 2014
AWS
◦ Lex, Polly, and Rekognition, Alexa voice service
◦ Speech recognition, natural language understanding, lifelike speech, image recognition, and image analysis
Salesforce?
◦ Yep
◦ Can distinguish customers from imposters by tracking action patterns
And others….

11. Setup
Add Cognitive services to your azure
subscription
◦ Manually
◦ ARM Template
◦ Powershell
Access keys
Location

12. Rest Call format
https://[location].api.cognitive.microsoft.com/[service]/[version]/[action]
Headers required
◦ Content-type
◦ Ocp-Apim-Subscription-Key

13. Sentiment
Simple Json structure to submit text documents
ID used to correlate response
https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment
Request with synchronous response
Supports English, Spanish, French, and Portuguese with 12 more in preview.

14. Demo
TextAnalytic Raw API usage
Is that positive or negative?

15. Key phrases
Same Json structure as Sentiment
https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/keyPhrases
Request with synchronous response
Currently supports English, German, Spanish and Japanese.

16. Demo
Key phrases Raw API usage
We can determine whether it is
positive or negative, but what are
the key phrases in that?

17. Language
Similar Json structure as Sentiment
Request with synchronous response
120 Languages are supported
https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/languages

18. Demo
Language Raw API usage
Watch your language 

19. Topics
Same Json structure as Sentiment
Stop words – excluded from topic detection
Stop phrases – excluded from returned results
Minimum 100 Documents
Submit request, then check on operation status.
https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/languages
https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/operations/{operationId}

20. Topics response
Initial submit
◦ 202 Accepted
◦ “operation-Location” header also returned
◦ https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/operations/54a25882d7ef494ca779b31
d46f842bb
OperationStatus

21. Emotion
Can detect
◦ Happiness
◦ Sadness
◦ Surprise
◦ Anger
◦ Fear
◦ Contempt
◦ Disgust
◦ Neutral
Works with video as well (via async operation)
https://westus.api.cognitive.microsoft.com/emotion/v1.0/recognize

22. Demo
Emotion API Raw API usage

23. Vision
Categorise content of an image
Find faces
Determine if contains mature/adult content
Url or image upload
https://westus.api.cognitive.microsoft.com/vision/v1.0/analyze?visualFeatures=...&details=...&l
anguage=...
Supports English as default and Simplified Chinese as language to return.

24. Vision
Specify what you want as query string parameters
visualFeatures
◦ Categories - categorizes according to taxonomy defined in documentation.
◦ Tags – returns list of words related to the image content.
◦ Description - describes image with a complete English sentence.
◦ Faces – if faces are present, return coordinates, gender and age.
◦ ImageType - clipart or line drawing.
◦ Color - accent color, dominant color, or black&white.
◦ Adult – Adult in nature? Racy in nature?
details
◦ Celebrities - identifies celebrities if detected in the image.
◦ Landmarks - identifies landmarks if detected in the image.

25. Demo
Vision API Raw Usage

26. A Fluent API
Best way to learn (IMHO) is do.
Fluent API on top of cognitive services
1. TextAnalytics
2. Emotion
3. Vision and more to come

27. Simple usage

28. Demo
Sentiment and keyphrase analysis of emails
Display sentiment result of emails.
http://dddsydney2017glavsdemo.azurewebsites.net

29. Demo
Sentiment and keyphrase analysis of emails
Send email to “FeedbackForGlavsTalk@gmail.com”

30. Demo - high level
Gmail
Scheduler Q
Collect
Email
Text
Analysis
Cosmos
DB
Website
Send
Collect
Analyse
Store
Analyse
Get Mail
Analysed
results
Analysed Email
Azure
Sentiment
Analysis

31. Other fluent examples
Show me the tests
Fluent API
◦ https://github.com/glav/CognitiveServicesFluentApi

32. What I think the future holds
More consumerisation of AI
Specialised services beyond “simple API’s”
More compute “on the device” with greater value
augmented by the cloud
Google’s API.AI is a step towards easier consumption
More seamless application integration.

33. Specialised offerings
As an example:
◦ https://www.customvision.ai/

34. What now?
Visual Studio
◦ File -> New -> Console Application (.Net core)
Install-Package Glav.CognitiveServices.FluentApi.TextAnalytic –Pre
var result = await
TextAnalyticConfigurationSettings.CreateUsingConfigurationKeys(TestConfig.TextAnalyticsApiKey,
LocationKeyIdentifier.WestUs)
.SetDiagnosticLoggingLevel(LoggingLevel.Everything)
.AddDebugDiagnosticLogging()
.UsingHttpCommunication()
.WithTextAnalyticAnalysisActions()
.AddSentimentAnalysis("I am having a fantastic time.")
.AnalyseAllSentimentsAsync();

35. Links and resources
The demo
◦ https://github.com/glav/DDDSydney2017SentimentDemo
Fluent API
◦ https://github.com/glav/CognitiveServicesFluentApi
Nuget Packages
◦ https://www.nuget.org/packages/Glav.CognitiveServices.FluentApi.TextAnalytic/0.5.0-alpha
◦ https://www.nuget.org/packages/Glav.CognitiveServices.FluentApi.Emotion/0.5.0-alpha