This document describes Microsoft's cognitive search capabilities for enriching and annotating content through natural language processing and computer vision. It discusses how unstructured data like text, images and videos can be ingested from various sources and stores, enriched with built-in and custom cognitive skills, and indexed for exploration and search. The enriched and annotated documents can then be used to train and deploy custom machine learning models.

1. Microsoft Confidential

2. Understanding the latent value in all content

3. Text
(1) Validate enrichment pipeline

4. Tags
“throwing”, “ball”, “girl”, “grass”, “basketball”
Caption
“A girl throwing a ball”

5. Entities
Persons
“Anita Christiansen”,
“Conrad Nuber”,
Locations
“Bothell”, “Woodinville”
Organization
“Litware Insurance Corp.”

7. Computer Vision
Face
Emotion
Content Moderator
Video Indexer
Custom Vision
Service
Custom Decision
Q-n-A Maker
Language
Understanding (LUIS)
Text Analytics
Bing Spell Check
Translator Text
Speaker
Recognition
Bing Speech
Custom Speech
Translator Speech
Unified Speech
Service
Bing Autosuggest
Bing Search
Bing Entity Search
Bing Statistics add-in
Bing Visual Search
Bing Custom Search

8. Management Free
Keyword search
Faceting
Geospatial support
Multi-Language Support
Suggestions/auto-complete
Customizable scoring models
Proximity Search
Synonyms
etc.

9. INGEST
Data in any
format, any
Azure store
ENRICH EXPLORE
Annotations
Cognitive skills
Search

10. Annotated
Documents
Customer
Data
Built-in Cognitive Skills
OCR,
Key Phrase Extraction,
People Names,
Company Names,
Sentiment Analyzer,
Computer Vision,
etc.
Search
Index
.pdf
.doc
.jpeg
…
Third Party Enrichers
Custom classification models,
Custom entity extraction,
etc.
Azure Machine
Learning

11. Annotated
Documents
Built-in Cognitive Skills
OCR,
Key Phrase Extraction,
People Names,
Company Names,
Sentiment Analyzer,
Computer Vision,
etc.
Search
Index
Third Party Enrichers
Custom classification models,
Custom entity extraction,
etc.
Azure Machine
Learning
Customer
Data
.pdf
.doc
.jpeg
…

13. Annotated
Documents
Search
Index
Built-in Cognitive Skills
OCR,
Key Phrase Extraction,
People Names,
Company Names,
Sentiment Analyzer,
Computer Vision,
etc.
Third Party Enrichers
Custom classification models,
Custom entity extraction,
etc.
Customer
Data
.pdf
.doc
.jpeg
…

14. Key Phrase Extraction
Sentiment Analysis
Organization Entity Extraction
Location Entity Extraction
Persons Entity Extraction
Language Detection
Face Detection
Tag Extraction
Celebrity Recognition
Landmark Detection
Handwriting Recognition (Preview)
Printed Text Recognition

16. …,
{
"@odata.type": "#Microsoft.Skills.Custom.WebApiSkill",
"uri" "https://myskill.azurewebsites.net/api/OrgId"
"httpHeaders": {"Api-Key": "mySecret" },
"context": "/document/organizations/*" ,
"inputs":
[
{ "name": “organizationName", "source": "/document/organizations/*" },
],
"outputs":
[
{ "name": "organizationId", "targetName": "organizationId" }
]
},

17. {
"values": [
{
"recordId": "7cad2",
"data":
{
"myOuput1": “animals"
}
},
{
"recordId": "7cad3",
"data":
{
"myOutput1": “colors"
}
},
…
]
}
{
"values": [
{
"recordId": "7cad2",
"data":
{
"myInput1": "fox",
"myInput2": "cat",
}
},
{
"recordId": "7cad3",
"data":
{
"myInput1": "blue",
"myInput2": "red",
}
},
…
]
}

18. Azure Machine
Learning

19. content
keyPhrases
organizations
docClass

20. content
normalized
images
language
tags
orgs
content
content

21. "skills": [
{
"@odata.type": "#Microsoft.Skills.Text.LanguageDetectionSkill",
"inputs":
[
{ "name": "text", "source": "/document/content" }
],
"outputs":
[
{ "name": "languageCode", "targetName": "myLanguageCode" },
{ "name": "languageName", "targetName": "myLanguageName" }
]
},

22. …,
{
"@odata.type": "#Microsoft.Skills.Text.NamedEntityRecognitionSkill",
"categories": [ "Organization" ],
"defaultLanguageCode": "en",
"inputs":
[
{ "name": "text", "source": "/document/content" },
"name" "languageCode" "source" "/document/myLanguageCode"
],
"outputs":
[
{ "name": "organizations", "targetName": "organizations" }
]
},

23. content
normalized
images
language
tags
orgs
content
content

24. …,
{
"@odata.type": "#Microsoft.Skills.Text.NamedEntityRecognitionSkill",
"categories": [ "Organization" ],
"defaultLanguageCode": "en",
"inputs":
[
{ "name": "text", "source": "/document/content" },
"name" "languageCode" "source" "/document/myLanguagecode"
],
"outputs":
[
{ "name": "organizations", "targetName": "organizations" }
]
},

25. …,
{
"@odata.type": "#Microsoft.Skills.Custom.WebApiSkill",
"uri" "https://myskill.azurewebsites.net/api/OrgId"
"context": "/document/organizations/*" ,
"httpHeaders": {"Api-Key": "mySecret" },
"inputs":
[
{ "name": “organizationName", "source": "/document/organizations/*" },
],
"outputs":
[
{ "name": "organizationId", "targetName": "organizationId" }
]
},

26. Search
Index
Built-in Cognitive Skills
OCR,
Key Phrase Extraction,
People Names,
Company Names,
Sentiment Analyzer,
Computer Vision,
etc.
Third Party Enrichers
Custom classification models,
Custom entity extraction,
etc.
Customer
Data
.pdf
.doc
.jpeg
…
Annotated
Documents

27. /document
/languageCode /keyPhrases /organizations /images
/1
/2
/…
/n
/1
/2
/…
/n
organizationId
organizationId
organizationId
organizationId
/1
/2
/…
/n
tags
tags
tags
tags
document.pdf

28. Annotated
Documents
Built-in Cognitive Skills
OCR,
Key Phrase Extraction,
People Names,
Company Names,
Sentiment Analyzer,
Computer Vision,
etc.
Third Party Enrichers
Custom classification models,
Custom entity extraction,
etc.
Customer
Data
.pdf
.doc
.jpeg
…
Search
Index

29. /document
/keyPhrases
/0
/1
/…
/n
/organizations
/0
/1
/…
/n
organizationId
organizationId
organizationId
organizationId
/images
/0
/1
/…
/n
tags
tags
tags
tags
New Indexer Property
{
…
"outputFieldMappings":
[
{
"sourceFieldName":
"/document/organizations/*/organizationId",
"targetFieldName":
"orgIds"
} ,
…
]
}

30. Annotated
Documents
Customer
Data
Built-in Cognitive Skills
OCR,
Key Phrase Extraction,
People Names,
Company Names,
Sentiment Analyzer,
Computer Vision,
etc.
Search
Index
.pdf
.doc
.jpeg
…
Third Party Enrichers
Custom classification models,
Custom entity extraction,
etc.
Azure Machine
Learning

32. “Lorem ipsum dolor sit amet,
consectetur adipiscing elit, sed
do eiusmod tempor incididunt ut
labore et dolore magna aliqua. Ut
enim ad minim veniam, quis
nostrud exercitation ullamco
laboris nisi…”
Class A
Class B
Class C

33. “Lorem ipsum dolor sit amet,
consectetur adipiscing elit, sed
do eiusmod tempor incididunt ut
labore et dolore magna aliqua. Ut
enim ad minim veniam, quis
nostrud exercitation ullamco
laboris nisi…”
“Lorem ipsum dolor sit amet,
consectetur adipiscing elit, sed
do eiusmod tempor incididunt ut
labore et dolore magna aliqua. Ut
enim ad minim veniam, quis
nostrud exercitation ullamco
laboris nisi…”
Entity type A
Entity type B

36. Labeled
Data
Custom
Entity
Extraction
Template
Azure ML
Annotated
Documents
Customer
Data
Search
Index

38. Cognitive Search
Documentation | Sign up for Azure Search
Azure Machine Learning Package for Text Analytics
Documentation | Create a Data Science Virtual Machine
Cognitive Services
Documentation | Sign up