Jeff Fried
CTO, BA Insight
Use Office 365 and Azure Cognitive Services
for Intelligent Search
October 2017
Longtime
Search Nerd
• CTO, BA Insight
• Senior PM, Microsoft
• VP, FAST
• SVP, LingoMotors
Jeff Fried
Passionate About
• Search
• Search-driven applications
• Information Strategy
• Text Analytics
• Machine Learning
Email: jeff.fried@bainsight.com
Twitter: @jefffried
Blog: www.DoMoreWithSearch.com
Outline
Wait, isn’t search
ALREADY ‘intelligent’?
Matching IntentContent
Content Intent
Lookup
Learn
ExploreMessy
Human
Language
A new wave of Intelligent Search
Engines?
Machine Learning (ML), Natural Language Processing (NLP),
Semantics, Cognitive, Artificial Intelligence (AI), Self-Learning, …
http://www.cmswire.com/information-management/intelligent-cognitive-ai-based-search-separating-hype-from-reality/
Forrester: “Keyword search of the past has evolved into Cognitive Search”
Forrester, June 2017
Evolution of enterprise search technology
TODAYYESTERDAY TOMORROW
Relevancy based on general criteria that
compare how well a single item matches a
query, compared to the rest of the index.
Matching Content & Popularity
By leveraging your personal activity, we
can signal highly relevant content that
would otherwise drown in the noise
Personalized & behavior based Drive actionable insights
Take action on data and empower
decision making by understanding
intention across many facets
Scope & scale of Microsoft Graph
Use machine learning to create
insights from your data
Construct pairs for questions & answers
Query match quality in content or page
Popularity of document or page
Content metadata such as created date
Bing entity extraction
Understanding organizational priorities
as well as your extended network
Using AI for semantic understanding
+ +
11
Semantics! Machine Learning! AI!
Latest Enhancements to SPO search
Making O365 search intelligent using Azure Cognitive Services
more
ˇ
Start with understanding how it works
Matching IntentContent
Context
Content
Enrichment
UX
Connections
Query
Augmentation
Insight Data Sets
Trending
Documents
People I
Work With
Used
Documents
Email
Files
Contacts
Groups
Meetings
Org
Tasks
File Enrichments
Item
Metadata
# Activity
History
myGraph
Sites
I Use
Suggested
Sites
Usage
Statistics
How does the search box work?
Used
Documents
Trending
Documents
Sites
I Use
Suggested
Sites
Beautiful
Bergen
Contoso
Norway
Kathrine
Hammervold
Metadata Drives Great User Experiences
Documents from many sources
All client or matter-relevant documents are integrated. Rich MetaData
Content annotated automatically – concepts,
categories, citations, matters, clients, etc
Navigation Controls
Explore, Discover, Drill-down
Where is the Metadata in Delve?
So….Idon’tneed
metadata,right?
Delve doesn’t have refinement
Chart courtesy of xkcd
Outline
The Cognitive Arms Race
Agent Bots Cognitive Machine Learning Augmented Reality
Microsoft Cortana Bot Framework Cognitive Services
- Vision
- Speech
- Language
- Knowledge
- Search
- Labs
Azure ML
Cognitive Toolkit
Hololens
Google Google Assistant API.AI Cloud Vision API
Cloud Video Intelligence
Cloud Speech API
Cloud Natural Language API
Google Knowledge Graph
Google Custom Search
ML Advanced Solutions Lab
Cloud ML Engine
TensorFlow
Glass at Work
Daydream/Tango
IBM Watson Virtual Agent Watson Conversation Watson APIs
- Vision
- Speech
- Language
- Data Insights
Watson ML Service
Apache SystemML
Amazon Alexa Amazon Lex Rekognition
Polly
Amazon ML
Apple Siri SiriKit
Speech
Core ML ARKit
Facebook Facebook M ParlAI FastText
CommAI
Torch Oculus Rift
The Cognitive Arms Race
Matching IntentContent
Context
Content
Enrichment
UXConnections
Query
Augmentation
Making SharePoint search intelligent with Cognitive Services
more
ˇ
BA Insight Product Portfolio
Smart Analytics
AutoClassifier
Content
from 60+
Enterprise
Systems
Cognitive Services (AI, Machine Learning)
Search
Index(es)
Search for
Dynamics
Connectors
Expertise
Locator
Smart
Previews
SmartHub
Content
Content
Processing
Search
Index
/ Graph
Search UI
Enhancing Content with Cognitive Services
Query
Processing
Content
Enrichment
AutoClassifier
rules-driven core, combined with learning-based modules
Enhanced Content
Enriched with
Metadata and
Content Types
Search Visualization Workflow
Adding Text Analytics to Content
User
Uploads
Content Event
Fires
Content
Text
Analysis
Sentiment - Is text positive or
negative?
Key Phrases - What are
people discussing in a single
article?
Topics - What are people
discussing across many
articles?
Language - What language is
text written in?
Add
metadata
to item
Shows in
O365
search
Adding Translation to Content
User
Uploads
Content Event
Fires
Content
Text
Analysis
Translation – Copy existing
content and translate to
specified language
save in
SharePoint
Shows in
O365
search
Create new file/content for
specific language
Image Search
User
Uploads
Content Event
Fires
Computer
Vision
Image Processing – recognize
objects and text inside
images
Shows in
O365
search
Add
metadata
to item
Extract
image(s)
from
documents
Image Classifier and Search
project libraries
and
issue detection
Search Visualization Workflow
Feed content from over
60 enterprise systems
Search
Index(es)
Search UI
Connectors
Smart
Analytics
Extensible
content processing
Pluggable query and
results pipelines
Auto
Classifier
Cognitive
Services
SmartHub
“Architectural” View
37
Content is spread across lots of systems
• Aderant
• Amazon S3
• Alfresco
• Box
• Confluence
• CuadraSTAR
• Elite / 3E
• EMC Documentum
• EMC eRoom
• Google Drive
• HP Consolidated Archive
• (EAS, aka Zantaz)
• HPE Records Manager/HP TRIM
• IBM Connections
• IBM Content Manager
• IBM DB2
• IBM FileNet P8
• IBM Lotus Notes
• IBM WebSphere
• iManage Work
• Jive
• LegalKEY
• LexisNexis Interaction
• Lotus Notes Databases
• Microsoft Dynamics CRM
• Microsoft Exchange
• Microsoft Exchange Public Folders
• Microsoft SQL Server
• MySQL
• NetDocuments
• Neudesic The Firm Directory
• Objective
• OpenText LiveLink/RM
• OpenText eDOCS DM
• Oracle Database
• Oracle WebCenter
• Oracle WebCenter Content (UCM/Stellent)
• PLC/Practical Law
• ProLaw
• Salesforce.com
• SAP ERP
• ServiceNow
• SharePoint Online
• SharePoint 2016
• SharePoint 2013
• SharePoint 2010
• SharePoint 2007
• Sitecore
• Any SQL-based CRM system
• Veeva Vault
• Veritas Enterprise Vault
(Symantec eVault)
• West km
• Xerox DocuShare
• Yammer
List of standard connectors, plus a proven architecture and
process for creating new connectors to complex systems
Feed content from over
60 enterprise systems
Search
Index(es)
Search UI
Connectors
Smart
Analytics
Extensible
content processing
Pluggable query and
results pipelines
Auto
Classifier
Cognitive
Services
SmartHub
Bots
AR (hololens) apps
Bots and AR
Better Search
- data enrichment
- query augmentation
- relevance
- recommendations
New Types of Search
and Discovery
- multi-media search
- multi-lingual search
- spoken language search
- search bots
- search in AR
AI for Intelligent Search & Discovery
Content
Content
Processing
Search
Index
Search UI
ML applied to search
Query
Processing
Learning about content structure Learning about user intent
Analytics
Behavioral data to drive learning
Learning to rank
BA Insight Product Portfolio
Smart Analytics
AutoClassifier
Content
from 60+
Enterprise
Systems
Cognitive Services (AI, Machine Learning)
Search
Index(es)
Search for
Dynamics
Connectors
Expertise
Locator
Smart
Previews
SmartHub
www.BAinsight.com
Jeff.Fried@BAinsight.com
@jefffried

Use O365 and Azure Cognitive Services for intelligent search

  • 1.
    Jeff Fried CTO, BAInsight Use Office 365 and Azure Cognitive Services for Intelligent Search October 2017
  • 2.
    Longtime Search Nerd • CTO,BA Insight • Senior PM, Microsoft • VP, FAST • SVP, LingoMotors Jeff Fried Passionate About • Search • Search-driven applications • Information Strategy • Text Analytics • Machine Learning Email: jeff.fried@bainsight.com Twitter: @jefffried Blog: www.DoMoreWithSearch.com
  • 4.
  • 5.
    Wait, isn’t search ALREADY‘intelligent’?
  • 6.
  • 7.
  • 8.
    A new waveof Intelligent Search Engines? Machine Learning (ML), Natural Language Processing (NLP), Semantics, Cognitive, Artificial Intelligence (AI), Self-Learning, … http://www.cmswire.com/information-management/intelligent-cognitive-ai-based-search-separating-hype-from-reality/
  • 9.
    Forrester: “Keyword searchof the past has evolved into Cognitive Search” Forrester, June 2017
  • 10.
    Evolution of enterprisesearch technology TODAYYESTERDAY TOMORROW Relevancy based on general criteria that compare how well a single item matches a query, compared to the rest of the index. Matching Content & Popularity By leveraging your personal activity, we can signal highly relevant content that would otherwise drown in the noise Personalized & behavior based Drive actionable insights Take action on data and empower decision making by understanding intention across many facets Scope & scale of Microsoft Graph Use machine learning to create insights from your data Construct pairs for questions & answers Query match quality in content or page Popularity of document or page Content metadata such as created date Bing entity extraction Understanding organizational priorities as well as your extended network Using AI for semantic understanding + +
  • 11.
  • 12.
  • 14.
    Making O365 searchintelligent using Azure Cognitive Services more ˇ Start with understanding how it works
  • 15.
  • 16.
    Insight Data Sets Trending Documents PeopleI Work With Used Documents Email Files Contacts Groups Meetings Org Tasks File Enrichments Item Metadata # Activity History myGraph Sites I Use Suggested Sites Usage Statistics
  • 17.
    How does thesearch box work? Used Documents Trending Documents Sites I Use Suggested Sites Beautiful Bergen Contoso Norway Kathrine Hammervold
  • 18.
    Metadata Drives GreatUser Experiences Documents from many sources All client or matter-relevant documents are integrated. Rich MetaData Content annotated automatically – concepts, categories, citations, matters, clients, etc Navigation Controls Explore, Discover, Drill-down
  • 19.
    Where is theMetadata in Delve?
  • 20.
  • 21.
  • 22.
  • 23.
    Agent Bots CognitiveMachine Learning Augmented Reality Microsoft Cortana Bot Framework Cognitive Services - Vision - Speech - Language - Knowledge - Search - Labs Azure ML Cognitive Toolkit Hololens Google Google Assistant API.AI Cloud Vision API Cloud Video Intelligence Cloud Speech API Cloud Natural Language API Google Knowledge Graph Google Custom Search ML Advanced Solutions Lab Cloud ML Engine TensorFlow Glass at Work Daydream/Tango IBM Watson Virtual Agent Watson Conversation Watson APIs - Vision - Speech - Language - Data Insights Watson ML Service Apache SystemML Amazon Alexa Amazon Lex Rekognition Polly Amazon ML Apple Siri SiriKit Speech Core ML ARKit Facebook Facebook M ParlAI FastText CommAI Torch Oculus Rift The Cognitive Arms Race
  • 24.
  • 26.
    BA Insight ProductPortfolio Smart Analytics AutoClassifier Content from 60+ Enterprise Systems Cognitive Services (AI, Machine Learning) Search Index(es) Search for Dynamics Connectors Expertise Locator Smart Previews SmartHub
  • 27.
    Content Content Processing Search Index / Graph Search UI EnhancingContent with Cognitive Services Query Processing Content Enrichment
  • 28.
    AutoClassifier rules-driven core, combinedwith learning-based modules Enhanced Content Enriched with Metadata and Content Types Search Visualization Workflow
  • 29.
    Adding Text Analyticsto Content User Uploads Content Event Fires Content Text Analysis Sentiment - Is text positive or negative? Key Phrases - What are people discussing in a single article? Topics - What are people discussing across many articles? Language - What language is text written in? Add metadata to item Shows in O365 search
  • 30.
    Adding Translation toContent User Uploads Content Event Fires Content Text Analysis Translation – Copy existing content and translate to specified language save in SharePoint Shows in O365 search Create new file/content for specific language
  • 31.
    Image Search User Uploads Content Event Fires Computer Vision ImageProcessing – recognize objects and text inside images Shows in O365 search Add metadata to item Extract image(s) from documents
  • 32.
    Image Classifier andSearch project libraries and issue detection Search Visualization Workflow
  • 34.
    Feed content fromover 60 enterprise systems Search Index(es) Search UI Connectors Smart Analytics Extensible content processing Pluggable query and results pipelines Auto Classifier Cognitive Services SmartHub “Architectural” View
  • 36.
    37 Content is spreadacross lots of systems • Aderant • Amazon S3 • Alfresco • Box • Confluence • CuadraSTAR • Elite / 3E • EMC Documentum • EMC eRoom • Google Drive • HP Consolidated Archive • (EAS, aka Zantaz) • HPE Records Manager/HP TRIM • IBM Connections • IBM Content Manager • IBM DB2 • IBM FileNet P8 • IBM Lotus Notes • IBM WebSphere • iManage Work • Jive • LegalKEY • LexisNexis Interaction • Lotus Notes Databases • Microsoft Dynamics CRM • Microsoft Exchange • Microsoft Exchange Public Folders • Microsoft SQL Server • MySQL • NetDocuments • Neudesic The Firm Directory • Objective • OpenText LiveLink/RM • OpenText eDOCS DM • Oracle Database • Oracle WebCenter • Oracle WebCenter Content (UCM/Stellent) • PLC/Practical Law • ProLaw • Salesforce.com • SAP ERP • ServiceNow • SharePoint Online • SharePoint 2016 • SharePoint 2013 • SharePoint 2010 • SharePoint 2007 • Sitecore • Any SQL-based CRM system • Veeva Vault • Veritas Enterprise Vault (Symantec eVault) • West km • Xerox DocuShare • Yammer List of standard connectors, plus a proven architecture and process for creating new connectors to complex systems
  • 38.
    Feed content fromover 60 enterprise systems Search Index(es) Search UI Connectors Smart Analytics Extensible content processing Pluggable query and results pipelines Auto Classifier Cognitive Services SmartHub Bots AR (hololens) apps Bots and AR
  • 40.
    Better Search - dataenrichment - query augmentation - relevance - recommendations New Types of Search and Discovery - multi-media search - multi-lingual search - spoken language search - search bots - search in AR AI for Intelligent Search & Discovery
  • 41.
    Content Content Processing Search Index Search UI ML appliedto search Query Processing Learning about content structure Learning about user intent Analytics Behavioral data to drive learning Learning to rank
  • 42.
    BA Insight ProductPortfolio Smart Analytics AutoClassifier Content from 60+ Enterprise Systems Cognitive Services (AI, Machine Learning) Search Index(es) Search for Dynamics Connectors Expertise Locator Smart Previews SmartHub
  • 44.