High-level view of the Bing platform that powers the Cortana experiences.

1. the bing platform that powers cortana
savasp
http://savas.me

2. why a personal assistant?
cortana introduction
key scenarios and the cloud
lessons learnt
crazy scenarios we can build

3. increasing focus on information (& knowledge), personalization, context, wearables, …
anticipatory computing
importance of near-real time processing/reactive computing
transition from web to apps to personal assistants

5. PERSONAL
Cortana…
…is your truly personal assistant
…gets to know you
…is transparent
Examples:
 Learning
 Notebook
 Personal suggestions
 Transparency & control
LOOKS OUT FOR YOU
Cortana…
…looks out for you
…filters out the noise
…reminds you of what’s important
Examples:
 Useful and relevant alerts
 Planners
 Event scheduling
 Quiet hours and inner circle
 Reminders
DELIGHTFUL & EASY TO USE
Cortana…
…“just works”
…lets you interact on your terms
…has a fun & engaging personality
Examples:
 Voice & natural language
 Text input
 Personality (visual, spoken voice,
and behavior)
  

7. bing platform as the foundation for
personal assistant experiences
user centric
cloud-driven
service-oriented
asynchronous, reactive, functional, stateless
data-driven
configuration-driven
secure and available

8. ingredients – how to build a personal assistant
privacy/security
design
knowledge platform
machine learning platform
feedback loop infrastructure
notifications infrastructure
stream/complex event processing
metrics/data-driven engineering
speech recognition
natural language understanding
user understanding
conversational & intent understanding
personality
global datacenter footprint
legal
business development
…

9. the notebook

10. cortana home
rank
html
request
language
generation
bing knowledge
?retrieve the user’s profile,
context, pending questions
aggregate, filter
inferences
retrieve knowledge and
rank based on the
user’s current context

11. flight BA 49 is delayed
notification
information streams: sports, flights,
weather, traffic, news, packages,
user location, …
notification
information streams: sports, flights,
weather, traffic, news, packages,
user location, …
var subscription = streamPlatform
.GetObservable<FlightInfo>(Constants.FlightsStream)
.Where(f => f.StatusCode == FlightStatus.Landed)
.Select(f => string.Format("{0} {1}: {2}", f.Airline.Name, f.FlightNumber, f.StatusCode))
.Subscribe(...);
stream
processing
platform

12. is this your home?
commute to work around
8.30am on Mondays
inferences
user understanding/insight

13. “remind me to wish Paul
happy birthday”
speech recognition
natural language
understanding
conversation
management
representation
of intent
speech stream
Using Deep Neural Networks-trained models
and other machine-learning techniques, we
convert the voice stream to a representation
the computers can understand
We reason over the machine representation of the user’s
intent. We use what we know about the user (e.g. “home”
is transformed to an actual location) in order to fill in any
gaps in our understanding. We maintain a dialog with the
user (e.g. if the user had just said “remind me”, we would
have initiated a follow up question).
“when I get home, remind
me to take out the
garbage”
language
generation
user
profile/context
web socket
Using Deep Neural Networks-trained models
and other machine-learning techniques, we
convert the voice stream to a representation
the computers can understand
We reason over the machine representation of the user’s
intent. We use what we know about the user (e.g. “home”
is transformed to an actual location) in order to fill in any
gaps in our understanding. We maintain a dialog with the
user (e.g. if the user had just said “remind me”, we would
have initiated a follow up question).
“when I get home, remind
me to take out the
garbage”
language
generation
user
profile/context
web socket
mini reactor

14. “will I need a scarf
tomorrow?”
speech recognition
natural language
understanding
conversation
management
user
profile/context
html
speech stream
Using Deep Neural Networks-trained models
and other machine-learning techniques, we
convert the voice stream to a representation
the computers can understand
We reason over the machine representation of the user’s
intent. We consult Bing Knowledge and the user’s profile in
order to construct a response in the conversation with the
user. Since we are keeping the context of the conversation,
the user can follow up with a question such as “How about
next weekend?” or “What about Seattle?”.
language
generation
bing knowledge
rank

15. bing.com
rank
html
request
language
generation
bing knowledge
?retrieve the user’s profile,
context, pending questions
aggregate, filter
inferences

16. ideas
DISCLAIMER: what follows does NOT represent
future products or services by microsoft

17. while at Sydney 2011, you should talk to john s.
both of you were at paris 2010, rio 2009, and
istanbul 2002 conferences
you also seem to be reading the same journals on
knowledge representation
you both found the book “on intelligence”
interesting
finally, you both like sushi…
i suggest “blue sushi”, just two blocks from the
conference center

18. “what was the title of the song
that got me dancing during the
radiohead concert few days ago?”

19. emergence of personal assistant as ux
metaphore, consolidation of user experiences
anticipatory computing, task completion,
conversational
sensors, wearables
reactive, near-realtime

21. Coldplay at the Gorge
2009 – 07 – 10, 9.12pm
Weather: Warm and mostly sunny
Attending: 20,000
Song playing at the time of photograph: Viva La Vida
Did you know that it was Coldplay’s 1st appearance at the
Gorge?
Ad-hoc social streams: photos, tweets, emotions
i

22. savas parastatidis
the bing platform that powers Cortana
savasp
http://savas.me