Session from oracle code one 2018: After his first steps into robotics and IoT, this session’s speaker decided to take it one step further: a more realistic robot that knows who you are and responds to your questions. Using chatbot technology and voice and face recognition, this robot can become a real add-on to your daily life. In this session, you will learn how the speaker extended an off-the-shelf solution with additional cloud technology to make these things work

1. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
“Talk to Me Goose”
Going Beyond Your Regular
Chatbot
Luc Bors
Oracle Code One
Session DEV5557
22 October 2018 at 10:30 AM
Moscone West – Room 2011
Email: Luc.bors@eproseed.com
Twitter: @lucb_

2. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
Luc Bors
• Technical Director
• ADF, JET, MAF, MCS, BOTS, DA, IOT
• ACE Director / Groundbreaker Ambassador
• Using Oracle Tech since 1999
3
WHO AM I?
About eProseed
• Local offices in several parts of world
• HQ in Luxemburg
• Award winning Oracle partner
One more on
the way
Work in
progress

3. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
PROGRAM AGENDA
Introducing Chatbots
Towards Digital Assistants
Speech and Face Recognition
Putting things together
Summary
2
3
4
4
1
5

4. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
(CHAT)BOTS
5
• Generally speaking a bot is any software that performs an automated task
• A Chatbot is a computer program designed to simulate conversation with
human users, especially over the Internet.
• The biggest misconception that arises is that a chatbot is a bot that
converses with a human in the way that another human would converse
with a human.

5. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
• Intents
• Utterances
• Entities
• Machine Learning/NLP
• Dialog Flow
6
TALKING SOME CHATBOT SLANG

6. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential7
Derived from customer input
What does the user want?
Is this mapped to an action?
Order Pizza
Order Pasta
Cancel Order
“I want to order a cheese pizza”
WHAT ARE INTENTS?
Intent

7. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential8
Statements that map to intent
“Sample data” for an intent
Not exact string matching
Machine learning
Need many good utterances
WHAT ARE UTTERANCES?
Order Pizza
Order Pasta
Cancel Order
Utterances

8. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential9
Variable/parameter for intent
Important word in an input
Adds relevance to intent
Order Pizza
“Hi, I want to order a cheese pizza”
Cheese
Pepperoni
Ham
Possibly maps to domain object
WHAT ARE ENTITIES?
Entities

9. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential10
Language independent ML
NLP for added accuracy
Natural language processing
Prediction based on utterances
WHAT IS MACHINE LEARNING?
Machine Learning

10. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
Manages conversation flow
State and context
What to do based on an input
DIALOG FLOW
Source: https://bit.ly/2mBSIqI
Dialog Flow

11. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
PROGRAM AGENDA
Introducing Chatbots
Towards Digital Assistants
Speech and Face Recognition
Putting things together
Summary
2
3
4
12
1
5

12. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
WHAT IS A DIGITAL ASSISTANT?
• A virtual assistant, also called AI assistant or digital assistant, is an application
program that understands natural language voice commands and completes tasks for
the user.
13

13. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
• Amazon Echo (Alexa)
• Google Home
• Apple Siri
• IBM Watson Assistant
• Microsoft Cortana
• Samsung Bixby
• Oracle Digital Assistant
MANY VENDORS HAVE DIGITAL ASSISTANTS….…
14

14. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
ORACLE DIGITAL ASSISTANT
15

15. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential16
CHATBOT CODE INVOLVED

16. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
THE DEMO OF A ‘REGULAR’ CHATBOT
17
DEM
O

17. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
PROGRAM AGENDA
Introducing Chatbots
Towards Digital Assistants
Speech and Face Recognition
Putting things together
Summary
2
3
4
18
1
5

18. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
RALPH BREAKS THE INTERNET: WRECK-IT RALPH 2 OFFICIAL
TRAILER
19
• https://www.youtube.com/watch?v=_BcYBFC6zfY&start=53&end=79

19. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
• Amazon’s Alexa Voice Service
• face_recognition and Tensorflow
– The world's simplest face recognition
library
20
M(AN)Y CHOICE(S) FOR SPEECH AND FACE RECOGNITION
Nice comparisons found here for face
https://bit.ly/2AkXVby and here https://bit.ly/2Ozfptg
Nice comparisons found here for speech / voice assistants
found here: https://bit.ly/2q5lf7c

20. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
HOW DOES ALEXA WORK?
• Alexa Device à Alexa Service à Invoke Custom Skill Code
21

21. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential22
https://bit.ly/2P8yOAT

22. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
FACE RECOGNITION
• OpenCV or face_recognition
– For Face Recognition
– Aimed at real-time computer vision
• TensorFlow
– An open source machine learning framework
23

23. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
1. look at a picture and find all the
faces in it
2. focus on each face and be able to
understand that even if a face is
turned in a weird direction or in
bad lighting, it is still the same
person.
3. be able to pick out unique
features of the face that you can
use to tell it apart from other
people— like how big the eyes
are, how long the face is, etc.
4. compare the unique features of
that face to all the people you
already know to determine the
person’s name.
24
HOW DOES FACE RECOGNITION WORK?
from: https://bit.ly/2ab1mmR

24. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
• Hi Alexa.
• Get me the latest playoff scores
please.
• No, Thanks
• Hi Luc, what can I do for you?
• Boston plays LA in the finals Luc.
• Anything else I can do for you?
• Oke Luc, have a nice day.
25
MAKE THINGS PERSONAL
Combining speech- and face- recognition will enable your ‘thing / device / bot / digital assistant’ to see and recognize people

25. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
PROGRAM AGENDA
Introducing Chatbots
Towards Digital Assistants
Speech and Face Recognition
Putting things together
Summary
2
3
4
26
1
5

26. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
GOING BEYOND YOUR REGULAR CHATBOT
• Open Up Your Regular Chatbot (Oracle Digital Assistant)
– Webhooks
• Turn Your RaspberryPi into a Personal Assistant (The Robots’ Brain)
– AlexaPi
• Add (on command) Face Recognition
– PiCam
– IFTTT (with Alexa in and Webhooks out)
27

27. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
OPENING UP ORACLE DIGITAL ASSISTANT
• Webhook Channel
• REST Endpoints
– A POST call that enables the server to
receive messages from your bot.
– A POST call that enables the server to
send messages to your bot.
28

28. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
WEBHOOK CODE (FROM ALEXA TO BOT)
29

29. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
WEBHOOK CODE (FROM BOT TO ALEXA)
30

30. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
NODE CODE INVOLVED (FROM BOT TO ALEXA)
31

31. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
CREATING ALEXA CUSTOM SKILL
• Needs invocation sentence
• Needs intent:
– What is it that you want…
– A sentence
• Needs a slot:
– To define how data in the slot
is recognized and handled
– AMAZON.LITERAL
• Needs endpoint to call out to
32

32. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
• Enter (or say) invocation name
• Walk through a flow
33
TEST CUSTOM SKILL: TALK TO ORACLE ADA

33. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
TURN YOUR RASPBERRYPI INTO A PERSONAL ASSISTANT
• AlexaPi is the way to go.
• Install it on a fresh Raspbian image
• Follow installation manual here : https://github.com/alexa-pi/AlexaPi/wiki/Installation
• Important note:
– Installation guide needs some updates.
• sudo pip install CherryPy==10.2.1
• sudo pip install six==1.11.0
34

34. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
DEFINE AN ALEXA DEVICE ON AMAZON
35

35. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
RUNNING THE SETUP
36

36. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
ORDERING PIZZA (HANDSFREE)
37
Alexa, order me
a Pizza
Alexa Voice
Service Custom
Skill
Oracle
Compute Cloud
(node
webhook)
Oracle Digital
Assistant Cloud
What kind of
pizza would you
like?

37. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
SETUP IFTTT FOR FACE RECOGNITION WITH ALEXA
38

38. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
RECOGNIZE PEOPLE
39
Alexa, Verify
User
Oracle Application
Container Cloud
(node webhook)
User Identified:
Username is:
xyz

39. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential40
LIVE
DEM
O

40. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
PROGRAM AGENDA
Introducing Chatbots
Towards Digital Assistants
Speech and Face Recognition
Putting things together
Summary
2
3
4
41
1
5

41. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
• RaspberryPi 2
– Camera
– Speaker + Microphone
– Wifi
– Servos
• On the RaspberryPi
– AlexaPi to listen and speak
– AlexaPi to control motion and lights
– RaspberryPi Cam to look at faces
– Face_recognition to recognize
• Coding
– Python
– Node
– OBotML
• Oracle ADA Cloud
• Alexa Skill to invoke PizzaBot or
whatever other thing you want
(such as IFTTT)
• IFTTT Configuration
42
TECH SUMMARY

42. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
• What I learned
– Raspberry Pi Rocks
– Experimenting with new Tech is fun
– Think Different
– Many many Solutions Available
• My view of the near future:
– Chatbots
• Will no no longer simply Chat
• Will Be Digital Assistants
• Will Be Autonomous
– We will ALL have our own Autonomous
Digital Assistant very soon.
– It can do a lot more than just chat
43
SUMMARY

43. Copyright © 2018, eProseed and/or its affiliates. All rights reserved. | Confidential
Experts in Modern Development
• Cloud
• Microservices and Containers
• Java, JavaScript/Node.js, PHP, Python
• DevOps
developer.oracle.com/ambassador @groundbreakers
• Continuous Delivery
• Open Source Technologies
• SQL/NoSQL Databases
• Machine Learning, AI, Chatbots

44. 45