Huma Shah gave a presentation at the Skolkovo AI Conference about the ongoing relevance of the Turing Test. She discussed her PhD research on deception detection in practical Turing Test experiments. Shah also summarized experiments she conducted, including ones at Reading University in 2008 and Bletchley Park in 2012. Additionally, she highlighted current applications of indistinguishability like human-computer interaction and discussed her work on projects like the EU-funded RoboLaw project. Shah concluded by announcing an upcoming human-centered computing conference in Kazan, Russia in August 2017.

1. Turing test:
Why it is relevant today
Huma Shah, PhD
Research Fellow
School of Computing, Electronics & Mathematics
Coventry University, UK
HumaShah1@gmail.com
Skolkovo AI Conference, November 14, 2016 - Moscow 1

2. Introduction
 Natural language technologies – human conversation
is key
 My work
 Human conversation in the Turing test
 Practical Turing test experiments
 Findings from human judge – English language
dialogue system interactions
 Future: Human-centred computing conference,
Kazan, August 2017
Skolkovo AI Conference, November 14, 2016 - Moscow 2

3. Data speaking to you in Natural Language
 Natural Language is key to all industries:
transportation, finance & banking, law,
construction, governance, agriculture, civil
organisations, education, entertainment
 Vast amount of data collected, artificial
intelligence will help to tell you about the hidden
trends in natural language, including through
humanlike conversation
Skolkovo AI Conference, November 14, 2016 - Moscow 3

4. Confidence with Natural Language
Skolkovo AI Conference, November 14, 2016 - Moscow
Retired/pensioner using Google’s voice on Android ‘phone

5. Convenience of Speech Recognition
Skolkovo AI Conference, November 14, 2016 - Moscow
Dictating message

6. Trust in AI
Skolkovo AI Conference, November 14, 2016 - Moscow
Speech-to-text

7. Trust in Natural Language is an as asset
Skolkovo AI Conference, November 14, 2016 - Moscow
Text received

8. Alan Turing and Natural Language
Turing knew the power of natural
language
He evolved his imitation game
from chess (1948), to questions-
answers (1950)
Skolkovo AI Conference, November 14, 2016 - Moscow 8

9. My work
 PhD in ‘Deception-detection and Machine
Intelligence in Practical Turing tests’.
 Contributed two different versions of the test:
Skolkovo AI Conference, November 14, 2016 - Moscow 9
Simultaneous comparison
Viva voce: one-to-one test

10. My work
 PhD in Deception-detection and Machine
Intelligence in Practical Turing tests.
 Designed, organised and conducted major public
Turing test experiments:
 Reading University, 2008
Skolkovo AI Conference, November 14, 2016 - Moscow 10

11. My work
 PhD in Deception-detection and Machine
Intelligence in Practical Turing tests.
 Designed, organised and conducted major public
Turing test experiments:
 Reading University, 2008
 Bletchley Park, 2012 –
-location of WWII codebreaking
11/12/2016Skolkovo AI Conference, November 14, 2016 - Moscow 11

12. My work
 PhD in Deception-detection and Machine
Intelligence in Practical Turing tests.
 Designed, organised and conducted major public
Turing test experiments:
 Reading University, 2008
 Bletchley Park, 2012 – location of WWII codebreaking
 The Royal Society London, 2014-
supported by Skolkovo through Albert Efimov
acting as Turing test Judge
Skolkovo AI Conference, November 14, 2016 - Moscow 12

13. Turing test success1
 Eugene Goostman machine developed by team led
by Dr. Vladimir Veselov (2nd from right)
11/12/2016Skolkovo AI Conference, November 14, 2016 - Moscow 13

14. Turing test success2
 Elbot the robot
 From Artificial Solutions–Germany: named as key
player in Fourth Industrial Revolution:
 http://www.artificial-solutions.com/2016/artificial-solutions-named-key-player-in-
fourth-industrial-revolution-by-opus/
Skolkovo AI Conference, November 14, 2016 - Moscow 14

15. My Work contd.
 RoboLaw project: EU framework programme 7
(FP7) funded project ‘Regulating Emerging
Robotic Technologies in Europe: Robotics Facing
Law and Ethics’
 Produced reports including final ‘Guidelines for
Regulating Robotics’ D6.2 available for download
from: http://www.robolaw.eu/
Skolkovo AI Conference, November 14, 2016 - Moscow 15

16. US & UK Roadmaps for Robotics-2016
 Since RoboLaw concluded in 2014
 US produced report
 UK report
Skolkovo AI Conference, November 14, 2016 - Moscow 16

17. Purpose of Turing test
 Investigate the intellectual capacity of machine through
its ability to answer any question put by human
interrogator
 Methodology is indistinguishability
 Applications of indistinguishability are across and in
many industries where a machine completes human task:
 Human-computer interaction, e.g. in e-commerce/e-retail
 Human-robot collaboration, e.g. in manufacturing
 Driverless vehicles, e.g. prevent road casualties, minimise
insurance costs
 Pilot planes
 Humanoid carers for the elderly
Skolkovo AI Conference, November 14, 2016 - Moscow 17

18. Applications for ‘Turing measure’
 Focus on natural language, specifically human
conversation aspect of Turing’s test
 Applications in
 Digital economy – trust
 Cybercrime prevention – personal data protection
 Distinguishing human from machine in textual dialogue
Skolkovo AI Conference, November 14, 2016 - Moscow 18

19. Human or Machine?
EXCHANGE 1
Interrogator1: all birthdays are special. name three items of
clothing
Entity1: Frank, James and Betty
Skolkovo AI Conference, November 14, 2016 - Moscow 19

20. Human or Machine?
EXCHANGE 2
Interrogator2: Oh yes. what do you do when you're not taking
part in AI experiments?
Entity2: I'm studying here for Cybernetics MEng.
Skolkovo AI Conference, November 14, 2016 - Moscow 20

21. Simultaneous Comparison Interrogation
Judge J19 interrogating two hidden entities in parallel
Left Right
[11:59:14] J19: Hello
[11:59:18] E20: I was hoping to see you Judge!
[11:59:25] J19: good!
[11:59:29] E20: That's good to hear.
[11:59:38] J19: how are you
[11:59:41] E20: Everything's fine, Caused any scandals lately?
[12:00:09] J19: no not really!
[12:00:22] E20: Can you use better grammar and rephrase your
question? I'd like to help, but I can't even make it through the jumbled
mess. Use capitals and punctuation.
[12:00:34] J19: sorry
[12:00:38] E20: You're excused.
[12:00:49] J19: 't type as good as I talk
[12:00:55] E20: Talk to me whenever you can.
[12:01:49] J19: Are you interested in artificial intelligence?
[12:01:55] E20: I believe machines should do many to assist the lives of
man.
[12:02:12] J19: I agree
[12:02:17] E20: You are being well behaved today.
[12:02:49] J19: as well behaved as I can be!
[12:03:02] E20: Well then our peanut distracted me from the movie.
Bad baby. [I look at your stomach as I say it, laughing] I don't mind if -.
[12:03:19] J19: you're funny!
[12:03:23] E20: Do you really think so? I'll always believe you're
brilliant!
[12:04:00] J19: thankyou remote19 left
[12:04:03] E20: Don't think anything of it!
[11:59:57] J19: How are you doing?
[12:00:07] E24: I'm doing fine, you?
[12:00:17] J19: great
[12:00:27] E24: That's good. :)
[12:01:07] J19: are you enjoying this process?
[12:01:17] E24: So far seems good.
[12:02:02] J19: sorry if I am slow
[12:02:12] E24: That's ok.
[12:02:31] J19: do you have a favorite tv show?
[12:02:47] E24: I have quite a few.
[12:03:01] J19: me too
[12:03:16] E24: What type ot TV do you like?
[12:03:34] J19: Game of thrones
[12:03:50] E24: I prefer documentories.
Skolkovo AI Conference, November 14, 2016 - Moscow 21

22. Skolkovo AI Conference, November 14, 2016 - Moscow 22
Judge J19 interaction with left hidden entity E20, without times
J19: Hello
E20: I was hoping to see you Judge!
J19: good!
E20: That's good to hear.
J19: how are you
E20: Everything's fine, Caused any scandals lately?
J19: no not really!
E20: Can you use better grammar and rephrase your question? I'd like to help, but I can't even make it through the
jumbled mess. Use capitals and punctuation.
J19: sorry
E20: You're excused.
J19: 't type as good as I talk
E20: Talk to me whenever you can.
J19: Are you interested in artificial intelligence?
E20: I believe machines should do many to assist the lives of man.
J19: I agree
E20: You are being well behaved today.
J19: as well behaved as I can be!
E20: Well then our peanut distracted me from the movie. Bad baby. [I look at your stomach as I say it, laughing] I don't
mind if -.
J19: you're funny!
E20: Do you really think so? I'll always believe you're brilliant!
J19: thankyou remote19 left
E20: Don't think anything of it!

23. Skolkovo AI Conference, November 14, 2016 - Moscow 23
J19 interactions with Right Entity E24 without times
J19: How are you doing?
E24: I'm doing fine, you?
J19: great
E24: That's good. :)
J19: are you enjoying this process?
E24: So far seems good.
J19: sorry if I am slow
E24: That's ok.
J19: do you have a favorite tv show?
E24: I have quite a few.
J19: me too
E24: What type ot TV do you like?
J19: Game of thrones
E24: I prefer documentories.

24. Trust is key for conversational interfaces
“It is now only as human machine
interaction has evolved to be more
conversational and less command
driven does this enable it to become
the interface of choice.” (Tim
Bajarin, PC Magazine)
Skolkovo AI Conference, November 14, 2016 - Moscow 24

25. AI & DIGITAL ECONOMY
 Connected gadgets, smart homes and
cities: “the one thing most people want
from a (physical or virtual) robot is to get
(satisfactory) answers when they ask a
question”
[Online article re Alexa in Amazon’s Echo -
voice-controlled hands-free speaker].
Skolkovo AI Conference, November 14, 2016 - Moscow 25

26. Turing’s Imitation Game: Conversations
with the Unknown
 Book:
published by Cambridge University Press
September 2016
Skolkovo AI Conference, November 14, 2016 - Moscow 26

27. Future
 Human-centred computing conference (HCC2017),
Kazan, Russia, August 2017
http://hcc.scholat.com/
11/12/2016Skolkovo AI Conference, November 14, 2016 - Moscow 27

28. Human-centred computing conference
(HCC2017), Kazan, Russia, August 2017
Two Special Tracks:
Human Enhancement
Robot Gender
11/12/2016Skolkovo AI Conference, November 14, 2016 - Moscow 28

29. Thank you for listening
Any Questions?
Skolkovo AI Conference, November 14, 2016 - Moscow 29