From Eliza to Eugene:Approaching Fifty Years ofDialogue AgentsDr. Huma ShahAINL, St. Petersburg, 17-18 May, 2013The Eliza ...
Overview of today’s talk• Brief outline of current work on RoboLaw• Quick Delegate participation / exercise• Comparison of...
AINL, St. Petersburg, 17-18 May, 2013Research Fellow: RoboLawProject funded byEU FP7 Science in Society
AINL, St. Petersburg, 17-18 May, 2013I work with (my boss), Professor Kevin Warwick
In case you don’t know who he isAINL, St. Petersburg, 17-18 May, 2013Kevin Warwick, first human cyborg: http://www.kevinwa...
Main Role of UoR in RoboLaw• Contribute to– Taxonomy of robotic & emerging technologies– Investigate Human Enhancement & H...
UoR Bio-inspired quad-rotor droneAINL, St. Petersburg, 17-18 May, 2013
Drone discussion on BBCAINL, St. Petersburg, 17-18 May, 2013BBC 2 Newsnight Thursday 30 August 2012Professor Warwick inter...
UoR Robo-companion forelderlyhttp://www.companionable.net/index.php?option=com_content&view=category&id=7:project-partner...
AINL, St. Petersburg, 17-18 May, 2013Leo Millar (age 10) withoutfingers on his right-hand fittedwith special bionic grip h...
AINL, St. Petersburg, 17-18 May, 2013MOD Craig Lundberg soldierblinded in Iraq sees withBrainPort vision tongue deviceMore...
AINL, St. Petersburg, 17-18 May, 2013Honda’s Asimo technology leadsto ‘Stride Assist’ device aboveand in use for “elderind...
Rewalk assistAINL, St. Petersburg, 17-18 May, 2013Rewalk from Cyclone Technologies:http://www.cyclonemobility.com/rewalk/P...
Rex bionicsAINL, St. Petersburg, 17-18 May, 2013Robotic exoskeleton, Rexprovides solution enablingwheelchair users (and ot...
Rethinking ‘Able’ and ‘Impairment’AINL, St. Petersburg, 17-18 May, 2013Enhanced -impaired Athlete Johnnie Peacock:T43 (sin...
Delegate ParticipationAINL, St. Petersburg, 17-18 May, 2013
Quick Transcript Reading• Please read two real conversations from aTuring test (first hand out sheet): transcript 1and tra...
Transcripts 1 & 2Transcript 1L LEFT: J11c with Entity 1H / M / UnsureTrancsript 2: RIGHT: J11c with Entity 2H / M / Unsure...
Did you determine correctly?Which was the human entity and which was theartificial dialogue system?AINL, St. Petersburg, 1...
Transcripts 1 & 2Transcript 1L LEFT: J11c with Entity 1H / M / UnsureTrancsript 2: RIGHT: J11c with Entity 2H / M / Unsure...
Eliza Comparison StudyAINL, St. Petersburg, 17-18 May, 2013
Artificial Dialogue: ElizaAINL, St. Petersburg, 17-18 May, 2013Joseph Weizenbaum (1923-2008)Creator of pre-Internet dialog...
Eliza 1966• 200 lines of input/output code.• Developed to “imitate a psychiatrist by employing a smallset of simple strate...
Sample Eliza DialogueMen are all alike.IN WHAT WAY?Theyre always bugging us about something or other.CAN YOU THINK OF A SP...
Contd 1DO YOU THINK COMING HERE WILL HELP YOU NOT TOBE UNHAPPYI need some help, that much seems certain.WHAT WOULD IT MEAN...
Contd 2YOUR FATHERYou are like my father in some ways.WHAT RESEMBLANCE DO YOU SEEYou are not very aggressive but I think y...
Eliza and deception• Ned Block (1981) on Eliza: a computerprogramme– “totally without intelligence”– “remarkably good at f...
Talking to Eliza• Some people interacting with Eliza notonly showed respect to the computer butpreferred interacting with ...
PARRY system• 1970s:- a computer model “of a mythical 28 year old,single, Caucasian, native English man” completewith para...
Fast Forward 2008: Modern Dialogist“The importance of Eugene correctly answeringthe question["My car is red. What color is...
Eugene Goostman Performance• Times newspaper journalist at ReadingUniversity’s 2008 Turing test contest: confusedEugene Go...
Lead scientist of Eugene Goostman• Vladimir Veselov – outside The Colonnade Hotel Maida Vale inJune 2012 (hotel is locatio...
Dialogue Softbots in E-commerce• FAQ systems: 02, Sgt Star, Asda, Sky, IkeaAINL, St. Petersburg, 17-18 May, 2013
Sgt Star Dialogue System• Funded by the US Army:– The SGT Star project brings the web-based character fromGoArmy.com to a ...
Two chatbots meetMy name is Sgt. STAR, which stands for Strong, Trained, andReady. I’m an artificial intelligence agent cr...
Advantages of Artificial Dialogists inE-commerce• Customer satisfaction up by 10%• Call centre workload down by 20%• Retur...
Why Compare Eliza?• To challenge claim Modern Dialogue systemsare:“just like”“same old Elizas”- By Floridi et al. (2009)• ...
Distinguishing Machine from Human• No basis for claim modern dialoguesystems are the same old Elizas• In 2008 Turing Tests...
Artificial Dialogue Study• Designed simple one-to-one experiment usingAlan Turing’s Imitation Game: viva voce one-to-one t...
Turing’s viva voce test• Chess imitation game: human chess playerplaying an unseen player and tries todetermine if he is p...
Turing 100in2012• Part of international commemorations of100th anniversary of Alan Turing’s birth• Reading University orga...
Arranging Study• Internet-based experiment• Persuaded current best known developers inArtificial Dialogue from Turing test...
The Hidden Dialogists• Identified in study as:– E1: Eliza– E6: http://66.36.243.63/– E12: http://bd1.artificial-solutions....
The Judge/Interrogators• Males females• School pupils, university students• Members of the public answering calls forparti...
Human Judges’ task• Complete short questionnaire• Chat to six ‘entities’ on web sites• Judges informed one entity is defin...
Judge QuestionnaireAINL, St. Petersburg, 17-18 May, 2013SexAge RangeFirst language
Human JudgesJudgeInformationSex Age English/Non-EnglishUseSocialMediaPreviousinteractionwithachatbotKnowledge of privacyan...
Judge Score SheetAINL, St. Petersburg, 17-18 May, 2013
Entity ScoresEntity No ofreturnedscoresScoreRangeMeanconversationabilityscore%Mode(mostgivenscore)No of scores100=humanlik...
Results: 2012 Comparison of ElizaAINL, St. Petersburg, 17-18 May, 20130,0010,0020,0030,0040,0050,0060,0070,00Eliza Ultra H...
Judge feedback: Eliza• Eliza achieved mean score of almost 26% forconversation ability, which is halfway to “goodconversat...
Softbot Performance• Overall comments portray machine/artificiallinguistic progress• Judge J41 (male, age 25-44, Serbian, ...
Sample Conversation: Eugene E23User: J53 (male, 19-24, Spanish/Catalonia)User: hiBot (Eugene): Hi. I hope youll enjoy our ...
Artificial Dialogue Strategy• Attempt to take hold and direct theconversation towards its knowledge base:– Where are you f...
Same old technique?• Modern Elizas are, in a way, using a strategythat Weizenbaum’s Eliza system did• It is the same old t...
Teacher Sevenoaks School, UK“..students .. enjoyed their opportunity toparticipate in the Reading University computerdecep...
Ultra Hal• E6AINL, St. Petersburg, 17-18 May, 2013http://www.zabaware.com/assistant/
Elbot• E12AINL, St. Petersburg, 17-18 May, 2013http://www.elbot.com/
Cleverbot• E19AINL, St. Petersburg, 17-18 May, 2013http://www.cleverbot.com/
Eugene Goostman• E23AINL, St. Petersburg, 17-18 May, 2013http://www.princetonai.com/bot/bot.jsp
JFred / TuringHub• E41AINL, St. Petersburg, 17-18 May, 2013http://testing.turinghub.com/
Study Review• Comparing the incomparable:AINL, St. Petersburg, 17-18 May, 2013Eliza Modern SoftbotsSingle-domain: develope...
Analysis• Floridi et al. claim is wrong, however softbotslinguistically lacking, for example, unable tocreate:– Analogies ...
Softbot Possibilities• IBM Watson reverse question-answer systemshowed what can be achieved (2011)- Final Jeopardy! Man vs...
Future of Dialogue Systems• Personal cyber information-gatherers onwearable smart devices• Mother of all libraries at our ...
Finally: New Alan Turing bookMy chapter ‘Conversation, Deception andIntelligence: Turing’s Imitation Game’ in Part III:‘Bu...
References• Block, N. (1981). Psychologism and Behaviorism. In (Ed) Shieber, S. The Turing Test: VerbalBehavior as the Hal...
Acknowledgements– AINL 2013– European Union Seventh Framework Programme(FP72007-2013) under grant agreement no.289092. Rob...
Thank you for listening and any questions….AINL, St. Petersburg, 17-18 May, 2013
Upcoming SlideShare
Loading in …5
×

Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents

3,411 views

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
3,411
On SlideShare
0
From Embeds
0
Number of Embeds
140
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents

  1. 1. From Eliza to Eugene:Approaching Fifty Years ofDialogue AgentsDr. Huma ShahAINL, St. Petersburg, 17-18 May, 2013The Eliza study and results are being written up in a paper for submission to a ComputerScience Journal
  2. 2. Overview of today’s talk• Brief outline of current work on RoboLaw• Quick Delegate participation / exercise• Comparison of Artificial Dialogue systems (softrobots / virtual embodied entities)– Eliza / Modern Elizas/ Dialogists in E-commerce– Why compare Eliza with Modern Dialogists?– The study/experiment– Results/significance– Future of Dialogue Systems/ softbotsAINL, St. Petersburg, 17-18 May, 2013
  3. 3. AINL, St. Petersburg, 17-18 May, 2013Research Fellow: RoboLawProject funded byEU FP7 Science in Society
  4. 4. AINL, St. Petersburg, 17-18 May, 2013I work with (my boss), Professor Kevin Warwick
  5. 5. In case you don’t know who he isAINL, St. Petersburg, 17-18 May, 2013Kevin Warwick, first human cyborg: http://www.kevinwarwick.com/
  6. 6. Main Role of UoR in RoboLaw• Contribute to– Taxonomy of robotic & emerging technologies– Investigate Human Enhancement & Human/RobotRights in a world of robotic technologies• Consider ethical, legal & social implications ofhuman enhancement• Rethinking human capacities and disability in afuture of emerging technologies• Policy Recommendations: contribute toWhite Paper for EU CommissionAINL, St. Petersburg, 17-18 May, 2013
  7. 7. UoR Bio-inspired quad-rotor droneAINL, St. Petersburg, 17-18 May, 2013
  8. 8. Drone discussion on BBCAINL, St. Petersburg, 17-18 May, 2013BBC 2 Newsnight Thursday 30 August 2012Professor Warwick interviewed about privacy issues surrounding ‘civilian’ drone use
  9. 9. UoR Robo-companion forelderlyhttp://www.companionable.net/index.php?option=com_content&view=category&id=7:project-partners&Itemid=12&layout=defaultAINL, St. Petersburg, 17-18 May, 2013Hector, the assistive companionrobot.- in emergencies, such as afall, Hector can help a remotecontrol centre assess howserious the fall is and whatkind of emergency help maybe needed- reminders for takingmedicines on time- diary management- Hector meeting EU PresidentJose Manuel Barroso inBrussels, December 5-6, 2011
  10. 10. AINL, St. Petersburg, 17-18 May, 2013Leo Millar (age 10) withoutfingers on his right-hand fittedwith special bionic grip helps himachieve at playing golf.More here: http://www.dailymail.co.uk/health/article-2012179/Golf-Bionic-glove-help-golfing-prodigy-10-professional-despite-having-ONE-HAND.html
  11. 11. AINL, St. Petersburg, 17-18 May, 2013MOD Craig Lundberg soldierblinded in Iraq sees withBrainPort vision tongue deviceMore here:: http://www.mod.uk/DefenceInternet/DefenceNews/PeopleInDefence/TechnologyHelpsBlindedSoldierseeThroughHisTongue.htm
  12. 12. AINL, St. Petersburg, 17-18 May, 2013Honda’s Asimo technology leadsto ‘Stride Assist’ device aboveand in use for “elderindependence solutions”More here: http://world.honda.com/news/2012/c120729Stride-Management-Assist-Device/?from=r
  13. 13. Rewalk assistAINL, St. Petersburg, 17-18 May, 2013Rewalk from Cyclone Technologies:http://www.cyclonemobility.com/rewalk/Paralysed Claire Lomas completes 2012 London Marathon in robot suit:http://news.cnet.com/8301-17938_105-57431108-1/paralyzed-woman-completes-london-marathon-in-robot-suit/
  14. 14. Rex bionicsAINL, St. Petersburg, 17-18 May, 2013Robotic exoskeleton, Rexprovides solution enablingwheelchair users (and otherswith a mobility impairment) tostand and walk, allowing themto be independently uprightand mobile:http://www.rexbionics.com/aboutrex.php
  15. 15. Rethinking ‘Able’ and ‘Impairment’AINL, St. Petersburg, 17-18 May, 2013Enhanced -impaired Athlete Johnnie Peacock:T43 (single amputee) /T44 (double amputee)100m London 2012 Paralympics fastest:10.90sAble-bodied: Usain Bolt runsLondon 2012 Olympics 100m in9.63sImpaired: T13athlete JasonSmyth with <10%vision ran 100m in10.46s
  16. 16. Delegate ParticipationAINL, St. Petersburg, 17-18 May, 2013
  17. 17. Quick Transcript Reading• Please read two real conversations from aTuring test (first hand out sheet): transcript 1and transcript 2, between a humanjudge/interrogator (J11c) and two differenthidden interlocutors• Transcript 1: is the judge talking to a humanor a machine?• Transcript 2: is judge talking to a human or amachine?AINL, St. Petersburg, 17-18 May, 2013
  18. 18. Transcripts 1 & 2Transcript 1L LEFT: J11c with Entity 1H / M / UnsureTrancsript 2: RIGHT: J11c with Entity 2H / M / UnsureJudge: WHAT MUSIC DO YOU LIKE?Entity: electronica mainly and a bit of symphonic rockJudge: omg i lyke teenie tamparrrEntity: Trying to get into Grimes at the moment but strugglingJudge: yeah i knoooEntity: School of Seven Bells is the best new band Ive heard recentlyJudge: i see dead peopleEntity: Dead Cities was a good album, cant remember who byJudge: i think it was a band called the sixth sense?Entity: Sounds about right, perhaps theres a ghost in the machineJudge: yes indeed.Entity: Ever seen ghost in the shell?Judge: Casper the friendly ghost.Judge: what music do you like?Entity: I like Eminem - because he doesnt have a thing to do with "polite-correctness".Judge: OMG i love eminem, hes my favourite artist!Entity: I wonder what Eminems nick mean. I associate it only with a word"enema". But he is a cool guy, I like his "The Real Slim Shady".Judge: i know what his nick name means, his real name is Marshall Mathers.From the initials M&M phynetically sounding like Eminem.Entity: Eminem is a famous crapper. No, raper... ripper... Yeah, I mean - rapper. Ilike his "Stan" and "The Real Slim Shady".Judge: I like "Hailies song"Entity: Interesting tastes. I like science fiction books, by the way. Where do youcame from, by the way? Could you tell me about the place where you live?Judge: Epsom, home of the derby. yourself?Entity: My guinea pig says that name Derby sounds very nice.Judge: i shopuld talk with him sometimeEntity: I think you should take some typing lessons. Or just type more accurately.Judge: thats very rude?Entity: Ummm... Frankly, I didnt get your question:-(Judge: "Ummm" isnt a word. i think you need typing lessonsEntity: My thoughts are same. By the way, I still dont know your specialty - or,possibly, Ive missed it?AINL, St. Petersburg, 17-18 May, 2013
  19. 19. Did you determine correctly?Which was the human entity and which was theartificial dialogue system?AINL, St. Petersburg, 17-18 May, 2013
  20. 20. Transcripts 1 & 2Transcript 1L LEFT: J11c with Entity 1H / M / UnsureTrancsript 2: RIGHT: J11c with Entity 2H / M / UnsureJudge: WHAT MUSIC DO YOU LIKE?Entity: electronica mainly and a bit of symphonic rockJudge: omg i lyke teenie tamparrrEntity: Trying to get into Grimes at the moment but strugglingJudge: yeah i knoooEntity: School of Seven Bells is the best new band Ive heard recentlyJudge: i see dead peopleEntity: Dead Cities was a good album, cant remember who byJudge: i think it was a band called the sixth sense?Entity: Sounds about right, perhaps theres a ghost in the machineJudge: yes indeed.Entity: Ever seen ghost in the shell?Judge: Casper the friendly ghost.Judge: what music do you like?Entity: I like Eminem - because he doesnt have a thing to do with "polite-correctness".Judge: OMG i love eminem, hes my favourite artist!Entity: I wonder what Eminems nick mean. I associate it only with a word"enema". But he is a cool guy, I like his "The Real Slim Shady".Judge: i know what his nick name means, his real name is Marshall Mathers.From the initials M&M phynetically sounding like Eminem.Entity: Eminem is a famous crapper. No, raper... ripper... Yeah, I mean - rapper. Ilike his "Stan" and "The Real Slim Shady".Judge: I like "Hailies song"Entity: Interesting tastes. I like science fiction books, by the way. Where do youcame from, by the way? Could you tell me about the place where you live?Judge: Epsom, home of the derby. yourself?Entity: My guinea pig says that name Derby sounds very nice.Judge: i shopuld talk with him sometimeEntity: I think you should take some typing lessons. Or just type more accurately.Judge: thats very rude?Entity: Ummm... Frankly, I didnt get your question:-(Judge: "Ummm" isnt a word. i think you need typing lessonsEntity: My thoughts are same. By the way, I still dont know your specialty - or,possibly, Ive missed it?AINL, St. Petersburg, 17-18 May, 2013Left entity: human, male English adult – Right entity: machine
  21. 21. Eliza Comparison StudyAINL, St. Petersburg, 17-18 May, 2013
  22. 22. Artificial Dialogue: ElizaAINL, St. Petersburg, 17-18 May, 2013Joseph Weizenbaum (1923-2008)Creator of pre-Internet dialogue system, Eliza (1966) toinvestigate natural language understandingWeb-basedversion of Elizahttp://nlp-addiction.com/eliza/
  23. 23. Eliza 1966• 200 lines of input/output code.• Developed to “imitate a psychiatrist by employing a smallset of simple strategies” extracting “minimal context”(Block, 1981: p. 233).• System applied a ‘rank’ or “precedence number” to akeyword in the input sentence such as I, you, boyfriend.• Applying appropriate transformation, for instance turning“you” into “I”.• Eliza demonstrated system responding “roughly as wouldcertain psychotherapists (Rogerian)” (Weizenbaum,1966).AINL, St. Petersburg, 17-18 May, 2013
  24. 24. Sample Eliza DialogueMen are all alike.IN WHAT WAY?Theyre always bugging us about something or other.CAN YOU THINK OF A SPECIFIC EXAMPLE?Well, my boyfriend made me come here.YOUR BOYFRIEND MADE YOU COME HEREHe says Im depressed much of the time.I AM SORRY TO HEAR YOU ARE DEPRESSEDIts true. I am unhappyAINL, St. Petersburg, 17-18 May, 2013
  25. 25. Contd 1DO YOU THINK COMING HERE WILL HELP YOU NOT TOBE UNHAPPYI need some help, that much seems certain.WHAT WOULD IT MEAN TO YOU IF YOU GOT SOMEHELPPerhaps I could learn to get along with my mother.TELL ME MORE ABOUT YOUR FAMILYMy mother takes care of me.WHO ELSE IN YOUR FAMILY TAKES CARE OF YOUMy fatherAINL, St. Petersburg, 17-18 May, 2013
  26. 26. Contd 2YOUR FATHERYou are like my father in some ways.WHAT RESEMBLANCE DO YOU SEEYou are not very aggressive but I think you dont want me to notice that.WHAT MAKES YOU THINK I AM NOT VERY AGGRESSIVEYou dont argue with me.WHY DO YOU THINK I DONT ARGUE WITH YOUYou are afraid of me.DOES IT PLEASE YOU TO BELIEVE I AM AFRAID OF YOUMy father is afraid of everybody.WHAT ELSE COMES TO YOUR MIND WHEN YOU THINK OF YOUR FATHERBullies.DOES THAT HAVE ANYTHING TO DO WITH THE FACT THAT YOUR BOYFRIEND MADEYOU COME HEREAINL, St. Petersburg, 17-18 May, 2013
  27. 27. Eliza and deception• Ned Block (1981) on Eliza: a computerprogramme– “totally without intelligence”– “remarkably good at fooling people in shortconversations”- Weizenbaum’s secretary, one of the deceived,- “asked him to leave the room in order to talk tothe machine privately” (Block, 1981: p. 233).AINL, St. Petersburg, 17-18 May, 2013
  28. 28. Talking to Eliza• Some people interacting with Eliza notonly showed respect to the computer butpreferred interacting with machines overhumans:– “professing their feelings and struggles … evenseeking their empathy” (Ian Kerr, 2004: p.304).AINL, St. Petersburg, 17-18 May, 2013
  29. 29. PARRY system• 1970s:- a computer model “of a mythical 28 year old,single, Caucasian, native English man” completewith paranoid behaviour such as “fixeddelusions of persecution by gangsters”• In tests, psychiatrists found it difficult todistinguish PARRY from a 22 year-old humanpatient suffering from paranoia (Colby, 1971 &1972; Heiser et al., 1979)AINL, St. Petersburg, 17-18 May, 2013
  30. 30. Fast Forward 2008: Modern Dialogist“The importance of Eugene correctly answeringthe question["My car is red. What color is my car?"]- and then remembering its answer when laterasked cannot be understated. That is a GREATachievement.”[Judge comment on Eugene Goostman performance 2008 Reading University Turing tests]AINL, St. Petersburg, 17-18 May, 2013
  31. 31. Eugene Goostman Performance• Times newspaper journalist at ReadingUniversity’s 2008 Turing test contest: confusedEugene Goostman for a human in acomparison test with female human foil (whowas considered a machine!).AINL, St. Petersburg, 17-18 May, 2013
  32. 32. Lead scientist of Eugene Goostman• Vladimir Veselov – outside The Colonnade Hotel Maida Vale inJune 2012 (hotel is location of former Paddington Lodge where Alan Turingwas born in London, 1912)• Developer ofSPOONYAINL, St. Petersburg, 17-18 May, 2013
  33. 33. Dialogue Softbots in E-commerce• FAQ systems: 02, Sgt Star, Asda, Sky, IkeaAINL, St. Petersburg, 17-18 May, 2013
  34. 34. Sgt Star Dialogue System• Funded by the US Army:– The SGT Star project brings the web-based character fromGoArmy.com to a life-sized, realistic looking character who can visitconferences and travel in the Army adventure vans. The characterhas revised content as well as different NLP technology behind hisquestion-answering.http://projects.ict.usc.edu/nld/group/projects/sgt-star– SGT STAR: ”Hello. Im SGT STAR, the Virtual Guide for goarmy.com. Mytraining allows me to answer almost any question about what life islike in the US Army. Interacting with me is simple. Just type in yourquestion just as though you were chatting with a live recruiter.http://www.goarmy.com/ask-sgt-star.htmlAINL, St. Petersburg, 17-18 May, 2013
  35. 35. Two chatbots meetMy name is Sgt. STAR, which stands for Strong, Trained, andReady. I’m an artificial intelligence agent created for the U.S.Army to provide you with information about army life.I am Anna, the Ikea USA online assistant.AINL, St. Petersburg, 17-18 May, 2013http://web.archive.org/web/20080704142146/http://www.radaronline.com/features/2006/09/blind_date.phpAlan Turing (1951) predicted: machines would conversewith each other to sharpen their wits!
  36. 36. Advantages of Artificial Dialogists inE-commerce• Customer satisfaction up by 10%• Call centre workload down by 20%• Return on investment up by 200%(Chris Ezekiel, Creative Virtual, 2005)AINL, St. Petersburg, 17-18 May, 2013
  37. 37. Why Compare Eliza?• To challenge claim Modern Dialogue systemsare:“just like”“same old Elizas”- By Floridi et al. (2009)• What is the basis for this statement?AINL, St. Petersburg, 17-18 May, 2013
  38. 38. Distinguishing Machine from Human• No basis for claim modern dialoguesystems are the same old Elizas• In 2008 Turing Tests Floridi et al. not thatsuccessful at distinguishing human frommachine:–Correct recognition rate only 56% (Shah &Warwick, 2010)AINL, St. Petersburg, 17-18 May, 2013
  39. 39. Artificial Dialogue Study• Designed simple one-to-one experiment usingAlan Turing’s Imitation Game: viva voce one-to-one test toexamine machinethinking• Guidelines for humanand machineparticipants inweb-based studyAINL, St. Petersburg, 17-18 May, 2013
  40. 40. Turing’s viva voce test• Chess imitation game: human chess playerplaying an unseen player and tries todetermine if he is playing human or machine(Intelligent Machinery, 1948)NB: Man vs. machine chess challenge seen inhistoric Kasparov vs. Deep Blue 1997 match• Turing evolved his Imitation Game from chessto a machine dialogue test (ComputingMachinery and Intelligence, 1950)AINL, St. Petersburg, 17-18 May, 2013
  41. 41. Turing 100in2012• Part of international commemorations of100th anniversary of Alan Turing’s birth• Reading University organised publicawareness of Turing’s contribution tocomputer science:– Bletchley Park event, 23 June 2012– Year-round Eliza studyAINL, St. Petersburg, 17-18 May, 2013
  42. 42. Arranging Study• Internet-based experiment• Persuaded current best known developers inArtificial Dialogue from Turing test contests– Asked them to set up anonymous URLs– Make their dialogist available throughout 2012• Recruited judges via calls for participation,including social media (Twitter, Facebook,STEMNET UK, British ComputerSociety/Computing at School –BCS CAS)AINL, St. Petersburg, 17-18 May, 2013
  43. 43. The Hidden Dialogists• Identified in study as:– E1: Eliza– E6: http://66.36.243.63/– E12: http://bd1.artificial-solutions.com/_demos/entity12/cgi-bin/entity12.cgi– E19:http://entity19.turings.com/textevent?id=a1– E23: http://www.entry23.org/entry23/– E41: http://entity-41.org/Numbers 6, 12, 19, 23 & 41 relate to Turing(birth: 23 June 1912- 41 age at death)AINL, St. Petersburg, 17-18 May, 2013
  44. 44. The Judge/Interrogators• Males females• School pupils, university students• Members of the public answering calls forparticipation• English speakers• Non-Native English speakers (incl. Spanish,Chinese)• Age range: 13-64AINL, St. Petersburg, 17-18 May, 2013
  45. 45. Human Judges’ task• Complete short questionnaire• Chat to six ‘entities’ on web sites• Judges informed one entity is definitely amachine – Eliza – entity E1• Chat and score conversation ability (ratherthan say human or machine) of Eliza and fivehidden entities:– 0=machinelike 50= good conversationalist 100=humanlike• Return questionnaire & score sheetAINL, St. Petersburg, 17-18 May, 2013
  46. 46. Judge QuestionnaireAINL, St. Petersburg, 17-18 May, 2013SexAge RangeFirst language
  47. 47. Human JudgesJudgeInformationSex Age English/Non-EnglishUseSocialMediaPreviousinteractionwithachatbotKnowledge of privacyandInternet identity protection campaignsExperiencedidentitytheft/bankcardmisusedNo. M F <1819+E NE Yes Yes GetSafeOnlineGoogle‘Good toKnow’OthercampaignSamepassword118 72 29 24 87 34 83 111 18 14 26 18 32 12AINL, St. Petersburg, 17-18 May, 2013
  48. 48. Judge Score SheetAINL, St. Petersburg, 17-18 May, 2013
  49. 49. Entity ScoresEntity No ofreturnedscoresScoreRangeMeanconversationabilityscore%Mode(mostgivenscore)No of scores100=humanlikeconversationEntity 1 Eliza 118 0-100 25.78 0 1Entity 6 Ultra Hal 118 0-100 38.37 50 3Entity 12 Elbot 113 0-100 47.84 50 5Entity 19 Cleverbot 115 0-100 61.55 50 17Entity 23 EugeneGoostman114 0-100 63.24 50 15Entity 41 JFred/Turinghub103 0-100 46.51 50 6AINL, St. Petersburg, 17-18 May, 2013
  50. 50. Results: 2012 Comparison of ElizaAINL, St. Petersburg, 17-18 May, 20130,0010,0020,0030,0040,0050,0060,0070,00Eliza Ultra Hal Elbot Cleverbot Eugene Turinghub25,7838,3747,8461,5563,2446,51Mean Conversational Ability Score %
  51. 51. Judge feedback: Eliza• Eliza achieved mean score of almost 26% forconversation ability, which is halfway to “goodconversationalist” as defined in this study• Eliza scored maximum 100% from one judge:– J96, aged 19-24, Chinese speaker, no previousinteraction with chatbots– Gave scores of 10% for E6; 10% for E12; 65% forE19; 51% for E23, and 40% for E41• Conversation Analysis from judges moremeaningful than their subjective scoresAINL, St. Petersburg, 17-18 May, 2013
  52. 52. Softbot Performance• Overall comments portray machine/artificiallinguistic progress• Judge J41 (male, age 25-44, Serbian, Russian,Bulgarian speaker) feedback on conversationalexperience with Eliza and the five hidden entities:“During the interviews I could notice that either all entitieswere bots or at least one of them was a human who actedbot-like. If so, the person(s) succeeded mimicking bots. …”AINL, St. Petersburg, 17-18 May, 2013
  53. 53. Sample Conversation: Eugene E23User: J53 (male, 19-24, Spanish/Catalonia)User: hiBot (Eugene): Hi. I hope youll enjoy our conversation!User: I hope soBot: My hopes also! Lets continue. And I forgot to ask you where you arefrom...User: I am from SpainBot: Spain? Hope its a good place to live.User: Yes it isBot: Im glad you agreed. Some another topic?User: There are the best climate of the worldBot: And yes, do you make a secret of your name? I still dont know it!User: My name is OriolBot: Oriol? I dont really know such a name, but OK - Lets be friends!User: Its not an English nameBot: No? Well, it is up to you... Could you tell me about your job, by the way?User: I am studing at universityAINL, St. Petersburg, 17-18 May, 2013
  54. 54. Artificial Dialogue Strategy• Attempt to take hold and direct theconversation towards its knowledge base:– Where are you from?– Could you tell me about your job?• Give illusion of following conversation:– By asking questions, like Eliza does• Share personal interest:– User: Yes, what kind of music do you like?Bot: I like the young, but very talented Russianrap-singer Alla Pugatcheva.AINL, St. Petersburg, 17-18 May, 2013
  55. 55. Same old technique?• Modern Elizas are, in a way, using a strategythat Weizenbaum’s Eliza system did• It is the same old technique that humans use• Ask questions when we want to know moreabout, or become acquainted with someone• Modern dialogue systems are doing it betterthan Eliza 1966AINL, St. Petersburg, 17-18 May, 2013
  56. 56. Teacher Sevenoaks School, UK“..students .. enjoyed their opportunity toparticipate in the Reading University computerdeception project and despite almost beingdriven to distraction by Eliza they weregenuinely shocked by some of the moreadvanced Entities.”http://turing100.blogspot.co.uk/2012/08/guest-post-maths-teacher-david-vaccaro.htmlAINL, St. Petersburg, 17-18 May, 2013
  57. 57. Ultra Hal• E6AINL, St. Petersburg, 17-18 May, 2013http://www.zabaware.com/assistant/
  58. 58. Elbot• E12AINL, St. Petersburg, 17-18 May, 2013http://www.elbot.com/
  59. 59. Cleverbot• E19AINL, St. Petersburg, 17-18 May, 2013http://www.cleverbot.com/
  60. 60. Eugene Goostman• E23AINL, St. Petersburg, 17-18 May, 2013http://www.princetonai.com/bot/bot.jsp
  61. 61. JFred / TuringHub• E41AINL, St. Petersburg, 17-18 May, 2013http://testing.turinghub.com/
  62. 62. Study Review• Comparing the incomparable:AINL, St. Petersburg, 17-18 May, 2013Eliza Modern SoftbotsSingle-domain: developedto ‘listen’ as a RogerianpsychotherapistDeveloped to talk on ‘any subject’Polite Can be impolite and worse!Limited number ofresponsesPlethora of answersEmotionless Illusion of emotion throughpersonality/character (Elbot/Eugene)
  63. 63. Analysis• Floridi et al. claim is wrong, however softbotslinguistically lacking, for example, unable tocreate:– Analogies (e.g. “..St. Petersburg, most abstract andintentional city on the entire globe” Dostoyevsky in ‘Notesfrom Underground’)– Metaphors (e.g. phishing )• Slow progress/ lack of academic funding nothelped by pejorative view of Turing test(Hayes & Ford, 1995)AINL, St. Petersburg, 17-18 May, 2013
  64. 64. Softbot Possibilities• IBM Watson reverse question-answer systemshowed what can be achieved (2011)- Final Jeopardy! Man vs Machine US TV Quiz show• Best Modern Elizas are an improvement onWeizenbaum’s artificial psychotherapistAINL, St. Petersburg, 17-18 May, 2013
  65. 65. Future of Dialogue Systems• Personal cyber information-gatherers onwearable smart devices• Mother of all libraries at our fingertips/ideatelevel - we’ll all have one!AINL, St. Petersburg, 17-18 May, 2013Centralcharacter,HiroProtoganisthaspersonaldigitalLibrarian
  66. 66. Finally: New Alan Turing bookMy chapter ‘Conversation, Deception andIntelligence: Turing’s Imitation Game’ in Part III:‘Building a Brain: Intelligent Machines Practiceand Theory’Elsevier 2013http://store.elsevier.com/Alan-Turing-His-Work-and-Impact/isbn-9780123869807/?&locale=en_UKAINL, St. Petersburg, 17-18 May, 2013
  67. 67. References• Block, N. (1981). Psychologism and Behaviorism. In (Ed) Shieber, S. The Turing Test: VerbalBehavior as the Hallmark of Intelligence, 2004, MIT Press: UK: pp. 229 – 266• Hayes, P., and Ford, K. (1995). Turing Test Considered Harmful. Proceedings of the FourteenthInternational Joint Conference on Artificial Intelligence, 1:972–977• Heiser, J.F., Colby, K. M., Fraught, W.S. and Parkison, R.C. (1979). Can Psychiatrists Distinguisha Computer Simulation of Paranoia from the Real Thing?: The Limitation of Turing-like Testsas Measures of the Adequacy of Simulations. Journal of Psychiatric Research. Vol. 15, Issue 3:pp 149-162• Floridi, L., Taddeo, M. and Turilli, M. (2009). Turing’s Imitation Game - Still an ImpossibleChallenge for All Machines and Some Judges. An Evaluation of the 2008 LoebnerContest.Minds and Machines. 19(1), pp. 145-150.• Kerr, I. R. (2004). Bots, Babes and the Californication of Commerce. University of Ottowa Lawand Technology Journal. Vol. 1 (13), pp. 285-324• Shah, H. and Warwick, K. (2010). Hidden Interlocutor Misidentification in Practical Turingtests. Minds and Machines, Vol. 20 (3), pp. 441-454, DOI: 10.1007/s11023-010-9219-6• Turing, A.M. (1951). Intelligent Machinery, A Heretical Theory. In (Ed) B.J. Copeland). TheEssential Turing: The Ideas that Gave Birth to the Computer Age. Clarendon Press: Oxford,UK, 2004, pp 472-475• Turing, A.M. (1950). Computing, machinery and intelligence, Mind, LIX (236), pp. 433-460• Turing, A.M. (1948).“Intelligent Machinery”, in B.J. Copeland, The Essential Turing. Oxford:Clarendon, pp.410–432, 2004AINL, St. Petersburg, 17-18 May, 2013
  68. 68. Acknowledgements– AINL 2013– European Union Seventh Framework Programme(FP72007-2013) under grant agreement no.289092. RoboLaw – Regulating Emerging RobotTechnologies in Europe: Robotics facing law andethicsAINL, St. Petersburg, 17-18 May, 2013
  69. 69. Thank you for listening and any questions….AINL, St. Petersburg, 17-18 May, 2013

×