This document summarizes the results of testing three chatbots: Eliza, Alice, and Captain Kirk. It finds that Eliza only responds to keywords with questions, while Alice answers most questions properly and interacts by asking follow-ups. Captain Kirk's answers are sometimes the same as Alice's, but sometimes don't make sense. Overall, Alice is deemed the most intelligent and convincing as a human.

1. ChatBot Analysis
Eliza, Alice & Captain Kirk
Aung Thu Rha Hein

2. Introduction
• ChatBot- 3 main parts include in ChatBox-
• Parser/NLP
• Knowledge
• Reply/NLG
• Also known as ‘chatterbot’, ’Conversational Agents’

3. Evaluation Method
• Organize the questions into 4 categories according to
complexity
» Basic conversation
» Asking Opinion
» Knowledge
» Basic problem solving
• Turing test- C-Test
• Measure the intelligence based on quantity of
questions that can respond properly

4. Experiments (Q & A test cases)
Question Eliza Alice Captain
Types Kirk
Basic Conversation Respond with 70% Same as Alice
questions
Asking Opinion Respond with 60% 50%
questions
Knowledge Respond with 80% Same as Alice
questions
Problem Solving Respond with 30% 20%
questions

5. Analysis & Results
Eliza
• Psychiatrist
• only responding you with questions
• Trigger by only keywords
• Not intelligence
ALICE
• answer most of the questions properly
• Also interactive and ask you some follow-ups
• Can talk about most topics
Captain Kirk
• Some answers are the same as ALICE
 Especially in Knowledge and Opinion sections
• Answers doesn’t make sense in some questions

6. Discussions & Conclusions
• Eliza is design for medical purpose and have only machine
intelligence level
• Difficult to tell which bots in intelligence between Captain
Kirk and ALICE
– Using same resource
• In my opinion, ALICE is more convincing as humans