1. CIG 2014 - Human-Like Bots Competition
IEEE Computational Intelligence in Games. Dortmund, Germany. August, 2014
Organization:
Manuel G. Bedia
Juan Peralta
Joan Marc
Philip Hingston
Raúl Arrabales

2. Organization / Acknowledgements

3. Players (humans and bots)
PLAYER TYPE TEAM MEMBERS AFFILIATION COUNTRY
BotTracker BOT TETRIIS
Hunjoo Lee
Jee-Hyong Lee
ETRI,
Sungkyunkwan
University
SOUTH
KOREA
MirrorBot BOT IHSEV Mihai Polceanu
ENIB CERV Centre
de Réalité Virtuelle
FRANCE
NizorBot BOT
UMAG-BOT
José L. Jiménez López
Antonio J. Fernández-Leiva
Antonio M. Mora
Universidad de
Málaga
SPAIN
OvGUBot BOT OvGUBot
Xenija Neufeld
Sanaz Mostaghim
Otto von Guericke
University,
Magdeburg
GERMANY
ADANN BOT CVC
Juan Peralta Donate
Joan Marc Llargués A.
CVC. UAB SPAIN
CCBot BOT
Conscious
-Robots
Jorge Muñoz
Raúl Arrabales
Comaware SPAIN
Player HUMAN Judge - - -
Tmchojo HUMAN Judge - - -
Juan_CVC HUMAN Judge - - -
Xenija HUMAN Judge - - -

5. Original BotPrize Testing Protocol
(FPA – First-Person Assessment)
Human Judges
Artificial Bots UT 2004 Server
Real-time Online Anonymized interaction

6. BotPrize 2014 Edition: We add TPA
* FPA – First-Person Assessment
* TPA – Third-Person Assessment
Generation of Anonymized TPA Video
Clips featuring human and bot players
FPA Human Judges
Artificial Bots UT 2004 Server
TPA Judges
(Crowdsourcing
platform)
Third-Person
Crowdsourcing
Judging

7. BotPrize 2014 Edition: Humanness++ (H)
H = (FPA * FPWF) + (TPA * TPWF)
FPWF First-Person Weighting Factor = 0,5.
TPWF Third-Person Weighting Factor = 0,5.
FPA Human Judges
Artificial Bots UT 2004 Server
H = FPA / 2 + TPA / 2

8. Humanness scores based on SDT
Signal Detection Theory
Judge SDT Matrix Vote “Human” Vote “Bot”
Player is a Human Hit False Alarm
Player is a Bot Miss Hit
Tanner Jr., Wilson P.; John A. Swets (November 1954). "A decision-making theory of visual detection.". Psychological Review. 61 (6): 401–409.

9. Humanness scores based on SDT
Judge Reliability (JR)
A measure of how good a judge is in terms of
telling apart humans and bots
퐽푅푗 =
퐻푖푡푠 − (푀푖푠푠푒푠 + 퐹푎푙푠푒퐴푙푎푟푚푠
푁푗
Judge SDT Matrix Vote “Human” Vote “Bot”
Player is a Human Hit False Alarm
Player is a Bot Miss Hit

10. Judge Reliability can be used to adjust
Humanness Scores
Judge Relative Reliability (JRR)
A measure of how good a judge is in relation
with other judges
퐽푅푅푗 =
퐽푅푗
퐽 퐽푅푗
푗=1
퐴푣푔퐽푅 = 퐴푣푔퐽푅
퐽

11. Judge Reliability can be used to adjust
Humanness Scores
Judge Relative Reliability (JRR)
A measure of how good a judge is in relation
with other judges
judges JRmeasures JRR Weight
Player 0.28967254 1.2574682 0.31436704
tmchojo 0.43801653 1.9014292 0.47535731
Juan_CVC 0.16129032 0.7001611 0.17504027
Xenija 0.03246753 0.1409415 0.03523538

12. Judge Reliability can be used to
adjust Humanness Scores
Judge Relative Reliability (JRR)
“tmchojo” is the best FPA judge
44% Correct Guesses

13. Bots Judging Reliability
“BotTracker” is the best Bot telling apart
bots and humans (32%)

14. Humans & Bots Judging Reliability
H B H H B H B B B B

15. Humans & Bots Judging Reliability

16. Judge Reliability can be used to adjust
Humanness Scores
JRmeasures["Weight"] <- JRmeasures$JRR / nrow(JRmeasures)

17. Calculating FPA (First-Person Assessment)
Weighted First-Person Humanness Ratio
퐹푃퐴푖 =
푛 푊푒푖푔ℎ푡푗 ∗ 퐻푢푚푎푛푛푒푠푠푖,푗
푗=1
퐽
ℎ푢푚푎푛푛푒푠푠푖,푗 =
푀푖푠푠푖,푗
푁푖,푗
Judge j SDT Matrix Voted “Human” Voted “Bot”
Player is a Human Hit False Alarm
Player is a Bot Miss Hit
Sample proportion is an
unbiased estimator of p
in the population.
Humanness of player i
according to Judge j

18. Calculating FPA (First-Person Assessment)
Weighted First-Person Humanness Ratio
BotName Humanness FPA
MirrorBot 0.4996406 0.20164771
BotTracker 0.4231043 0.20070203
OvGUBot 0.3164826 0.10545765
NizorBot 0.2980527 0.11821633
ADANN 0.2432864 0.08351664
CCBot 0.1685606 0.06214746
BotName Humanness FPA
Player 0.5417464 0.1932813
tmchojo 0.5177169 0.1775752
Xenija 0.3847691 0.1713976
Juan_CVC 0.3172348 0.1237229

19. Calculating FPA (First-Person Assessment)
Weighted First-Person Humanness Ratio

20. Calculating FPA (First-Person Assessment)
Weighted First-Person Humanness Ratio
H H B B H H B B B B

21. Calculating FPA (First-Person Assessment)
Weighted First-Person Humanness Ratio
B B H H H H B B B B

22. BotPrize 2014 Edition: We add TPA
* TPA – Third-Person Assessment
Generation of Anonymized TPA Video
Clips featuring human and bot players
FPA Human Judges
Artificial Bots UT 2004 Server
TPA Judges
(Crowdsourcing
platform)
Third-Person
Crowdsourcing
Judging

23. Calculating TPA (Third-Person Assessment)
Crowdsourcing Judging
푇푃퐴푖,푗 =
푀푖푠푠푖,푗
푁푖,푗
J = 232 human judges
I = 12 characters (6+6)

24. Calculating TPA (Third-Person Assessment)
Crowdsourcing Judging
BotName FPA TPA H++
Xenija 0.17139763 0.8235294 0.4974635
MirrorBot 0.20164771 0.7333333 0.4674905
Player 0.19328127 0.6315789 0.4124301
tmchojo 0.17757519 0.6470588 0.4123170
NizorBot 0.11821633 0.7058824 0.4120493
BotTracker 0.20070203 0.5909091 0.3958056
CCBot 0.06214746 0.7058824 0.3840149
Juan_CVC 0.12372294 0.6190476 0.3713853
OvGUBot 0.10545765 0.6086957 0.3570767
ADANN 0.08351664 0.4761905 0.2798536

25. Calculating TPA (Third-Person Assessment)
Crowdsourcing Judging
H B B B H H H H H B B B

26. Final Results (FPA + TPA)
H B H H B B B H B B

27. Final Results (H++)
MirrorBot
NizorBot
BotTracker
OvGUBot
0.467
0.412
0.395
0.357
Mihai Polceanu
José L. Jiménez López
Antonio J. Fernández-Leiva
Antonio M. Mora
Hunjoo Lee
Jee-Hyong Lee
Xenija Neufeld
Sanaz Mostaghim

28. Congratulations for your results!!!
Hope to see you again
next year 
www.botprize.org
human-machine.unizar.es
Raul.Arrabales
@Conscious-Robots.com
@ConsciousRobots