Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

BotPrize 2014 Results. Human-Like Bots Competition at IEEE CIG

1,055 views

Published on

Human-Like Bots Competition (BotPrize 2014) presented by Raul Arrabales at IEEE CIG - Computational Ingelligence and Artificial Intelligence in Games.

The BotPrize is a Turing Test for First-Person Shooter video game bots.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

BotPrize 2014 Results. Human-Like Bots Competition at IEEE CIG

  1. 1. CIG 2014 - Human-Like Bots Competition IEEE Computational Intelligence in Games. Dortmund, Germany. August, 2014 Organization: Manuel G. Bedia Juan Peralta Joan Marc Philip Hingston Raúl Arrabales
  2. 2. Organization / Acknowledgements
  3. 3. Players (humans and bots) PLAYER TYPE TEAM MEMBERS AFFILIATION COUNTRY BotTracker BOT TETRIIS Hunjoo Lee Jee-Hyong Lee ETRI, Sungkyunkwan University SOUTH KOREA MirrorBot BOT IHSEV Mihai Polceanu ENIB CERV Centre de Réalité Virtuelle FRANCE NizorBot BOT UMAG-BOT José L. Jiménez López Antonio J. Fernández-Leiva Antonio M. Mora Universidad de Málaga SPAIN OvGUBot BOT OvGUBot Xenija Neufeld Sanaz Mostaghim Otto von Guericke University, Magdeburg GERMANY ADANN BOT CVC Juan Peralta Donate Joan Marc Llargués A. CVC. UAB SPAIN CCBot BOT Conscious -Robots Jorge Muñoz Raúl Arrabales Comaware SPAIN Player HUMAN Judge - - - Tmchojo HUMAN Judge - - - Juan_CVC HUMAN Judge - - - Xenija HUMAN Judge - - -
  4. 4. Original BotPrize Testing Protocol (FPA – First-Person Assessment) Human Judges Artificial Bots UT 2004 Server Real-time Online Anonymized interaction
  5. 5. BotPrize 2014 Edition: We add TPA * FPA – First-Person Assessment * TPA – Third-Person Assessment Generation of Anonymized TPA Video Clips featuring human and bot players FPA Human Judges Artificial Bots UT 2004 Server TPA Judges (Crowdsourcing platform) Third-Person Crowdsourcing Judging
  6. 6. BotPrize 2014 Edition: Humanness++ (H) H = (FPA * FPWF) + (TPA * TPWF) FPWF First-Person Weighting Factor = 0,5. TPWF Third-Person Weighting Factor = 0,5. FPA Human Judges Artificial Bots UT 2004 Server H = FPA / 2 + TPA / 2
  7. 7. Humanness scores based on SDT Signal Detection Theory Judge SDT Matrix Vote “Human” Vote “Bot” Player is a Human Hit False Alarm Player is a Bot Miss Hit Tanner Jr., Wilson P.; John A. Swets (November 1954). "A decision-making theory of visual detection.". Psychological Review. 61 (6): 401–409.
  8. 8. Humanness scores based on SDT Judge Reliability (JR) A measure of how good a judge is in terms of telling apart humans and bots 퐽푅푗 = 퐻푖푡푠 − (푀푖푠푠푒푠 + 퐹푎푙푠푒퐴푙푎푟푚푠 푁푗 Judge SDT Matrix Vote “Human” Vote “Bot” Player is a Human Hit False Alarm Player is a Bot Miss Hit
  9. 9. Judge Reliability can be used to adjust Humanness Scores Judge Relative Reliability (JRR) A measure of how good a judge is in relation with other judges 퐽푅푅푗 = 퐽푅푗 퐽 퐽푅푗 푗=1 퐴푣푔퐽푅 = 퐴푣푔퐽푅 퐽
  10. 10. Judge Reliability can be used to adjust Humanness Scores Judge Relative Reliability (JRR) A measure of how good a judge is in relation with other judges judges JRmeasures JRR Weight Player 0.28967254 1.2574682 0.31436704 tmchojo 0.43801653 1.9014292 0.47535731 Juan_CVC 0.16129032 0.7001611 0.17504027 Xenija 0.03246753 0.1409415 0.03523538
  11. 11. Judge Reliability can be used to adjust Humanness Scores Judge Relative Reliability (JRR) “tmchojo” is the best FPA judge 44% Correct Guesses
  12. 12. Bots Judging Reliability “BotTracker” is the best Bot telling apart bots and humans (32%)
  13. 13. Humans & Bots Judging Reliability H B H H B H B B B B
  14. 14. Humans & Bots Judging Reliability
  15. 15. Judge Reliability can be used to adjust Humanness Scores JRmeasures["Weight"] <- JRmeasures$JRR / nrow(JRmeasures)
  16. 16. Calculating FPA (First-Person Assessment) Weighted First-Person Humanness Ratio 퐹푃퐴푖 = 푛 푊푒푖푔ℎ푡푗 ∗ 퐻푢푚푎푛푛푒푠푠푖,푗 푗=1 퐽 ℎ푢푚푎푛푛푒푠푠푖,푗 = 푀푖푠푠푖,푗 푁푖,푗 Judge j SDT Matrix Voted “Human” Voted “Bot” Player is a Human Hit False Alarm Player is a Bot Miss Hit Sample proportion is an unbiased estimator of p in the population. Humanness of player i according to Judge j
  17. 17. Calculating FPA (First-Person Assessment) Weighted First-Person Humanness Ratio BotName Humanness FPA MirrorBot 0.4996406 0.20164771 BotTracker 0.4231043 0.20070203 OvGUBot 0.3164826 0.10545765 NizorBot 0.2980527 0.11821633 ADANN 0.2432864 0.08351664 CCBot 0.1685606 0.06214746 BotName Humanness FPA Player 0.5417464 0.1932813 tmchojo 0.5177169 0.1775752 Xenija 0.3847691 0.1713976 Juan_CVC 0.3172348 0.1237229
  18. 18. Calculating FPA (First-Person Assessment) Weighted First-Person Humanness Ratio
  19. 19. Calculating FPA (First-Person Assessment) Weighted First-Person Humanness Ratio H H B B H H B B B B
  20. 20. Calculating FPA (First-Person Assessment) Weighted First-Person Humanness Ratio B B H H H H B B B B
  21. 21. BotPrize 2014 Edition: We add TPA * TPA – Third-Person Assessment Generation of Anonymized TPA Video Clips featuring human and bot players FPA Human Judges Artificial Bots UT 2004 Server TPA Judges (Crowdsourcing platform) Third-Person Crowdsourcing Judging
  22. 22. Calculating TPA (Third-Person Assessment) Crowdsourcing Judging 푇푃퐴푖,푗 = 푀푖푠푠푖,푗 푁푖,푗 J = 232 human judges I = 12 characters (6+6)
  23. 23. Calculating TPA (Third-Person Assessment) Crowdsourcing Judging BotName FPA TPA H++ Xenija 0.17139763 0.8235294 0.4974635 MirrorBot 0.20164771 0.7333333 0.4674905 Player 0.19328127 0.6315789 0.4124301 tmchojo 0.17757519 0.6470588 0.4123170 NizorBot 0.11821633 0.7058824 0.4120493 BotTracker 0.20070203 0.5909091 0.3958056 CCBot 0.06214746 0.7058824 0.3840149 Juan_CVC 0.12372294 0.6190476 0.3713853 OvGUBot 0.10545765 0.6086957 0.3570767 ADANN 0.08351664 0.4761905 0.2798536
  24. 24. Calculating TPA (Third-Person Assessment) Crowdsourcing Judging H B B B H H H H H B B B
  25. 25. Final Results (FPA + TPA) H B H H B B B H B B
  26. 26. Final Results (H++) MirrorBot NizorBot BotTracker OvGUBot 0.467 0.412 0.395 0.357 Mihai Polceanu José L. Jiménez López Antonio J. Fernández-Leiva Antonio M. Mora Hunjoo Lee Jee-Hyong Lee Xenija Neufeld Sanaz Mostaghim
  27. 27. Congratulations for your results!!! Hope to see you again next year  www.botprize.org human-machine.unizar.es Raul.Arrabales @Conscious-Robots.com @ConsciousRobots

×