Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
CIG 2014 - Human-Like Bots Competition 
IEEE Computational Intelligence in Games. Dortmund, Germany. August, 2014 
Organiz...
Organization / Acknowledgements
Players (humans and bots) 
PLAYER TYPE TEAM MEMBERS AFFILIATION COUNTRY 
BotTracker BOT TETRIIS 
Hunjoo Lee 
Jee-Hyong Lee...
Original BotPrize Testing Protocol 
(FPA – First-Person Assessment) 
Human Judges 
Artificial Bots UT 2004 Server 
Real-ti...
BotPrize 2014 Edition: We add TPA 
* FPA – First-Person Assessment 
* TPA – Third-Person Assessment 
Generation of Anonymi...
BotPrize 2014 Edition: Humanness++ (H) 
H = (FPA * FPWF) + (TPA * TPWF) 
FPWF First-Person Weighting Factor = 0,5. 
TPWF ...
Humanness scores based on SDT 
Signal Detection Theory 
Judge SDT Matrix Vote “Human” Vote “Bot” 
Player is a Human Hit Fa...
Humanness scores based on SDT 
Judge Reliability (JR) 
A measure of how good a judge is in terms of 
telling apart humans ...
Judge Reliability can be used to adjust 
Humanness Scores 
Judge Relative Reliability (JRR) 
A measure of how good a judge...
Judge Reliability can be used to adjust 
Humanness Scores 
Judge Relative Reliability (JRR) 
A measure of how good a judge...
Judge Reliability can be used to 
adjust Humanness Scores 
Judge Relative Reliability (JRR) 
“tmchojo” is the best FPA jud...
Bots Judging Reliability 
“BotTracker” is the best Bot telling apart 
bots and humans (32%)
Humans & Bots Judging Reliability 
H B H H B H B B B B
Humans & Bots Judging Reliability
Judge Reliability can be used to adjust 
Humanness Scores 
JRmeasures["Weight"] <- JRmeasures$JRR / nrow(JRmeasures)
Calculating FPA (First-Person Assessment) 
Weighted First-Person Humanness Ratio 
퐹푃퐴푖 = 
푛 푊푒푖푔ℎ푡푗 ∗ 퐻푢푚푎푛푛푒푠푠푖,푗 
푗=1 
퐽...
Calculating FPA (First-Person Assessment) 
Weighted First-Person Humanness Ratio 
BotName Humanness FPA 
MirrorBot 0.49964...
Calculating FPA (First-Person Assessment) 
Weighted First-Person Humanness Ratio
Calculating FPA (First-Person Assessment) 
Weighted First-Person Humanness Ratio 
H H B B H H B B B B
Calculating FPA (First-Person Assessment) 
Weighted First-Person Humanness Ratio 
B B H H H H B B B B
BotPrize 2014 Edition: We add TPA 
* TPA – Third-Person Assessment 
Generation of Anonymized TPA Video 
Clips featuring hu...
Calculating TPA (Third-Person Assessment) 
Crowdsourcing Judging 
푇푃퐴푖,푗 = 
푀푖푠푠푖,푗 
푁푖,푗 
J = 232 human judges 
I = 12 ch...
Calculating TPA (Third-Person Assessment) 
Crowdsourcing Judging 
BotName FPA TPA H++ 
Xenija 0.17139763 0.8235294 0.49746...
Calculating TPA (Third-Person Assessment) 
Crowdsourcing Judging 
H B B B H H H H H B B B
Final Results (FPA + TPA) 
H B H H B B B H B B
Final Results (H++) 
MirrorBot 
NizorBot 
BotTracker 
OvGUBot 
0.467 
0.412 
0.395 
0.357 
Mihai Polceanu 
José L. Jiménez...
Congratulations for your results!!! 
Hope to see you again 
next year  
www.botprize.org 
human-machine.unizar.es 
Raul.A...
BotPrize 2014 Results. Human-Like Bots Competition at IEEE CIG
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
Integrating BI - Data Warehouse and Big Data
Next
Upcoming SlideShare
Integrating BI - Data Warehouse and Big Data
Next
Download to read offline and view in fullscreen.

Share

BotPrize 2014 Results. Human-Like Bots Competition at IEEE CIG

Download to read offline

Human-Like Bots Competition (BotPrize 2014) presented by Raul Arrabales at IEEE CIG - Computational Ingelligence and Artificial Intelligence in Games.

The BotPrize is a Turing Test for First-Person Shooter video game bots.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

BotPrize 2014 Results. Human-Like Bots Competition at IEEE CIG

  1. 1. CIG 2014 - Human-Like Bots Competition IEEE Computational Intelligence in Games. Dortmund, Germany. August, 2014 Organization: Manuel G. Bedia Juan Peralta Joan Marc Philip Hingston Raúl Arrabales
  2. 2. Organization / Acknowledgements
  3. 3. Players (humans and bots) PLAYER TYPE TEAM MEMBERS AFFILIATION COUNTRY BotTracker BOT TETRIIS Hunjoo Lee Jee-Hyong Lee ETRI, Sungkyunkwan University SOUTH KOREA MirrorBot BOT IHSEV Mihai Polceanu ENIB CERV Centre de Réalité Virtuelle FRANCE NizorBot BOT UMAG-BOT José L. Jiménez López Antonio J. Fernández-Leiva Antonio M. Mora Universidad de Málaga SPAIN OvGUBot BOT OvGUBot Xenija Neufeld Sanaz Mostaghim Otto von Guericke University, Magdeburg GERMANY ADANN BOT CVC Juan Peralta Donate Joan Marc Llargués A. CVC. UAB SPAIN CCBot BOT Conscious -Robots Jorge Muñoz Raúl Arrabales Comaware SPAIN Player HUMAN Judge - - - Tmchojo HUMAN Judge - - - Juan_CVC HUMAN Judge - - - Xenija HUMAN Judge - - -
  4. 4. Original BotPrize Testing Protocol (FPA – First-Person Assessment) Human Judges Artificial Bots UT 2004 Server Real-time Online Anonymized interaction
  5. 5. BotPrize 2014 Edition: We add TPA * FPA – First-Person Assessment * TPA – Third-Person Assessment Generation of Anonymized TPA Video Clips featuring human and bot players FPA Human Judges Artificial Bots UT 2004 Server TPA Judges (Crowdsourcing platform) Third-Person Crowdsourcing Judging
  6. 6. BotPrize 2014 Edition: Humanness++ (H) H = (FPA * FPWF) + (TPA * TPWF) FPWF First-Person Weighting Factor = 0,5. TPWF Third-Person Weighting Factor = 0,5. FPA Human Judges Artificial Bots UT 2004 Server H = FPA / 2 + TPA / 2
  7. 7. Humanness scores based on SDT Signal Detection Theory Judge SDT Matrix Vote “Human” Vote “Bot” Player is a Human Hit False Alarm Player is a Bot Miss Hit Tanner Jr., Wilson P.; John A. Swets (November 1954). "A decision-making theory of visual detection.". Psychological Review. 61 (6): 401–409.
  8. 8. Humanness scores based on SDT Judge Reliability (JR) A measure of how good a judge is in terms of telling apart humans and bots 퐽푅푗 = 퐻푖푡푠 − (푀푖푠푠푒푠 + 퐹푎푙푠푒퐴푙푎푟푚푠 푁푗 Judge SDT Matrix Vote “Human” Vote “Bot” Player is a Human Hit False Alarm Player is a Bot Miss Hit
  9. 9. Judge Reliability can be used to adjust Humanness Scores Judge Relative Reliability (JRR) A measure of how good a judge is in relation with other judges 퐽푅푅푗 = 퐽푅푗 퐽 퐽푅푗 푗=1 퐴푣푔퐽푅 = 퐴푣푔퐽푅 퐽
  10. 10. Judge Reliability can be used to adjust Humanness Scores Judge Relative Reliability (JRR) A measure of how good a judge is in relation with other judges judges JRmeasures JRR Weight Player 0.28967254 1.2574682 0.31436704 tmchojo 0.43801653 1.9014292 0.47535731 Juan_CVC 0.16129032 0.7001611 0.17504027 Xenija 0.03246753 0.1409415 0.03523538
  11. 11. Judge Reliability can be used to adjust Humanness Scores Judge Relative Reliability (JRR) “tmchojo” is the best FPA judge 44% Correct Guesses
  12. 12. Bots Judging Reliability “BotTracker” is the best Bot telling apart bots and humans (32%)
  13. 13. Humans & Bots Judging Reliability H B H H B H B B B B
  14. 14. Humans & Bots Judging Reliability
  15. 15. Judge Reliability can be used to adjust Humanness Scores JRmeasures["Weight"] <- JRmeasures$JRR / nrow(JRmeasures)
  16. 16. Calculating FPA (First-Person Assessment) Weighted First-Person Humanness Ratio 퐹푃퐴푖 = 푛 푊푒푖푔ℎ푡푗 ∗ 퐻푢푚푎푛푛푒푠푠푖,푗 푗=1 퐽 ℎ푢푚푎푛푛푒푠푠푖,푗 = 푀푖푠푠푖,푗 푁푖,푗 Judge j SDT Matrix Voted “Human” Voted “Bot” Player is a Human Hit False Alarm Player is a Bot Miss Hit Sample proportion is an unbiased estimator of p in the population. Humanness of player i according to Judge j
  17. 17. Calculating FPA (First-Person Assessment) Weighted First-Person Humanness Ratio BotName Humanness FPA MirrorBot 0.4996406 0.20164771 BotTracker 0.4231043 0.20070203 OvGUBot 0.3164826 0.10545765 NizorBot 0.2980527 0.11821633 ADANN 0.2432864 0.08351664 CCBot 0.1685606 0.06214746 BotName Humanness FPA Player 0.5417464 0.1932813 tmchojo 0.5177169 0.1775752 Xenija 0.3847691 0.1713976 Juan_CVC 0.3172348 0.1237229
  18. 18. Calculating FPA (First-Person Assessment) Weighted First-Person Humanness Ratio
  19. 19. Calculating FPA (First-Person Assessment) Weighted First-Person Humanness Ratio H H B B H H B B B B
  20. 20. Calculating FPA (First-Person Assessment) Weighted First-Person Humanness Ratio B B H H H H B B B B
  21. 21. BotPrize 2014 Edition: We add TPA * TPA – Third-Person Assessment Generation of Anonymized TPA Video Clips featuring human and bot players FPA Human Judges Artificial Bots UT 2004 Server TPA Judges (Crowdsourcing platform) Third-Person Crowdsourcing Judging
  22. 22. Calculating TPA (Third-Person Assessment) Crowdsourcing Judging 푇푃퐴푖,푗 = 푀푖푠푠푖,푗 푁푖,푗 J = 232 human judges I = 12 characters (6+6)
  23. 23. Calculating TPA (Third-Person Assessment) Crowdsourcing Judging BotName FPA TPA H++ Xenija 0.17139763 0.8235294 0.4974635 MirrorBot 0.20164771 0.7333333 0.4674905 Player 0.19328127 0.6315789 0.4124301 tmchojo 0.17757519 0.6470588 0.4123170 NizorBot 0.11821633 0.7058824 0.4120493 BotTracker 0.20070203 0.5909091 0.3958056 CCBot 0.06214746 0.7058824 0.3840149 Juan_CVC 0.12372294 0.6190476 0.3713853 OvGUBot 0.10545765 0.6086957 0.3570767 ADANN 0.08351664 0.4761905 0.2798536
  24. 24. Calculating TPA (Third-Person Assessment) Crowdsourcing Judging H B B B H H H H H B B B
  25. 25. Final Results (FPA + TPA) H B H H B B B H B B
  26. 26. Final Results (H++) MirrorBot NizorBot BotTracker OvGUBot 0.467 0.412 0.395 0.357 Mihai Polceanu José L. Jiménez López Antonio J. Fernández-Leiva Antonio M. Mora Hunjoo Lee Jee-Hyong Lee Xenija Neufeld Sanaz Mostaghim
  27. 27. Congratulations for your results!!! Hope to see you again next year  www.botprize.org human-machine.unizar.es Raul.Arrabales @Conscious-Robots.com @ConsciousRobots

Human-Like Bots Competition (BotPrize 2014) presented by Raul Arrabales at IEEE CIG - Computational Ingelligence and Artificial Intelligence in Games. The BotPrize is a Turing Test for First-Person Shooter video game bots.

Views

Total views

1,660

On Slideshare

0

From embeds

0

Number of embeds

12

Actions

Downloads

17

Shares

0

Comments

0

Likes

0

×