Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

WBA Prize at Animal-AI Olympics


Published on

The WBA Prize at the Animal-AI Olympics 2019 and how to evaluate AGI (by Hiroshi Yamakawa)

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

WBA Prize at Animal-AI Olympics

  1. 1. 16:00-17:30 Hiroshi Yamakawa WBA prize at Animal AI Olympics 4 Oct. 2019“Animal-AI Olympics” Kick-off meetup
  2. 2. Whole Brain Architecture approach “Animal-AI Olympics” Kick-off meetup #2 ‘to create a human-like AGI by learning from the architecture of the entire brain’ To reach AGI, we start from mimicking high-level architectures of the brain, and gradually introduce details as needed 2
  3. 3. The WBA (Whole Brain Architecture) Prize The WBA Prize will be awarded to the most biologically plausible entry • US$4,000 grant (from WBAI) • Up to US$1,000 travel grant to speak at NeurIPS 2019 (from the organizer) “Animal-AI Olympics” Kick-off meetup #23
  4. 4. The WBA Prize: Application The Summary You must write your team ID and the date of application. Detail how your methodology with a particular focus on the ways in which it is biologically plausible (1,000 ~ 1,500 English words.) More specifically,: • Structural correspondence between your architecture and brain architecture, if any (a block diagram would help). • Mechanisms being used that are similar to those found in the brain, • If you have modeled your solution on certain animals (including humans). • Include details about how you think your approach can help us understand biological intelligence. Source Code We appreciate code readable for the reviewers “Animal-AI Olympics” Kick-off meetup #24
  5. 5. The WBA Prize: Review Process • An independent panel of experts will be convened for review. • Biological plausibility as well as generality and the simplicity of the code will be appreciated (see the GPS criteria). • Biological plausibility will be evaluated by your summary. Generality will be evaluated from your competition record. The code will be used to evaluate the simplicity and to validate the summary. “Animal-AI Olympics” Kick-off meetup #25
  6. 6. Functionally General Biologically Plausible Computationally Simple GPS criteria for evaluating brain-inspired AGIs “Animal-AI Olympics” Kick-off meetup #26 Can the system solve various tasks? Evaluate the number of successfully executed tasks out of the presented tasks. Is the mechanism similar to that of the brain? Evaluate whether the presented design or sample code is used. Points are added for additional biological plausibility. Is the mechanism composed of reusable functional modules? Evaluate that multiple tasks are accomplished by fewer processing units.
  7. 7. Judging Committee • Expert of Cognitive Architecture • Expert of Computational Neuroscience • Expert of Reinforcement learning • Etc. “Animal-AI Olympics” Kick-off meetup #27
  8. 8. Organizing Committee of AAO “Animal-AI Olympics” Kick-off meetup #28
  9. 9. 9 José Hernández-Orallo • Valencia Institute of Technology – -Information system – -Computational • 4 books and over 100 articles and papers on artificial intelligence, machine learning, data mining, cognitive science, and information systems • Algorithm Information Theory for Artificial Intelligence Test Development “Animal-AI Olympics” Kick-off meetup #2
  10. 10. Interview on YouTube “Animal-AI Olympics” Kick-off meetup #210 What is intelligence? In «The Measure of All Minds», José Hernández-Orallo reflects on how to measure human, animal and artificial intelligence. The book aims to create links between the evaluation of artificial intelligence, very little developed, and a centuries-old tradition of evaluation of human and animal intelligence. 出典:
  11. 11. Why general test is necessary? “Animal-AI Olympics” Kick-off meetup #211 Heterogeneity Homo sapiens Animal Kingdom Possible Organisms Machine Kingdom Universal Psychometrics Added: - Cognitively enhanced organisms - Biologically enhaced computers Anthroposen (Niocene) extinction-100 times the creatures of nature are extinct. On the other hand, there is a reverse explosion. Computers.
  12. 12. Extension of traditional approach ← Negative “Animal-AI Olympics” Kick-off meetup #212 Metrological psychology, comparative psychology, AI evaluation • A specific group is subject to evaluation – Unable to evaluate the ability of non-target groups • comparison of humans and animals • Anthropocentric – Estimates of other groups are only estimates (comparison between human and AI) • A powerful falsification tool
  13. 13. 13 Concept of Difficulty – Quantitative psychology: importance in item response theory – Logarithm of the number of calculation steps required to solve the problem • Levin’s Universal search – “Animal-AI Olympics” Kick-off meetup #2 Implications • Interpretation given in the framework of calculation theory • Measurement unit: Logarithm of the number of steps
  14. 14. Item Response Theory (IRT) “Animal-AI Olympics” Kick-off meetup #214 • θ: Human parameter • i: test item • b: difficulty • a : discrimination • c : pseudo-guessing IRT is a way to analyze responses to item (test or questionnaires) with the goal of improving measurement accuracy and reliability. Probability of correct answer for item j Figure 1: Example of 3PL IRF, with dotted lines overlaid to demonstrate parameters. (Wikipedia)
  15. 15. Common procedure to build a test “Animal-AI Olympics” Kick-off meetup #215 1. Choose a range of items 2.Analyse item difficulty 3. Designing test for appropriate difficulty 4. Defining distribution of difficulty 5. Appropriate interface for the agents 6. Draw agent response curve and drive indicator
  16. 16. “Animal-AI Olympics” Kick-off meetup #216 The WBA award requires an additional technical summary and code submission. However, it is a competition section that has spaces to appealing your team’s unique ideas. Please challenge by all means.