Successfully reported this slideshow.
Your SlideShare is downloading. ×

The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: Annotation, Guidelines, and Student Research Projects

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 15 Ad

The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: Annotation, Guidelines, and Student Research Projects

Download to read offline

Slides of the talk at the 14th annual International Conference of Education, Research and Innovation, ICERI 2021, November 8th-9th 2021 (online).

Slides of the talk at the 14th annual International Conference of Education, Research and Innovation, ICERI 2021, November 8th-9th 2021 (online).

Advertisement
Advertisement

More Related Content

Slideshows for you (18)

Similar to The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: Annotation, Guidelines, and Student Research Projects (20)

Advertisement

More from Dagmar Monett (20)

Recently uploaded (20)

Advertisement

The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: Annotation, Guidelines, and Student Research Projects

  1. 1. The Intelligence Corpus, an annotated corpus of definitions of intelligence Annotation, guidelines, and student research projects @dmonett Dagmar Monett1, Luisa Hoge2, Laura Haase3, Lena Schwarz4, Marc Normann5, Linus Scheibe6 1Berlin School of Economics and Law 2Robert Koch Institute 3Technical University of Applied Sciences Wildau 4Hochschule Stralsund 5NORDAKADEMIE Graduate School 6DB Systel GmbH 14th annual International Conference of Education, Research and Innovation 8th - 9th of November, 2021
  2. 2. How to cite this work Monett, D., Hoge, L., Haase, L., Schwarz, L., Normann, M., & Scheibe, L. (2021). The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: Annotation, Guidelines, and Student Research Projects. In Proceedings of the 14th annual International Conference of Education, Research and Innovation, ICERI 2021, November 8th-9th 2021 [online]. Available at: https://www.slideshare.net/dmonett/monett-etal-2021-iceri (Accessed: access date). 2
  3. 3. Annotating definitions of intelligence - Fact: Lack of consensus on defining intelligence - Needed: To provide better insights into definitions and how to define them - Why: It is central to the basics of AI literacy - How: In this work, by evaluating the properties of available definitions 3
  4. 4. Annotation and Annotators Automatic data annotation still biased, error prone, and far from being entirely satisfactory. Domain knowledge The annotation might require special insights into the problem domain. Software solutions Software solutions are available for supporting annotators in their work (Neves and Ševa, 2021), but not for all kinds of data and domains. Annotation The annotation of data either its nature can be a very challenging and time consuming process. Annotators Undergraduate 3rd year Computer Science students Annotators Parallel course on AI and student research projects M. Neves and J. Ševa, “An extensive review of tools for manual annotation of documents,” Briefings in Bioinformatics, vol. 22, no. 1, pp. 146–163, 2021. 4
  5. 5. The Annotation Data A: 213 new, suggested definitions of machine or artificial intelligence by participants to the survey on defining intelligence (Monett & Lewis, 2018) B: 125 new, suggested definitions of human intelligence by participants to the survey on defining intelligence (Monett & Lewis, 2018) C: 34 definitions of intelligence from the literature to agree upon in the initial edition of the survey on defining intelligence (Monett & Lewis, 2018) D: 71 definitions of intelligence from the collection presented in (Legg & Hutter, 2007) D. Monett and C.W.P. Lewis, “Getting clarity by defining Artificial Intelligence—A Survey,” in Philosophy and Theory of Artificial Intelligence (V.C. Müller, ed.), SAPERE vol. 44, pp. 212–214. Springer, Berlin, 2018. S. Legg and M. Hutter, “A Collection of Definitions of Intelligence,” in Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms (B. Goertzel and P. Wang, eds.), vol. 157, pp. 17–24. IOS Press, UK, 2007. 443 definitions of (machine) intelligence 5
  6. 6. Examples of Definitions From collection C “Intelligence measures an agent’s ability to achieve goals in a wide range of environments.” From collection D “[Intelligence is] the capacity to learn, reason, and understand.” From collection B “[Human intelligence is] the ability to use information to accomplish goals.” From collection A “Machine Intelligence is concerned with building systems that can adapt and learn in unstructured noisy domains.” 6
  7. 7. Example: „A good definition of intelligence is affirmative.“ for definitions as suggested in (Monett & Lewis, 2020) Quality criteria D. Monett and C.W.P. Lewis, “Definitional Foundations for Intelligent Systems, Part I: Quality Criteria for Definitions of Intelligence,” in Proceedings of The 10th Anniversary Conference of the Academic Conference Association (J. Vopava, V. Douda, R. Kratochvil, and M. Konecki, eds.), pp. 73–80, Prague, Czech Republic. MAC Prague Consulting Ltd., 2020. 7
  8. 8. Definitions and Annotators Distribution of definitions per groups of annotators 8
  9. 9. Annotation Guidelines … on a cell if the corresponding definition fulfils the quality criterion Write a 1 E.g. by column (i.e. one quality criterion at a time); by row (i.e. definition by definition) How to proceed … you spend annotating whenever possible Record the time … when no idea about how to evaluate a given quality criterion Set background color … grammatical errors you might find in the definitions. Do not fix Do not discuss with other; this could introduce some bias Annotate alone … among others from the literature! 9
  10. 10. Examples of Annotations Definition It is affirmative It is short It includes cognitive abilities or functions It defines the “what” It does not distinguish between human and machine intelligence “An intelligent machine must first of all be a machine with interests, otherwise there are no interests to be served by its intelligence. Human interests can be served by machine ‘competence,’ not by machine intelligence.” 1 “[Human intelligence is the] ability to achieve objectives in a variety of environments.” 1 1 1 “Intelligence is the computational part of the ability to achieve goals in the world.” 1 1 1 1 “[Intelligence is] the capacity to acquire and apply knowledge.” 1 1 1 1 1 10
  11. 11. Inter-Annotator Agreement (IAA) Avg. Cohen’s κ = 0.4 (per group per collection) J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, pp. 37–46, 1960. J.R. Landis and G.G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, pp. 159–174, 1977.  IAA between fair and moderate for all collections (i.e. Cohen’s κ ranging from 0.344 to 0.465, interpretation according to (Landis and Koch, 1977)).  The number of agreements among annotators was higher for the collection containing definitions of human intelligence (collection B).  The quality criteria with the highest IAA values were those simpler, more intuitive, and easier to understand.  The annotators were more agreeable when evaluating the fulfilment of the quality criteria for the following definition: “Intelligence is a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience.” 11
  12. 12. The “best” definition: Gottfredson’s L.S. Gottfredson, “Mainstream science on intelligence: An editorial with 52 signatories, history, and bibliography,” Intelligence, vol. 24, pp. 13–23, 1997. R.J. Haier, “The Neuroscience of Intelligence,” Cambridge University Press, New York, NY, 2017. 12
  13. 13. Usage: Examples Part of the Intelligence Corpus All definitions from (Monett and Lewis, 18)  71 definitions of machine or artificial intelligence (from a total of 213) from collection A.  42 definitions of human intelligence (from a total of 125) from collection B.  12 definitions of intelligence (from a total of 34) from collection C.  23 definitions of intelligence (from a total of 71) from collection D. https://bit.ly/AnnotatedDefsIntelligence https://goo.gl/KDPtKT The collection of definitions of intelligence used in the research survey “Defining (machine) Intelligence” D. Monett and C.W.P. Lewis, “Getting clarity by defining Artificial Intelligence—A Survey,” in Philosophy and Theory of Artificial Intelligence (V.C. Müller, ed.), SAPERE vol. 44, pp. 212–214. Springer, Berlin, 2018. 13
  14. 14. Conclusions Prior experience available in mentoring and supervising student research projects. Parallel course on AI delivered by the same instructor; ad hoc discussions in class. A peculiar annotation case study that evaluates whether definitions of human and machine intelligence satisfy desirable properties or quality criteria of good definitions. Information, materials, and tools carefully prepared and discussed in advance, also throughout the project’s execution. AI course AI Experience Intelligence Corpus Project management Training on the process of defining a good definition of any concept, which could be of interest to regulators or lawyers, for instance. E.g. detailed, manually-conducted quality control of all available annotations. Further work Other uses 14
  15. 15. Contact /monettdiaz @dmonett Prof. Dr. Dagmar Monett 15 http://monettdiaz.com dagmar.monett-diaz@hwr-berlin.de Prof. Dr. Computer Science (Artificial Intelligence, Software Engineering) Computer Science Dept. Co-Director M.Sc. Digital Transformation Berlin School of Economics and Law (HWR Berlin) Faculty of Cooperative Studies Template: www.allppt.com

×