SlideShare a Scribd company logo
The Intelligence Corpus,
an annotated corpus of
definitions of intelligence
Annotation, guidelines, and student research projects
@dmonett
Dagmar Monett1, Luisa Hoge2, Laura Haase3, Lena Schwarz4, Marc Normann5, Linus Scheibe6
1Berlin School of Economics and Law
2Robert Koch Institute
3Technical University of Applied Sciences Wildau
4Hochschule Stralsund
5NORDAKADEMIE Graduate School
6DB Systel GmbH
14th annual International Conference of Education, Research and Innovation
8th - 9th of November, 2021
How to cite this work
Monett, D., Hoge, L., Haase, L., Schwarz, L., Normann, M., & Scheibe, L. (2021).
The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence:
Annotation, Guidelines, and Student Research Projects.
In Proceedings of the 14th annual International Conference of Education, Research
and Innovation, ICERI 2021, November 8th-9th 2021 [online].
Available at: https://www.slideshare.net/dmonett/monett-etal-2021-iceri
(Accessed: access date).
2
Annotating definitions
of intelligence
- Fact: Lack of consensus on defining
intelligence
- Needed: To provide better insights into
definitions and how to define them
- Why: It is central to the basics of AI
literacy
- How: In this work, by evaluating the
properties of available definitions
3
Annotation and Annotators
Automatic data annotation
still biased, error prone, and
far from being entirely
satisfactory.
Domain knowledge
The annotation might require
special insights into the
problem domain.
Software solutions
Software solutions are available for
supporting annotators in their work
(Neves and Ševa, 2021), but not for
all kinds of data and domains.
Annotation
The annotation of data either
its nature can be a very
challenging and time
consuming process.
Annotators
Undergraduate 3rd year
Computer Science students
Annotators
Parallel course on AI and
student research projects
M. Neves and J. Ševa, “An extensive review of tools for manual annotation of documents,” Briefings in Bioinformatics, vol. 22, no. 1, pp. 146–163, 2021.
4
The Annotation Data
A: 213 new, suggested definitions of machine or
artificial intelligence by participants to the survey
on defining intelligence (Monett & Lewis, 2018)
B: 125 new, suggested definitions of human
intelligence by participants to the survey on
defining intelligence (Monett & Lewis, 2018)
C: 34 definitions of intelligence from the literature to
agree upon in the initial edition of the survey on
defining intelligence (Monett & Lewis, 2018)
D: 71 definitions of intelligence from the collection
presented in (Legg & Hutter, 2007)
D. Monett and C.W.P. Lewis, “Getting clarity by defining Artificial Intelligence—A Survey,” in Philosophy and Theory of Artificial Intelligence (V.C. Müller, ed.),
SAPERE vol. 44, pp. 212–214. Springer, Berlin, 2018.
S. Legg and M. Hutter, “A Collection of Definitions of Intelligence,” in Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms (B. Goertzel
and P. Wang, eds.), vol. 157, pp. 17–24. IOS Press, UK, 2007.
443 definitions of (machine) intelligence
5
Examples of Definitions
From collection C
“Intelligence measures an agent’s ability to
achieve goals in a wide range of
environments.”
From collection D
“[Intelligence is] the capacity to learn,
reason, and understand.”
From collection B
“[Human intelligence is] the
ability to use information to
accomplish goals.”
From collection A
“Machine Intelligence is concerned
with building systems that can adapt
and learn in unstructured noisy
domains.”
6
Example:
„A good
definition of
intelligence
is
affirmative.“
for definitions as suggested in
(Monett & Lewis, 2020)
Quality criteria
D. Monett and C.W.P. Lewis, “Definitional Foundations for Intelligent Systems,
Part I: Quality Criteria for Definitions of Intelligence,” in Proceedings of The 10th
Anniversary Conference of the Academic Conference Association (J. Vopava, V.
Douda, R. Kratochvil, and M. Konecki, eds.), pp. 73–80, Prague, Czech
Republic. MAC Prague Consulting Ltd., 2020.
7
Definitions and Annotators
Distribution of definitions per groups of annotators
8
Annotation Guidelines
… on a cell if the
corresponding
definition fulfils the
quality criterion
Write a 1
E.g. by column (i.e.
one quality criterion
at a time); by row (i.e.
definition by definition)
How to proceed
… you spend
annotating
whenever possible
Record the time
… when no idea
about how to
evaluate a given
quality criterion
Set background color
… grammatical
errors you might
find in the
definitions.
Do not fix
Do not discuss with
other; this could
introduce some bias
Annotate alone
… among others from the literature!
9
Examples of Annotations
Definition It is affirmative It is short
It includes
cognitive
abilities or
functions
It defines the
“what”
It does not
distinguish
between
human and
machine
intelligence
“An intelligent machine must first
of all be a machine with interests,
otherwise there are no interests
to be served by its intelligence.
Human interests can be served
by machine ‘competence,’ not by
machine intelligence.”
1
“[Human intelligence is the]
ability to achieve objectives in a
variety of environments.”
1 1 1
“Intelligence is the computational
part of the ability to achieve
goals in the world.”
1 1 1 1
“[Intelligence is] the capacity to
acquire and apply knowledge.”
1 1 1 1 1
10
Inter-Annotator Agreement (IAA)
Avg. Cohen’s κ = 0.4
(per group per
collection)
J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, pp. 37–46, 1960.
J.R. Landis and G.G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, pp. 159–174, 1977.
 IAA between fair and moderate for all collections (i.e.
Cohen’s κ ranging from 0.344 to 0.465, interpretation
according to (Landis and Koch, 1977)).
 The number of agreements among annotators was higher for
the collection containing definitions of human intelligence
(collection B).
 The quality criteria with the highest IAA values were those
simpler, more intuitive, and easier to understand.
 The annotators were more agreeable when evaluating the
fulfilment of the quality criteria for the following definition:
“Intelligence is a very general mental capability that, among
other things, involves the ability to reason, plan, solve
problems, think abstractly, comprehend complex ideas, learn
quickly and learn from experience.”
11
The “best” definition: Gottfredson’s
L.S. Gottfredson, “Mainstream science on intelligence: An editorial with 52 signatories, history, and bibliography,”
Intelligence, vol. 24, pp. 13–23, 1997.
R.J. Haier, “The Neuroscience of Intelligence,” Cambridge University Press, New York, NY, 2017.
12
Usage: Examples
Part of the
Intelligence Corpus
All definitions from
(Monett and Lewis, 18)
 71 definitions of machine or
artificial intelligence (from a
total of 213) from collection A.
 42 definitions of human
intelligence (from a total of
125) from collection B.
 12 definitions of intelligence
(from a total of 34) from
collection C.
 23 definitions of intelligence
(from a total of 71) from
collection D.
https://bit.ly/AnnotatedDefsIntelligence
https://goo.gl/KDPtKT
The collection of definitions of
intelligence used in the research
survey “Defining (machine)
Intelligence”
D. Monett and C.W.P. Lewis, “Getting clarity by defining Artificial Intelligence—A Survey,” in Philosophy and Theory of Artificial Intelligence (V.C. Müller, ed.),
SAPERE vol. 44, pp. 212–214. Springer, Berlin, 2018.
13
Conclusions
Prior experience available in mentoring and
supervising student research projects.
Parallel course on AI delivered by the same
instructor; ad hoc discussions in class.
A peculiar annotation case study that evaluates
whether definitions of human and machine
intelligence satisfy desirable properties or quality
criteria of good definitions.
Information, materials, and tools carefully
prepared and discussed in advance, also
throughout the project’s execution.
AI course
AI
Experience
Intelligence Corpus Project management
Training on the process of defining a good
definition of any concept, which could be of
interest to regulators or lawyers, for instance.
E.g. detailed, manually-conducted quality control
of all available annotations.
Further work
Other uses
14
Contact
/monettdiaz
@dmonett
Prof. Dr.
Dagmar Monett
15
http://monettdiaz.com
dagmar.monett-diaz@hwr-berlin.de
Prof. Dr. Computer Science (Artificial Intelligence, Software
Engineering)
Computer Science Dept.
Co-Director M.Sc. Digital Transformation
Berlin School of Economics and Law (HWR Berlin)
Faculty of Cooperative Studies
Template: www.allppt.com

More Related Content

What's hot

History of AI, Current Trends, Prospective Trajectories
History of AI, Current Trends, Prospective TrajectoriesHistory of AI, Current Trends, Prospective Trajectories
History of AI, Current Trends, Prospective Trajectories
Giovanni Sileno
 
Artificial Intelligence power point presentation document
Artificial Intelligence power point presentation documentArtificial Intelligence power point presentation document
Artificial Intelligence power point presentation document
David Raj Kanthi
 
Artificial Intelligence Report
Artificial Intelligence Report Artificial Intelligence Report
Artificial Intelligence Report
Shubham Verma
 
Artificial Intelligence AI Topics History and Overview
Artificial Intelligence AI Topics History and OverviewArtificial Intelligence AI Topics History and Overview
Artificial Intelligence AI Topics History and Overview
butest
 
Narrative: Text Generation Model from Data
Narrative: Text Generation Model from DataNarrative: Text Generation Model from Data

“Towards Machine Intelligence” - Daniyal Shahrokhian

“Towards Machine Intelligence” - Daniyal Shahrokhian
“Towards Machine Intelligence” - Daniyal Shahrokhian

“Towards Machine Intelligence” - Daniyal Shahrokhian
EIT Digital Alumni
 
Machine learning and language v2
Machine learning  and language v2Machine learning  and language v2
Machine learning and language v2
William Moore
 
Cognitive Computing : IBM Watson
Cognitive Computing : IBM WatsonCognitive Computing : IBM Watson
Cognitive Computing : IBM Watson
Amit Ranjan
 
AI 3.0
AI 3.0AI 3.0
AI 3.0
InnoTech
 
Artificial Intelligence Future |Impact Of Artificial Intelligence On Society
Artificial Intelligence Future |Impact Of Artificial Intelligence On SocietyArtificial Intelligence Future |Impact Of Artificial Intelligence On Society
Artificial Intelligence Future |Impact Of Artificial Intelligence On Society
YashShah445
 
Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...
Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...
Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...
Melanie Swan
 
ARTIFICIAL INTELLIGENT ( ITS / TASK 6 ) done by Wael Saad Hameedi / P71062
ARTIFICIAL INTELLIGENT ( ITS / TASK 6 ) done by Wael Saad Hameedi / P71062ARTIFICIAL INTELLIGENT ( ITS / TASK 6 ) done by Wael Saad Hameedi / P71062
ARTIFICIAL INTELLIGENT ( ITS / TASK 6 ) done by Wael Saad Hameedi / P71062
Wael Alawsey
 
AI for Everyone: Master the Basics
AI for Everyone: Master the BasicsAI for Everyone: Master the Basics
AI for Everyone: Master the Basics
Stutty Srivastava
 
Innovation report: Artificial Intelligence
Innovation report: Artificial IntelligenceInnovation report: Artificial Intelligence
Innovation report: Artificial Intelligence
Youssef Rahoui
 
Notes of AI for everyone - by Andrew Ng
Notes of AI for everyone - by Andrew NgNotes of AI for everyone - by Andrew Ng
Notes of AI for everyone - by Andrew Ng
mgopalani
 
FOR A MEANINGFUL ARTIFICIAL INTELLIGENCE TOWARDS A FRENCH AND EUROPEAN ST...
FOR A  MEANINGFUL  ARTIFICIAL  INTELLIGENCE TOWARDS A FRENCH  AND EUROPEAN ST...FOR A  MEANINGFUL  ARTIFICIAL  INTELLIGENCE TOWARDS A FRENCH  AND EUROPEAN ST...
FOR A MEANINGFUL ARTIFICIAL INTELLIGENCE TOWARDS A FRENCH AND EUROPEAN ST...
Willy Marroquin (WillyDevNET)
 
AI Lecture 1 (introduction)
AI Lecture 1 (introduction)AI Lecture 1 (introduction)
AI Lecture 1 (introduction)
Tajim Md. Niamat Ullah Akhund
 
Artificial intelligence uses in productive systems and impacts on the world...
Artificial intelligence   uses in productive systems and impacts on the world...Artificial intelligence   uses in productive systems and impacts on the world...
Artificial intelligence uses in productive systems and impacts on the world...
Fernando Alcoforado
 

What's hot (18)

History of AI, Current Trends, Prospective Trajectories
History of AI, Current Trends, Prospective TrajectoriesHistory of AI, Current Trends, Prospective Trajectories
History of AI, Current Trends, Prospective Trajectories
 
Artificial Intelligence power point presentation document
Artificial Intelligence power point presentation documentArtificial Intelligence power point presentation document
Artificial Intelligence power point presentation document
 
Artificial Intelligence Report
Artificial Intelligence Report Artificial Intelligence Report
Artificial Intelligence Report
 
Artificial Intelligence AI Topics History and Overview
Artificial Intelligence AI Topics History and OverviewArtificial Intelligence AI Topics History and Overview
Artificial Intelligence AI Topics History and Overview
 
Narrative: Text Generation Model from Data
Narrative: Text Generation Model from DataNarrative: Text Generation Model from Data
Narrative: Text Generation Model from Data
 

“Towards Machine Intelligence” - Daniyal Shahrokhian

“Towards Machine Intelligence” - Daniyal Shahrokhian
“Towards Machine Intelligence” - Daniyal Shahrokhian

“Towards Machine Intelligence” - Daniyal Shahrokhian
 
Machine learning and language v2
Machine learning  and language v2Machine learning  and language v2
Machine learning and language v2
 
Cognitive Computing : IBM Watson
Cognitive Computing : IBM WatsonCognitive Computing : IBM Watson
Cognitive Computing : IBM Watson
 
AI 3.0
AI 3.0AI 3.0
AI 3.0
 
Artificial Intelligence Future |Impact Of Artificial Intelligence On Society
Artificial Intelligence Future |Impact Of Artificial Intelligence On SocietyArtificial Intelligence Future |Impact Of Artificial Intelligence On Society
Artificial Intelligence Future |Impact Of Artificial Intelligence On Society
 
Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...
Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...
Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...
 
ARTIFICIAL INTELLIGENT ( ITS / TASK 6 ) done by Wael Saad Hameedi / P71062
ARTIFICIAL INTELLIGENT ( ITS / TASK 6 ) done by Wael Saad Hameedi / P71062ARTIFICIAL INTELLIGENT ( ITS / TASK 6 ) done by Wael Saad Hameedi / P71062
ARTIFICIAL INTELLIGENT ( ITS / TASK 6 ) done by Wael Saad Hameedi / P71062
 
AI for Everyone: Master the Basics
AI for Everyone: Master the BasicsAI for Everyone: Master the Basics
AI for Everyone: Master the Basics
 
Innovation report: Artificial Intelligence
Innovation report: Artificial IntelligenceInnovation report: Artificial Intelligence
Innovation report: Artificial Intelligence
 
Notes of AI for everyone - by Andrew Ng
Notes of AI for everyone - by Andrew NgNotes of AI for everyone - by Andrew Ng
Notes of AI for everyone - by Andrew Ng
 
FOR A MEANINGFUL ARTIFICIAL INTELLIGENCE TOWARDS A FRENCH AND EUROPEAN ST...
FOR A  MEANINGFUL  ARTIFICIAL  INTELLIGENCE TOWARDS A FRENCH  AND EUROPEAN ST...FOR A  MEANINGFUL  ARTIFICIAL  INTELLIGENCE TOWARDS A FRENCH  AND EUROPEAN ST...
FOR A MEANINGFUL ARTIFICIAL INTELLIGENCE TOWARDS A FRENCH AND EUROPEAN ST...
 
AI Lecture 1 (introduction)
AI Lecture 1 (introduction)AI Lecture 1 (introduction)
AI Lecture 1 (introduction)
 
Artificial intelligence uses in productive systems and impacts on the world...
Artificial intelligence   uses in productive systems and impacts on the world...Artificial intelligence   uses in productive systems and impacts on the world...
Artificial intelligence uses in productive systems and impacts on the world...
 

Similar to The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: Annotation, Guidelines, and Student Research Projects

Analytical-frameworks - Methods in user-technology studies
Analytical-frameworks - Methods in user-technology studiesAnalytical-frameworks - Methods in user-technology studies
Analytical-frameworks - Methods in user-technology studies
Antti Salovaara
 
Multimodal Tutor - Adaptive feedback from multimodal experience capturing
Multimodal Tutor - Adaptive feedback from multimodal experience capturingMultimodal Tutor - Adaptive feedback from multimodal experience capturing
Multimodal Tutor - Adaptive feedback from multimodal experience capturing
Daniele Di Mitri
 
Lect#1 (Artificial Intelligence )
Lect#1 (Artificial Intelligence )Lect#1 (Artificial Intelligence )
Lect#1 (Artificial Intelligence )
Zeeshan_Jadoon
 
The era of artificial intelligence
The era of artificial intelligenceThe era of artificial intelligence
The era of artificial intelligence
Prajjwal Kushwaha
 
Tools for a whole range of Scholarly Activities (at DH2015)
Tools for a whole range of Scholarly Activities (at DH2015)Tools for a whole range of Scholarly Activities (at DH2015)
Tools for a whole range of Scholarly Activities (at DH2015)
John Bradley
 
Conflicts, Bounded Rationality and Collective Wisdom. Lecce 2014
Conflicts, Bounded Rationality and Collective Wisdom. Lecce 2014Conflicts, Bounded Rationality and Collective Wisdom. Lecce 2014
Conflicts, Bounded Rationality and Collective Wisdom. Lecce 2014
Jose Francisco Álvarez Álvarez
 
BCII 2016 - Visualizing Complexity
BCII 2016 - Visualizing ComplexityBCII 2016 - Visualizing Complexity
BCII 2016 - Visualizing Complexity
Simon Buckingham Shum
 
Introduction to AI
Introduction to AIIntroduction to AI
Introduction to AI
Dymytr Yovchev
 
Making our mark: the important role of social scientists in the ‘era of big d...
Making our mark: the important role of social scientists in the ‘era of big d...Making our mark: the important role of social scientists in the ‘era of big d...
Making our mark: the important role of social scientists in the ‘era of big d...
The Higher Education Academy
 
Pliny: 4 perspectives
Pliny: 4 perspectivesPliny: 4 perspectives
Pliny: 4 perspectives
John Bradley
 
Ai notes
Ai notesAi notes
Ai notes
AbdullahGubbi1
 
Caps2015alvarezuned
Caps2015alvarezunedCaps2015alvarezuned
Gardner's Theory of Multiple Intelligences
Gardner's Theory of Multiple IntelligencesGardner's Theory of Multiple Intelligences
Gardner's Theory of Multiple Intelligences
LOVERJOHNAGASPAR
 
Design Science in Information Systems
Design Science in Information SystemsDesign Science in Information Systems
Design Science in Information Systems
Sergej Lugovic
 
A Knowledge Concept Map Structured Concept Analysis From Systematic Literatu...
A Knowledge Concept Map  Structured Concept Analysis From Systematic Literatu...A Knowledge Concept Map  Structured Concept Analysis From Systematic Literatu...
A Knowledge Concept Map Structured Concept Analysis From Systematic Literatu...
Dustin Pytko
 
4th_paradigm_book_complete_lr
4th_paradigm_book_complete_lr4th_paradigm_book_complete_lr
4th_paradigm_book_complete_lr
Dominic A Ienco
 
Toward a science of group cognition in CSCL
Toward a science of group cognition in CSCLToward a science of group cognition in CSCL
Toward a science of group cognition in CSCL
Gerry Stahl
 
How Cognitive Science Has Influenced the Applied Science of HCI “The evolutio...
How Cognitive Science Has Influenced the Applied Science of HCI “The evolutio...How Cognitive Science Has Influenced the Applied Science of HCI “The evolutio...
How Cognitive Science Has Influenced the Applied Science of HCI “The evolutio...
IOSR Journals
 
IS
ISIS
Jacob Sherson (University of Aarhus) - Science@home
Jacob Sherson (University of Aarhus) - Science@homeJacob Sherson (University of Aarhus) - Science@home
Jacob Sherson (University of Aarhus) - Science@home
CitizenCyberlab
 

Similar to The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: Annotation, Guidelines, and Student Research Projects (20)

Analytical-frameworks - Methods in user-technology studies
Analytical-frameworks - Methods in user-technology studiesAnalytical-frameworks - Methods in user-technology studies
Analytical-frameworks - Methods in user-technology studies
 
Multimodal Tutor - Adaptive feedback from multimodal experience capturing
Multimodal Tutor - Adaptive feedback from multimodal experience capturingMultimodal Tutor - Adaptive feedback from multimodal experience capturing
Multimodal Tutor - Adaptive feedback from multimodal experience capturing
 
Lect#1 (Artificial Intelligence )
Lect#1 (Artificial Intelligence )Lect#1 (Artificial Intelligence )
Lect#1 (Artificial Intelligence )
 
The era of artificial intelligence
The era of artificial intelligenceThe era of artificial intelligence
The era of artificial intelligence
 
Tools for a whole range of Scholarly Activities (at DH2015)
Tools for a whole range of Scholarly Activities (at DH2015)Tools for a whole range of Scholarly Activities (at DH2015)
Tools for a whole range of Scholarly Activities (at DH2015)
 
Conflicts, Bounded Rationality and Collective Wisdom. Lecce 2014
Conflicts, Bounded Rationality and Collective Wisdom. Lecce 2014Conflicts, Bounded Rationality and Collective Wisdom. Lecce 2014
Conflicts, Bounded Rationality and Collective Wisdom. Lecce 2014
 
BCII 2016 - Visualizing Complexity
BCII 2016 - Visualizing ComplexityBCII 2016 - Visualizing Complexity
BCII 2016 - Visualizing Complexity
 
Introduction to AI
Introduction to AIIntroduction to AI
Introduction to AI
 
Making our mark: the important role of social scientists in the ‘era of big d...
Making our mark: the important role of social scientists in the ‘era of big d...Making our mark: the important role of social scientists in the ‘era of big d...
Making our mark: the important role of social scientists in the ‘era of big d...
 
Pliny: 4 perspectives
Pliny: 4 perspectivesPliny: 4 perspectives
Pliny: 4 perspectives
 
Ai notes
Ai notesAi notes
Ai notes
 
Caps2015alvarezuned
Caps2015alvarezunedCaps2015alvarezuned
Caps2015alvarezuned
 
Gardner's Theory of Multiple Intelligences
Gardner's Theory of Multiple IntelligencesGardner's Theory of Multiple Intelligences
Gardner's Theory of Multiple Intelligences
 
Design Science in Information Systems
Design Science in Information SystemsDesign Science in Information Systems
Design Science in Information Systems
 
A Knowledge Concept Map Structured Concept Analysis From Systematic Literatu...
A Knowledge Concept Map  Structured Concept Analysis From Systematic Literatu...A Knowledge Concept Map  Structured Concept Analysis From Systematic Literatu...
A Knowledge Concept Map Structured Concept Analysis From Systematic Literatu...
 
4th_paradigm_book_complete_lr
4th_paradigm_book_complete_lr4th_paradigm_book_complete_lr
4th_paradigm_book_complete_lr
 
Toward a science of group cognition in CSCL
Toward a science of group cognition in CSCLToward a science of group cognition in CSCL
Toward a science of group cognition in CSCL
 
How Cognitive Science Has Influenced the Applied Science of HCI “The evolutio...
How Cognitive Science Has Influenced the Applied Science of HCI “The evolutio...How Cognitive Science Has Influenced the Applied Science of HCI “The evolutio...
How Cognitive Science Has Influenced the Applied Science of HCI “The evolutio...
 
IS
ISIS
IS
 
Jacob Sherson (University of Aarhus) - Science@home
Jacob Sherson (University of Aarhus) - Science@homeJacob Sherson (University of Aarhus) - Science@home
Jacob Sherson (University of Aarhus) - Science@home
 

More from Dagmar Monett

Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...
Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...
Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...
Dagmar Monett
 
Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...
Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...
Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...
Dagmar Monett
 
University-Industry Collaboration's Next Level: A Comparative Study as Basis ...
University-Industry Collaboration's Next Level: A Comparative Study as Basis ...University-Industry Collaboration's Next Level: A Comparative Study as Basis ...
University-Industry Collaboration's Next Level: A Comparative Study as Basis ...
Dagmar Monett
 
The Changing Landscape of Digital Technologies for Learning
The Changing Landscape of Digital Technologies for Learning The Changing Landscape of Digital Technologies for Learning
The Changing Landscape of Digital Technologies for Learning
Dagmar Monett
 
Will Robots Take all the Jobs? Not yet.
Will Robots Take all the Jobs? Not yet.Will Robots Take all the Jobs? Not yet.
Will Robots Take all the Jobs? Not yet.
Dagmar Monett
 
Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...
Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...
Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...
Dagmar Monett
 
Simulating the Fractional Reserve Banking using Agent-based Modelling with Ne...
Simulating the Fractional Reserve Banking using Agent-based Modelling with Ne...Simulating the Fractional Reserve Banking using Agent-based Modelling with Ne...
Simulating the Fractional Reserve Banking using Agent-based Modelling with Ne...
Dagmar Monett
 
Predicting Star Ratings based on Annotated Reviewss of Mobile Apps [Slides]
Predicting Star Ratings based on Annotated Reviewss of Mobile Apps [Slides]Predicting Star Ratings based on Annotated Reviewss of Mobile Apps [Slides]
Predicting Star Ratings based on Annotated Reviewss of Mobile Apps [Slides]
Dagmar Monett
 
Teaching Students Collaborative Requirements Engineering. Case Study Red:Wire
Teaching Students Collaborative Requirements Engineering. Case Study Red:WireTeaching Students Collaborative Requirements Engineering. Case Study Red:Wire
Teaching Students Collaborative Requirements Engineering. Case Study Red:Wire
Dagmar Monett
 
E-Learning Adoption in a Higher Education Setting: An Empirical Study
E-Learning Adoption in a Higher Education Setting: An Empirical StudyE-Learning Adoption in a Higher Education Setting: An Empirical Study
E-Learning Adoption in a Higher Education Setting: An Empirical Study
Dagmar Monett
 
Evolving Lesson Plans to Assist Educators: From Paper-Based to Adaptive Lesso...
Evolving Lesson Plans to Assist Educators: From Paper-Based to Adaptive Lesso...Evolving Lesson Plans to Assist Educators: From Paper-Based to Adaptive Lesso...
Evolving Lesson Plans to Assist Educators: From Paper-Based to Adaptive Lesso...
Dagmar Monett
 
Joint Software Engineering to support STEM Education: Experiences before, dur...
Joint Software Engineering to support STEM Education: Experiences before, dur...Joint Software Engineering to support STEM Education: Experiences before, dur...
Joint Software Engineering to support STEM Education: Experiences before, dur...
Dagmar Monett
 
Methods for Validating and Testing Software Requirements (lecture slides)
Methods for Validating and Testing Software Requirements (lecture slides)Methods for Validating and Testing Software Requirements (lecture slides)
Methods for Validating and Testing Software Requirements (lecture slides)
Dagmar Monett
 
Modelling Software Requirements: Important diagrams and templates (lecture sl...
Modelling Software Requirements: Important diagrams and templates (lecture sl...Modelling Software Requirements: Important diagrams and templates (lecture sl...
Modelling Software Requirements: Important diagrams and templates (lecture sl...
Dagmar Monett
 
Requirements Engineering Methods for Documenting Requirements (lecture slides)
Requirements Engineering Methods for Documenting Requirements (lecture slides)Requirements Engineering Methods for Documenting Requirements (lecture slides)
Requirements Engineering Methods for Documenting Requirements (lecture slides)
Dagmar Monett
 
Requirements Engineering Techniques for Eliciting Requirements (lecture slides)
Requirements Engineering Techniques for Eliciting Requirements (lecture slides)Requirements Engineering Techniques for Eliciting Requirements (lecture slides)
Requirements Engineering Techniques for Eliciting Requirements (lecture slides)
Dagmar Monett
 
A Structured Approach to Requirements Analysis (lecture slides)
A Structured Approach to Requirements Analysis (lecture slides)A Structured Approach to Requirements Analysis (lecture slides)
A Structured Approach to Requirements Analysis (lecture slides)
Dagmar Monett
 
Walking the path from the MOOC to my classroom: My collection of methods and ...
Walking the path from the MOOC to my classroom: My collection of methods and ...Walking the path from the MOOC to my classroom: My collection of methods and ...
Walking the path from the MOOC to my classroom: My collection of methods and ...
Dagmar Monett
 
Understanding the Cuban Blogosphere: Retrospective and Perspectives based on ...
Understanding the Cuban Blogosphere: Retrospective and Perspectives based on ...Understanding the Cuban Blogosphere: Retrospective and Perspectives based on ...
Understanding the Cuban Blogosphere: Retrospective and Perspectives based on ...
Dagmar Monett
 
Genetic Algorithms and Ant Colony Optimisation (lecture slides)
Genetic Algorithms and Ant Colony Optimisation (lecture slides)Genetic Algorithms and Ant Colony Optimisation (lecture slides)
Genetic Algorithms and Ant Colony Optimisation (lecture slides)
Dagmar Monett
 

More from Dagmar Monett (20)

Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...
Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...
Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...
 
Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...
Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...
Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...
 
University-Industry Collaboration's Next Level: A Comparative Study as Basis ...
University-Industry Collaboration's Next Level: A Comparative Study as Basis ...University-Industry Collaboration's Next Level: A Comparative Study as Basis ...
University-Industry Collaboration's Next Level: A Comparative Study as Basis ...
 
The Changing Landscape of Digital Technologies for Learning
The Changing Landscape of Digital Technologies for Learning The Changing Landscape of Digital Technologies for Learning
The Changing Landscape of Digital Technologies for Learning
 
Will Robots Take all the Jobs? Not yet.
Will Robots Take all the Jobs? Not yet.Will Robots Take all the Jobs? Not yet.
Will Robots Take all the Jobs? Not yet.
 
Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...
Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...
Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...
 
Simulating the Fractional Reserve Banking using Agent-based Modelling with Ne...
Simulating the Fractional Reserve Banking using Agent-based Modelling with Ne...Simulating the Fractional Reserve Banking using Agent-based Modelling with Ne...
Simulating the Fractional Reserve Banking using Agent-based Modelling with Ne...
 
Predicting Star Ratings based on Annotated Reviewss of Mobile Apps [Slides]
Predicting Star Ratings based on Annotated Reviewss of Mobile Apps [Slides]Predicting Star Ratings based on Annotated Reviewss of Mobile Apps [Slides]
Predicting Star Ratings based on Annotated Reviewss of Mobile Apps [Slides]
 
Teaching Students Collaborative Requirements Engineering. Case Study Red:Wire
Teaching Students Collaborative Requirements Engineering. Case Study Red:WireTeaching Students Collaborative Requirements Engineering. Case Study Red:Wire
Teaching Students Collaborative Requirements Engineering. Case Study Red:Wire
 
E-Learning Adoption in a Higher Education Setting: An Empirical Study
E-Learning Adoption in a Higher Education Setting: An Empirical StudyE-Learning Adoption in a Higher Education Setting: An Empirical Study
E-Learning Adoption in a Higher Education Setting: An Empirical Study
 
Evolving Lesson Plans to Assist Educators: From Paper-Based to Adaptive Lesso...
Evolving Lesson Plans to Assist Educators: From Paper-Based to Adaptive Lesso...Evolving Lesson Plans to Assist Educators: From Paper-Based to Adaptive Lesso...
Evolving Lesson Plans to Assist Educators: From Paper-Based to Adaptive Lesso...
 
Joint Software Engineering to support STEM Education: Experiences before, dur...
Joint Software Engineering to support STEM Education: Experiences before, dur...Joint Software Engineering to support STEM Education: Experiences before, dur...
Joint Software Engineering to support STEM Education: Experiences before, dur...
 
Methods for Validating and Testing Software Requirements (lecture slides)
Methods for Validating and Testing Software Requirements (lecture slides)Methods for Validating and Testing Software Requirements (lecture slides)
Methods for Validating and Testing Software Requirements (lecture slides)
 
Modelling Software Requirements: Important diagrams and templates (lecture sl...
Modelling Software Requirements: Important diagrams and templates (lecture sl...Modelling Software Requirements: Important diagrams and templates (lecture sl...
Modelling Software Requirements: Important diagrams and templates (lecture sl...
 
Requirements Engineering Methods for Documenting Requirements (lecture slides)
Requirements Engineering Methods for Documenting Requirements (lecture slides)Requirements Engineering Methods for Documenting Requirements (lecture slides)
Requirements Engineering Methods for Documenting Requirements (lecture slides)
 
Requirements Engineering Techniques for Eliciting Requirements (lecture slides)
Requirements Engineering Techniques for Eliciting Requirements (lecture slides)Requirements Engineering Techniques for Eliciting Requirements (lecture slides)
Requirements Engineering Techniques for Eliciting Requirements (lecture slides)
 
A Structured Approach to Requirements Analysis (lecture slides)
A Structured Approach to Requirements Analysis (lecture slides)A Structured Approach to Requirements Analysis (lecture slides)
A Structured Approach to Requirements Analysis (lecture slides)
 
Walking the path from the MOOC to my classroom: My collection of methods and ...
Walking the path from the MOOC to my classroom: My collection of methods and ...Walking the path from the MOOC to my classroom: My collection of methods and ...
Walking the path from the MOOC to my classroom: My collection of methods and ...
 
Understanding the Cuban Blogosphere: Retrospective and Perspectives based on ...
Understanding the Cuban Blogosphere: Retrospective and Perspectives based on ...Understanding the Cuban Blogosphere: Retrospective and Perspectives based on ...
Understanding the Cuban Blogosphere: Retrospective and Perspectives based on ...
 
Genetic Algorithms and Ant Colony Optimisation (lecture slides)
Genetic Algorithms and Ant Colony Optimisation (lecture slides)Genetic Algorithms and Ant Colony Optimisation (lecture slides)
Genetic Algorithms and Ant Colony Optimisation (lecture slides)
 

Recently uploaded

Oliver Asks for More by Charles Dickens (9)
Oliver Asks for More by Charles Dickens (9)Oliver Asks for More by Charles Dickens (9)
Oliver Asks for More by Charles Dickens (9)
nitinpv4ai
 
Standardized tool for Intelligence test.
Standardized tool for Intelligence test.Standardized tool for Intelligence test.
Standardized tool for Intelligence test.
deepaannamalai16
 
Ch-4 Forest Society and colonialism 2.pdf
Ch-4 Forest Society and colonialism 2.pdfCh-4 Forest Society and colonialism 2.pdf
Ch-4 Forest Society and colonialism 2.pdf
lakshayrojroj
 
How to Manage Reception Report in Odoo 17
How to Manage Reception Report in Odoo 17How to Manage Reception Report in Odoo 17
How to Manage Reception Report in Odoo 17
Celine George
 
مصحف القراءات العشر أعد أحرف الخلاف سمير بسيوني.pdf
مصحف القراءات العشر   أعد أحرف الخلاف سمير بسيوني.pdfمصحف القراءات العشر   أعد أحرف الخلاف سمير بسيوني.pdf
مصحف القراءات العشر أعد أحرف الخلاف سمير بسيوني.pdf
سمير بسيوني
 
How to Download & Install Module From the Odoo App Store in Odoo 17
How to Download & Install Module From the Odoo App Store in Odoo 17How to Download & Install Module From the Odoo App Store in Odoo 17
How to Download & Install Module From the Odoo App Store in Odoo 17
Celine George
 
Elevate Your Nonprofit's Online Presence_ A Guide to Effective SEO Strategies...
Elevate Your Nonprofit's Online Presence_ A Guide to Effective SEO Strategies...Elevate Your Nonprofit's Online Presence_ A Guide to Effective SEO Strategies...
Elevate Your Nonprofit's Online Presence_ A Guide to Effective SEO Strategies...
TechSoup
 
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) CurriculumPhilippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
MJDuyan
 
Educational Technology in the Health Sciences
Educational Technology in the Health SciencesEducational Technology in the Health Sciences
Educational Technology in the Health Sciences
Iris Thiele Isip-Tan
 
Skimbleshanks-The-Railway-Cat by T S Eliot
Skimbleshanks-The-Railway-Cat by T S EliotSkimbleshanks-The-Railway-Cat by T S Eliot
Skimbleshanks-The-Railway-Cat by T S Eliot
nitinpv4ai
 
INTRODUCTION TO HOSPITALS & AND ITS ORGANIZATION
INTRODUCTION TO HOSPITALS & AND ITS ORGANIZATION INTRODUCTION TO HOSPITALS & AND ITS ORGANIZATION
INTRODUCTION TO HOSPITALS & AND ITS ORGANIZATION
ShwetaGawande8
 
Geography as a Discipline Chapter 1 __ Class 11 Geography NCERT _ Class Notes...
Geography as a Discipline Chapter 1 __ Class 11 Geography NCERT _ Class Notes...Geography as a Discipline Chapter 1 __ Class 11 Geography NCERT _ Class Notes...
Geography as a Discipline Chapter 1 __ Class 11 Geography NCERT _ Class Notes...
ImMuslim
 
220711130097 Tulip Samanta Concept of Information and Communication Technology
220711130097 Tulip Samanta Concept of Information and Communication Technology220711130097 Tulip Samanta Concept of Information and Communication Technology
220711130097 Tulip Samanta Concept of Information and Communication Technology
Kalna College
 
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptxRESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
zuzanka
 
skeleton System.pdf (skeleton system wow)
skeleton System.pdf (skeleton system wow)skeleton System.pdf (skeleton system wow)
skeleton System.pdf (skeleton system wow)
Mohammad Al-Dhahabi
 
Bonku-Babus-Friend by Sathyajith Ray (9)
Bonku-Babus-Friend by Sathyajith Ray  (9)Bonku-Babus-Friend by Sathyajith Ray  (9)
Bonku-Babus-Friend by Sathyajith Ray (9)
nitinpv4ai
 
Andreas Schleicher presents PISA 2022 Volume III - Creative Thinking - 18 Jun...
Andreas Schleicher presents PISA 2022 Volume III - Creative Thinking - 18 Jun...Andreas Schleicher presents PISA 2022 Volume III - Creative Thinking - 18 Jun...
Andreas Schleicher presents PISA 2022 Volume III - Creative Thinking - 18 Jun...
EduSkills OECD
 
220711130100 udita Chakraborty Aims and objectives of national policy on inf...
220711130100 udita Chakraborty  Aims and objectives of national policy on inf...220711130100 udita Chakraborty  Aims and objectives of national policy on inf...
220711130100 udita Chakraborty Aims and objectives of national policy on inf...
Kalna College
 
78 Microsoft-Publisher - Sirin Sultana Bora.pptx
78 Microsoft-Publisher - Sirin Sultana Bora.pptx78 Microsoft-Publisher - Sirin Sultana Bora.pptx
78 Microsoft-Publisher - Sirin Sultana Bora.pptx
Kalna College
 
BPSC-105 important questions for june term end exam
BPSC-105 important questions for june term end examBPSC-105 important questions for june term end exam
BPSC-105 important questions for june term end exam
sonukumargpnirsadhan
 

Recently uploaded (20)

Oliver Asks for More by Charles Dickens (9)
Oliver Asks for More by Charles Dickens (9)Oliver Asks for More by Charles Dickens (9)
Oliver Asks for More by Charles Dickens (9)
 
Standardized tool for Intelligence test.
Standardized tool for Intelligence test.Standardized tool for Intelligence test.
Standardized tool for Intelligence test.
 
Ch-4 Forest Society and colonialism 2.pdf
Ch-4 Forest Society and colonialism 2.pdfCh-4 Forest Society and colonialism 2.pdf
Ch-4 Forest Society and colonialism 2.pdf
 
How to Manage Reception Report in Odoo 17
How to Manage Reception Report in Odoo 17How to Manage Reception Report in Odoo 17
How to Manage Reception Report in Odoo 17
 
مصحف القراءات العشر أعد أحرف الخلاف سمير بسيوني.pdf
مصحف القراءات العشر   أعد أحرف الخلاف سمير بسيوني.pdfمصحف القراءات العشر   أعد أحرف الخلاف سمير بسيوني.pdf
مصحف القراءات العشر أعد أحرف الخلاف سمير بسيوني.pdf
 
How to Download & Install Module From the Odoo App Store in Odoo 17
How to Download & Install Module From the Odoo App Store in Odoo 17How to Download & Install Module From the Odoo App Store in Odoo 17
How to Download & Install Module From the Odoo App Store in Odoo 17
 
Elevate Your Nonprofit's Online Presence_ A Guide to Effective SEO Strategies...
Elevate Your Nonprofit's Online Presence_ A Guide to Effective SEO Strategies...Elevate Your Nonprofit's Online Presence_ A Guide to Effective SEO Strategies...
Elevate Your Nonprofit's Online Presence_ A Guide to Effective SEO Strategies...
 
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) CurriculumPhilippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) Curriculum
 
Educational Technology in the Health Sciences
Educational Technology in the Health SciencesEducational Technology in the Health Sciences
Educational Technology in the Health Sciences
 
Skimbleshanks-The-Railway-Cat by T S Eliot
Skimbleshanks-The-Railway-Cat by T S EliotSkimbleshanks-The-Railway-Cat by T S Eliot
Skimbleshanks-The-Railway-Cat by T S Eliot
 
INTRODUCTION TO HOSPITALS & AND ITS ORGANIZATION
INTRODUCTION TO HOSPITALS & AND ITS ORGANIZATION INTRODUCTION TO HOSPITALS & AND ITS ORGANIZATION
INTRODUCTION TO HOSPITALS & AND ITS ORGANIZATION
 
Geography as a Discipline Chapter 1 __ Class 11 Geography NCERT _ Class Notes...
Geography as a Discipline Chapter 1 __ Class 11 Geography NCERT _ Class Notes...Geography as a Discipline Chapter 1 __ Class 11 Geography NCERT _ Class Notes...
Geography as a Discipline Chapter 1 __ Class 11 Geography NCERT _ Class Notes...
 
220711130097 Tulip Samanta Concept of Information and Communication Technology
220711130097 Tulip Samanta Concept of Information and Communication Technology220711130097 Tulip Samanta Concept of Information and Communication Technology
220711130097 Tulip Samanta Concept of Information and Communication Technology
 
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptxRESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
 
skeleton System.pdf (skeleton system wow)
skeleton System.pdf (skeleton system wow)skeleton System.pdf (skeleton system wow)
skeleton System.pdf (skeleton system wow)
 
Bonku-Babus-Friend by Sathyajith Ray (9)
Bonku-Babus-Friend by Sathyajith Ray  (9)Bonku-Babus-Friend by Sathyajith Ray  (9)
Bonku-Babus-Friend by Sathyajith Ray (9)
 
Andreas Schleicher presents PISA 2022 Volume III - Creative Thinking - 18 Jun...
Andreas Schleicher presents PISA 2022 Volume III - Creative Thinking - 18 Jun...Andreas Schleicher presents PISA 2022 Volume III - Creative Thinking - 18 Jun...
Andreas Schleicher presents PISA 2022 Volume III - Creative Thinking - 18 Jun...
 
220711130100 udita Chakraborty Aims and objectives of national policy on inf...
220711130100 udita Chakraborty  Aims and objectives of national policy on inf...220711130100 udita Chakraborty  Aims and objectives of national policy on inf...
220711130100 udita Chakraborty Aims and objectives of national policy on inf...
 
78 Microsoft-Publisher - Sirin Sultana Bora.pptx
78 Microsoft-Publisher - Sirin Sultana Bora.pptx78 Microsoft-Publisher - Sirin Sultana Bora.pptx
78 Microsoft-Publisher - Sirin Sultana Bora.pptx
 
BPSC-105 important questions for june term end exam
BPSC-105 important questions for june term end examBPSC-105 important questions for june term end exam
BPSC-105 important questions for june term end exam
 

The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: Annotation, Guidelines, and Student Research Projects

  • 1. The Intelligence Corpus, an annotated corpus of definitions of intelligence Annotation, guidelines, and student research projects @dmonett Dagmar Monett1, Luisa Hoge2, Laura Haase3, Lena Schwarz4, Marc Normann5, Linus Scheibe6 1Berlin School of Economics and Law 2Robert Koch Institute 3Technical University of Applied Sciences Wildau 4Hochschule Stralsund 5NORDAKADEMIE Graduate School 6DB Systel GmbH 14th annual International Conference of Education, Research and Innovation 8th - 9th of November, 2021
  • 2. How to cite this work Monett, D., Hoge, L., Haase, L., Schwarz, L., Normann, M., & Scheibe, L. (2021). The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: Annotation, Guidelines, and Student Research Projects. In Proceedings of the 14th annual International Conference of Education, Research and Innovation, ICERI 2021, November 8th-9th 2021 [online]. Available at: https://www.slideshare.net/dmonett/monett-etal-2021-iceri (Accessed: access date). 2
  • 3. Annotating definitions of intelligence - Fact: Lack of consensus on defining intelligence - Needed: To provide better insights into definitions and how to define them - Why: It is central to the basics of AI literacy - How: In this work, by evaluating the properties of available definitions 3
  • 4. Annotation and Annotators Automatic data annotation still biased, error prone, and far from being entirely satisfactory. Domain knowledge The annotation might require special insights into the problem domain. Software solutions Software solutions are available for supporting annotators in their work (Neves and Ševa, 2021), but not for all kinds of data and domains. Annotation The annotation of data either its nature can be a very challenging and time consuming process. Annotators Undergraduate 3rd year Computer Science students Annotators Parallel course on AI and student research projects M. Neves and J. Ševa, “An extensive review of tools for manual annotation of documents,” Briefings in Bioinformatics, vol. 22, no. 1, pp. 146–163, 2021. 4
  • 5. The Annotation Data A: 213 new, suggested definitions of machine or artificial intelligence by participants to the survey on defining intelligence (Monett & Lewis, 2018) B: 125 new, suggested definitions of human intelligence by participants to the survey on defining intelligence (Monett & Lewis, 2018) C: 34 definitions of intelligence from the literature to agree upon in the initial edition of the survey on defining intelligence (Monett & Lewis, 2018) D: 71 definitions of intelligence from the collection presented in (Legg & Hutter, 2007) D. Monett and C.W.P. Lewis, “Getting clarity by defining Artificial Intelligence—A Survey,” in Philosophy and Theory of Artificial Intelligence (V.C. Müller, ed.), SAPERE vol. 44, pp. 212–214. Springer, Berlin, 2018. S. Legg and M. Hutter, “A Collection of Definitions of Intelligence,” in Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms (B. Goertzel and P. Wang, eds.), vol. 157, pp. 17–24. IOS Press, UK, 2007. 443 definitions of (machine) intelligence 5
  • 6. Examples of Definitions From collection C “Intelligence measures an agent’s ability to achieve goals in a wide range of environments.” From collection D “[Intelligence is] the capacity to learn, reason, and understand.” From collection B “[Human intelligence is] the ability to use information to accomplish goals.” From collection A “Machine Intelligence is concerned with building systems that can adapt and learn in unstructured noisy domains.” 6
  • 7. Example: „A good definition of intelligence is affirmative.“ for definitions as suggested in (Monett & Lewis, 2020) Quality criteria D. Monett and C.W.P. Lewis, “Definitional Foundations for Intelligent Systems, Part I: Quality Criteria for Definitions of Intelligence,” in Proceedings of The 10th Anniversary Conference of the Academic Conference Association (J. Vopava, V. Douda, R. Kratochvil, and M. Konecki, eds.), pp. 73–80, Prague, Czech Republic. MAC Prague Consulting Ltd., 2020. 7
  • 8. Definitions and Annotators Distribution of definitions per groups of annotators 8
  • 9. Annotation Guidelines … on a cell if the corresponding definition fulfils the quality criterion Write a 1 E.g. by column (i.e. one quality criterion at a time); by row (i.e. definition by definition) How to proceed … you spend annotating whenever possible Record the time … when no idea about how to evaluate a given quality criterion Set background color … grammatical errors you might find in the definitions. Do not fix Do not discuss with other; this could introduce some bias Annotate alone … among others from the literature! 9
  • 10. Examples of Annotations Definition It is affirmative It is short It includes cognitive abilities or functions It defines the “what” It does not distinguish between human and machine intelligence “An intelligent machine must first of all be a machine with interests, otherwise there are no interests to be served by its intelligence. Human interests can be served by machine ‘competence,’ not by machine intelligence.” 1 “[Human intelligence is the] ability to achieve objectives in a variety of environments.” 1 1 1 “Intelligence is the computational part of the ability to achieve goals in the world.” 1 1 1 1 “[Intelligence is] the capacity to acquire and apply knowledge.” 1 1 1 1 1 10
  • 11. Inter-Annotator Agreement (IAA) Avg. Cohen’s κ = 0.4 (per group per collection) J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, pp. 37–46, 1960. J.R. Landis and G.G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, pp. 159–174, 1977.  IAA between fair and moderate for all collections (i.e. Cohen’s κ ranging from 0.344 to 0.465, interpretation according to (Landis and Koch, 1977)).  The number of agreements among annotators was higher for the collection containing definitions of human intelligence (collection B).  The quality criteria with the highest IAA values were those simpler, more intuitive, and easier to understand.  The annotators were more agreeable when evaluating the fulfilment of the quality criteria for the following definition: “Intelligence is a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience.” 11
  • 12. The “best” definition: Gottfredson’s L.S. Gottfredson, “Mainstream science on intelligence: An editorial with 52 signatories, history, and bibliography,” Intelligence, vol. 24, pp. 13–23, 1997. R.J. Haier, “The Neuroscience of Intelligence,” Cambridge University Press, New York, NY, 2017. 12
  • 13. Usage: Examples Part of the Intelligence Corpus All definitions from (Monett and Lewis, 18)  71 definitions of machine or artificial intelligence (from a total of 213) from collection A.  42 definitions of human intelligence (from a total of 125) from collection B.  12 definitions of intelligence (from a total of 34) from collection C.  23 definitions of intelligence (from a total of 71) from collection D. https://bit.ly/AnnotatedDefsIntelligence https://goo.gl/KDPtKT The collection of definitions of intelligence used in the research survey “Defining (machine) Intelligence” D. Monett and C.W.P. Lewis, “Getting clarity by defining Artificial Intelligence—A Survey,” in Philosophy and Theory of Artificial Intelligence (V.C. Müller, ed.), SAPERE vol. 44, pp. 212–214. Springer, Berlin, 2018. 13
  • 14. Conclusions Prior experience available in mentoring and supervising student research projects. Parallel course on AI delivered by the same instructor; ad hoc discussions in class. A peculiar annotation case study that evaluates whether definitions of human and machine intelligence satisfy desirable properties or quality criteria of good definitions. Information, materials, and tools carefully prepared and discussed in advance, also throughout the project’s execution. AI course AI Experience Intelligence Corpus Project management Training on the process of defining a good definition of any concept, which could be of interest to regulators or lawyers, for instance. E.g. detailed, manually-conducted quality control of all available annotations. Further work Other uses 14
  • 15. Contact /monettdiaz @dmonett Prof. Dr. Dagmar Monett 15 http://monettdiaz.com dagmar.monett-diaz@hwr-berlin.de Prof. Dr. Computer Science (Artificial Intelligence, Software Engineering) Computer Science Dept. Co-Director M.Sc. Digital Transformation Berlin School of Economics and Law (HWR Berlin) Faculty of Cooperative Studies Template: www.allppt.com