This document discusses learning analytics for evaluating competencies and behaviors in serious games. It begins by introducing the presenters and their affiliations. It then discusses motivations for using games for learning and assessment, noting that games can assess complex skills and be engaging for learners. The document outlines the design, development, and evaluation process for game-based assessment, including gathering data during design and implementing assessment models. It provides an example game called Shadowspect and describes how evidence from the game informs constructs and algorithms to measure skills like efficiency. The document notes future work could include evaluating models with external measures and ensuring generalizability.
Learning Analytics Design in Game-based LearningMIT
Summary: The workshop will deal with the problematic of designing learning analytics in games for learning, it makes special emphasis on the process and the design side, and will prepare assistants to start facing this or similar analytical challenges in the future.
- Methodology: It will be an active workshop where the instructor will do short introductions, present step-by-step examples and then participants will work in their own designs in groups, with the support of the instructor. We finalize by sharing with the rest of the class to see different designs for different games and constructs.
- Intended audience: Will definitely be interesting for anyone working around learning analytics, games for learning and alternative assessment methods. But anyone can enjoy this workshop as it will be dynamic and scaffolded. No requisites needed.
Analytic and strategic challenges of serious gamesDavid Gibson
How higher education learning and teaching can learn from serious game developers. Keynote at the 5th annual SeGAH conference concurrent with WWW 2017 held in Perth, Western Australia
Learning Analytics Design in Game-based LearningMIT
Summary: The workshop will deal with the problematic of designing learning analytics in games for learning, it makes special emphasis on the process and the design side, and will prepare assistants to start facing this or similar analytical challenges in the future.
- Methodology: It will be an active workshop where the instructor will do short introductions, present step-by-step examples and then participants will work in their own designs in groups, with the support of the instructor. We finalize by sharing with the rest of the class to see different designs for different games and constructs.
- Intended audience: Will definitely be interesting for anyone working around learning analytics, games for learning and alternative assessment methods. But anyone can enjoy this workshop as it will be dynamic and scaffolded. No requisites needed.
Analytic and strategic challenges of serious gamesDavid Gibson
How higher education learning and teaching can learn from serious game developers. Keynote at the 5th annual SeGAH conference concurrent with WWW 2017 held in Perth, Western Australia
Machine learning is a type of artificial intelligence (AI) that provides computers with the ability to learn without being explicitly programmed. Machine learning focuses on the development of computer programs that can teach themselves to grow and change when exposed to new data.
Investigating learning strategies in a dispositional learning analytics conte...Bart Rienties
This study aims to contribute to recent developments in empirical studies on students’ learning strategies, whereby the use of trace data is combined with self-report data to distinguish profiles of learning strategy use [3, 4, 5]. We do so in the context of an application of dispositional learning analytics in a large introductory course mathematics and statistics, based on blended learning. Building on our previous work which showed marked differences in how students used worked examples as a learning strategy [7, 11], this study compares different profiles of learning strategies with learning approaches, learning outcomes, and learning dispositions. One of our key findings is that deep learners were less dependent on worked examples as a resource for learning, and that students who only sporadically used worked examples achieved higher test scores.
How AI will change the way you help students succeed - SchooLinksKatie Fang
In this presentation, we are going to uncover
1) why there's so much hype about AI/Machine Learning (and what these things really are)
2) Whirlwind tour of machine learning/statistics techniques and what they mean for counselors
3) Optimism for what the future brings - data as your friend rather than something to be managed.
Report on the First Knowledge Graph Reasoning Challenge 2018 -Toward the eXp...KnowledgeGraph
JIST2019: The 9th Joint International Semantic Technology Conference
The premium Asian forum on Semantic Web, Knowledge Graph, Linked Data and AI on the Web. Nov. 25-27, 2019, Hangzhou, China.
http://jist2019.openkg.cn/
Shou-de Lin is currently a full professor in the CSIE department of National Taiwan University. He holds a BS in EE department from National Taiwan University, an MS-EE from the University of Michigan, and an MS in Computational Linguistics and PhD in Computer Science both from the University of Southern California. He leads the Machine Discovery and Social Network Mining Lab in NTU. Before joining NTU, he was a post-doctoral research fellow at the Los Alamos National Lab. Prof. Lin's research includes the areas of machine learning and data mining, social network analysis, and natural language processing. His international recognition includes the best paper award in IEEE Web Intelligent conference 2003, Google Research Award in 2007, Microsoft research award in 2008, merit paper award in TAAI 2010, best paper award in ASONAM 2011, US Aerospace AFOSR/AOARD research award winner for 5 years. He is the all-time winners in ACM KDD Cup, leading or co-leading the NTU team to win 5 championships. He also leads a team to win WSDM Cup 2016 Champion. He has served as the senior PC for SIGKDD and area chair for ACL. He is currently the associate editor for International Journal on Social Network Mining, Journal of Information Science and Engineering, and International Journal of Computational Linguistics and Chinese Language Processing. He receives the Young Scholars' Creativity Award from Foundation for the Advancement of Outstanding Scholarship and Ta-You Wu Memorial Award.
Invited Talk:
Challenge-Based Learning: Creating engagement by learning from games and gamification
Speaker: Dr. David Gibson, Curtin University
Time: 9:15 – 10:00, 29 May 2015 (Friday)
Venue: Room 408A, 409A & 410, 4/F, Meng Wah Complex, The University of Hong Kong
http://citers2015.cite.hku.hk/program-highlights/talk-gibson/
Learning Analytics and how to use in educational or serious games for improving the use of the games
game traces
evidence based education
Talk at the Ecole Normal Superior, Lyon, France
Complex skill mastery requires not only acquiring individual basic component skills, but also practicing integrating such basic skills. However, traditional approaches to knowledge modeling, such as Bayesian Knowledge Tracing, only trace knowledge of each decomposed basic component skill. This risks early assertion of mastery or ineffective remediation failing to address skill integration. We propose a diagnostic Bayesian network based on a hierarchical integration graph for learner knowledge modeling. We assess the value of such a model from four aspects: performance prediction, parameter plausibility, expected instructional effectiveness, and real-world recommendation helpfulness. Our experiments with a Java programming dataset and a user study based on a Java programming tutor show that proposed model significantly improves two popular multiple skill knowledge tracing models on all these four aspects. Our work serves as a first step towards building skill application context sensitive learner model for modeling and promoting students’ robust learning.
Multiplatform MOOC Analytics: Comparing Global and Regional Patterns in edX a...MIT
Presentation of the full paper at Learning@Scale 2019 in Chicago (IL), USA.
Abstract:While global massive open online course (MOOC) providers such as edX, Coursera, and FutureLearn have garnered the bulk of attention from researchers and the popular press, MOOCs are also provisioned by a series of regional providers, who are often using the Open edX platform. We leverage the data infrastructure shared by the main edX instance and one regional Open edX provider, Edraak in Jordan, to compare the experience of learners from Arab countries on both platforms. Comparing learners from Arab countries on edX to those on Edraak, the Edraak population has a more even gender balance, more learners with lower education levels, greater participation from more developing countries, higher levels of persistence and completion, and a larger total population of learners. This "apples to apples" comparison of MOOC learners is facilitated by an approach to multiplatform MOOC analytics, which employs parallel research processes to create joint aggregate datasets without sharing identifiable data across institutions. Our findings suggest that greater research attention should be paid towards regional MOOC providers, and regional providers may have an important role to play in expanding access to higher education.
Investigación en Learning Analyticsvs.Learning Analytics en la UniversidadMIT
Presentación breve sobre la tensión entre la investigación en learning analytics y learning analytics aplicado en práctica en la universidad.
Video aqui: https://youtu.be/IzGEUdnPbgQ
Programa jornadas: https://congresos.uned.es/w19314/actividad_programa
More Related Content
Similar to Learning Analytics for the Evaluation of Competencies and Behaviors in Serious Games
Machine learning is a type of artificial intelligence (AI) that provides computers with the ability to learn without being explicitly programmed. Machine learning focuses on the development of computer programs that can teach themselves to grow and change when exposed to new data.
Investigating learning strategies in a dispositional learning analytics conte...Bart Rienties
This study aims to contribute to recent developments in empirical studies on students’ learning strategies, whereby the use of trace data is combined with self-report data to distinguish profiles of learning strategy use [3, 4, 5]. We do so in the context of an application of dispositional learning analytics in a large introductory course mathematics and statistics, based on blended learning. Building on our previous work which showed marked differences in how students used worked examples as a learning strategy [7, 11], this study compares different profiles of learning strategies with learning approaches, learning outcomes, and learning dispositions. One of our key findings is that deep learners were less dependent on worked examples as a resource for learning, and that students who only sporadically used worked examples achieved higher test scores.
How AI will change the way you help students succeed - SchooLinksKatie Fang
In this presentation, we are going to uncover
1) why there's so much hype about AI/Machine Learning (and what these things really are)
2) Whirlwind tour of machine learning/statistics techniques and what they mean for counselors
3) Optimism for what the future brings - data as your friend rather than something to be managed.
Report on the First Knowledge Graph Reasoning Challenge 2018 -Toward the eXp...KnowledgeGraph
JIST2019: The 9th Joint International Semantic Technology Conference
The premium Asian forum on Semantic Web, Knowledge Graph, Linked Data and AI on the Web. Nov. 25-27, 2019, Hangzhou, China.
http://jist2019.openkg.cn/
Shou-de Lin is currently a full professor in the CSIE department of National Taiwan University. He holds a BS in EE department from National Taiwan University, an MS-EE from the University of Michigan, and an MS in Computational Linguistics and PhD in Computer Science both from the University of Southern California. He leads the Machine Discovery and Social Network Mining Lab in NTU. Before joining NTU, he was a post-doctoral research fellow at the Los Alamos National Lab. Prof. Lin's research includes the areas of machine learning and data mining, social network analysis, and natural language processing. His international recognition includes the best paper award in IEEE Web Intelligent conference 2003, Google Research Award in 2007, Microsoft research award in 2008, merit paper award in TAAI 2010, best paper award in ASONAM 2011, US Aerospace AFOSR/AOARD research award winner for 5 years. He is the all-time winners in ACM KDD Cup, leading or co-leading the NTU team to win 5 championships. He also leads a team to win WSDM Cup 2016 Champion. He has served as the senior PC for SIGKDD and area chair for ACL. He is currently the associate editor for International Journal on Social Network Mining, Journal of Information Science and Engineering, and International Journal of Computational Linguistics and Chinese Language Processing. He receives the Young Scholars' Creativity Award from Foundation for the Advancement of Outstanding Scholarship and Ta-You Wu Memorial Award.
Invited Talk:
Challenge-Based Learning: Creating engagement by learning from games and gamification
Speaker: Dr. David Gibson, Curtin University
Time: 9:15 – 10:00, 29 May 2015 (Friday)
Venue: Room 408A, 409A & 410, 4/F, Meng Wah Complex, The University of Hong Kong
http://citers2015.cite.hku.hk/program-highlights/talk-gibson/
Learning Analytics and how to use in educational or serious games for improving the use of the games
game traces
evidence based education
Talk at the Ecole Normal Superior, Lyon, France
Complex skill mastery requires not only acquiring individual basic component skills, but also practicing integrating such basic skills. However, traditional approaches to knowledge modeling, such as Bayesian Knowledge Tracing, only trace knowledge of each decomposed basic component skill. This risks early assertion of mastery or ineffective remediation failing to address skill integration. We propose a diagnostic Bayesian network based on a hierarchical integration graph for learner knowledge modeling. We assess the value of such a model from four aspects: performance prediction, parameter plausibility, expected instructional effectiveness, and real-world recommendation helpfulness. Our experiments with a Java programming dataset and a user study based on a Java programming tutor show that proposed model significantly improves two popular multiple skill knowledge tracing models on all these four aspects. Our work serves as a first step towards building skill application context sensitive learner model for modeling and promoting students’ robust learning.
Multiplatform MOOC Analytics: Comparing Global and Regional Patterns in edX a...MIT
Presentation of the full paper at Learning@Scale 2019 in Chicago (IL), USA.
Abstract:While global massive open online course (MOOC) providers such as edX, Coursera, and FutureLearn have garnered the bulk of attention from researchers and the popular press, MOOCs are also provisioned by a series of regional providers, who are often using the Open edX platform. We leverage the data infrastructure shared by the main edX instance and one regional Open edX provider, Edraak in Jordan, to compare the experience of learners from Arab countries on both platforms. Comparing learners from Arab countries on edX to those on Edraak, the Edraak population has a more even gender balance, more learners with lower education levels, greater participation from more developing countries, higher levels of persistence and completion, and a larger total population of learners. This "apples to apples" comparison of MOOC learners is facilitated by an approach to multiplatform MOOC analytics, which employs parallel research processes to create joint aggregate datasets without sharing identifiable data across institutions. Our findings suggest that greater research attention should be paid towards regional MOOC providers, and regional providers may have an important role to play in expanding access to higher education.
Investigación en Learning Analyticsvs.Learning Analytics en la UniversidadMIT
Presentación breve sobre la tensión entre la investigación en learning analytics y learning analytics aplicado en práctica en la universidad.
Video aqui: https://youtu.be/IzGEUdnPbgQ
Programa jornadas: https://congresos.uned.es/w19314/actividad_programa
Ph.D. in Telematics Engineering Defense - 31st of May, 2017
Title: Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments
URL Thesis: http://eprints.networks.imdea.org/1582/1/ThesisJoseRuiperez_IMDEA.pdf
YouTube video: https://www.youtube.com/watch?v=_k571iTNaCA
Abstract: The 'big data' scene has brought new improvement opportunities to most products and services, including education. Web-based learning has become very widespread over the last decade, which in conjunction with the MOOC phenomenon, it has enabled the collection of large and rich data samples regarding the interaction of students with these educational online environments. We have detected different areas in the literature that still need improvement and more research studies. Particularly, in the context of MOOC and SPOC, where we focus our data analysis on the platforms Khan Academy, Open edX and Coursera. More specifically, we are going to work towards learning analytics visualization dashboards, carrying out an evaluation of these visual analytics tools. Additionally, we will delve into the activity and behavior of students with regular and optional activities, badges and their online academically dishonest conduct. The analysis of activity and behavior of students is divided first in exploratory analysis providing descriptive and inferential statistics, like correlations and group comparisons, as well as numerous visualizations that facilitate conveying understandable information. Second, we apply clustering analysis to find different profiles of students for different purposes e.g., to analyze potential adaptation of learning experiences and pedagogical implications. Third, we also provide three machine learning models, two of them to predict learning outcomes (learning gains and certificate accomplishment) and one to classify submissions as illicit or not. We also use these models to discuss about the importance of variables. Finally, we discuss our results in terms of the motivation of students, student profiling, instructional design, potential actuators and the evaluation of visual analytics dashboards providing different recommendations to improve future educational experiments.
A Data-driven Method for the Detection of Close Submitters in Online Learning...MIT
Online learning has become very popular over the last decade. However, there are still many details that remain unknown about the strategies that students follow while studying online. In this study, we focus on the direction of detecting `invisible' collaboration ties between students in online learning environments. Specifically, the paper presents a method developed to detect student ties based on temporal proximity of their assignment submissions. The paper reports on findings of a study that made use of the proposed method to investigate the presence of close submitters in two different massive open online courses. The results show that most of the students (i.e., student user accounts) were grouped as couples, though some bigger communities were also detected. The study also compared the population detected by the algorithm with the rest of user accounts and found that close submitters needed a statistically significant lower amount of activity with the platform to achieve a certificate of completion in a MOOC. These results confirm that the detected close submitters were performing some collaboration or even engaged in unethical behaviors, which facilitates their way into a certificate. However, more work is required in the future to specify various strategies adopted by close submitters and possible associations between the user accounts.
Using Multiple Accounts for Harvesting Solutions in MOOCs MIT
The study presented in this paper deals with copying answers
in MOOCs. Our findings show that a significant fraction of
the certificate earners in the course that we studied have used
what we call harvesting accounts to find correct answers that
they later submitted in their main account, the account for
which they earned a certificate. In total, 2.5% of the users
who earned a certificate in the course obtained the majority
of their points by using this method, and 10% of them used
it to some extent. This paper has two main goals. The first is
to define the phenomenon and demonstrate its severity. The
second is characterizing key factors within the course that affect
it, and suggesting possible remedies that are likely to decrease
the amount of cheating. The immediate implication of
this study is to MOOCs. However, we believe that the results
generalize beyond MOOCs, since this strategy can be used in
any learning environments that do not identify all registrants.
Diseño e Implementación de un Módulo de Analítica de Aprendizaje en la Plataf...MIT
Resumen: El proceso de aprendizaje está sufriendo cambios debido a las posibilidades que proporcionan las nuevas tecnologías en la educación. Una de estas posibilidades es la de recopilación exhaustiva de datos. Actualmente, la mayoría de plataformas de e-learning son capaces de recopilar un gran conjunto de datos de las interacciones de los alumnos en forma de eventos. Sin embargo, esos datos de bajo nivel son difícilmente interpretables directamente por los actores que intervienen en el aprendizaje. Un gran reto es como transformar esos datos de bajo nivel en información inteligente y mostrarla a profesores y alumnos de una manera que sea sencilla de interpretar por ellos. Estos aspectos los trata el área de analítica de aprendizaje (learning analytics) que ha emergido con fuerza en los últimos años. La plataforma Khan Academy es una de las pioneras en mostrar información relevante del proceso de aprendizaje, pero su funcionalidad puede ser ampliamente mejorada para incluir nueva información inteligente que sea de utilidad para mejorar el proceso de aprendizaje. En este trabajo, se ha diseñado e implementado un módulo de analítica de aprendizaje para la plataforma Khan Academy, que extiende el soporte que proporciona esta plataforma por defecto. Para ello, se han definido una serie de parámetros interesantes para conocer más acerca del proceso de aprendizaje y se ha establecido la manera de procesarlos a partir de datos de bajo nivel. Además, se han implementado estos parámetros, así como visualizaciones basadas en ellos de manera que se muestren informaciones tanto individuales como de la clase. Finalmente, se muestra como este módulo y los parámetros definidos pueden ser utilizados para evaluar el proceso de aprendizaje, ilustrándolo en los cursos 0 de la Universidad Carlos III de Madrid, donde se ha utilizado la plataforma Khan Academy así como el módulo ALAS-KA (Add-on of the Learning Analytics Support of the Khan Academy)
Macroeconomics- Movie Location
This will be used as part of your Personal Professional Portfolio once graded.
Objective:
Prepare a presentation or a paper using research, basic comparative analysis, data organization and application of economic information. You will make an informed assessment of an economic climate outside of the United States to accomplish an entertainment industry objective.
Introduction to AI for Nonprofits with Tapp NetworkTechSoup
Dive into the world of AI! Experts Jon Hill and Tareq Monaur will guide you through AI's role in enhancing nonprofit websites and basic marketing strategies, making it easy to understand and apply.
The French Revolution, which began in 1789, was a period of radical social and political upheaval in France. It marked the decline of absolute monarchies, the rise of secular and democratic republics, and the eventual rise of Napoleon Bonaparte. This revolutionary period is crucial in understanding the transition from feudalism to modernity in Europe.
For more information, visit-www.vavaclasses.com
Instructions for Submissions thorugh G- Classroom.pptxJheel Barad
This presentation provides a briefing on how to upload submissions and documents in Google Classroom. It was prepared as part of an orientation for new Sainik School in-service teacher trainees. As a training officer, my goal is to ensure that you are comfortable and proficient with this essential tool for managing assignments and fostering student engagement.
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
Operation “Blue Star” is the only event in the history of Independent India where the state went into war with its own people. Even after about 40 years it is not clear if it was culmination of states anger over people of the region, a political game of power or start of dictatorial chapter in the democratic setup.
The people of Punjab felt alienated from main stream due to denial of their just demands during a long democratic struggle since independence. As it happen all over the word, it led to militant struggle with great loss of lives of military, police and civilian personnel. Killing of Indira Gandhi and massacre of innocent Sikhs in Delhi and other India cities was also associated with this movement.
Biological screening of herbal drugs: Introduction and Need for
Phyto-Pharmacological Screening, New Strategies for evaluating
Natural Products, In vitro evaluation techniques for Antioxidants, Antimicrobial and Anticancer drugs. In vivo evaluation techniques
for Anti-inflammatory, Antiulcer, Anticancer, Wound healing, Antidiabetic, Hepatoprotective, Cardio protective, Diuretics and
Antifertility, Toxicity studies as per OECD guidelines
Model Attribute Check Company Auto PropertyCeline George
In Odoo, the multi-company feature allows you to manage multiple companies within a single Odoo database instance. Each company can have its own configurations while still sharing common resources such as products, customers, and suppliers.
2024.06.01 Introducing a competency framework for languag learning materials ...Sandy Millin
http://sandymillin.wordpress.com/iateflwebinar2024
Published classroom materials form the basis of syllabuses, drive teacher professional development, and have a potentially huge influence on learners, teachers and education systems. All teachers also create their own materials, whether a few sentences on a blackboard, a highly-structured fully-realised online course, or anything in between. Despite this, the knowledge and skills needed to create effective language learning materials are rarely part of teacher training, and are mostly learnt by trial and error.
Knowledge and skills frameworks, generally called competency frameworks, for ELT teachers, trainers and managers have existed for a few years now. However, until I created one for my MA dissertation, there wasn’t one drawing together what we need to know and do to be able to effectively produce language learning materials.
This webinar will introduce you to my framework, highlighting the key competencies I identified from my research. It will also show how anybody involved in language teaching (any language, not just English!), teacher training, managing schools or developing language learning materials can benefit from using the framework.
6. Main contributors to this research
José A. Ruipérez-Valiente
BEng Telecomunications Systems (UCAM),
MEng Telecomunications, MSc y PhD
Telematics (UC3M), Postdoc (MIT)
6 years working in learning analytics across
many objectives and contexts
Currently focused in large scale trends in
MOOCs and game-based assessment
Juan de la Cierva Researcher at UMU and
affiliate at MIT Playful Journey Lab
YJ (Yoon Jeon) Kim
Executive Director Playful
Journey Lab located at
MIT Open Learning
Assessment scientist
Focus on games and
playful approaches for
assessment
7. Topics related to this talk
- Games for Learning
- Game-based Assessment
- Learning Analytics
- … and Design (which is transverse to numerous areas and applications)
9. A game is a voluntary interactive
activity, in which one or more players
follow rules that constrain their
behavior, enacting an artificial conflict
that ends in a quantifiable outcome.
~Eric Zimmerman (2004)
10. Why Games?
● Games are “flexible enough for players to
inhabit and explore through meaningful
play” (Salen & Zimmerman) (deep learning)
● Majority of children grow up playing games
● Learners have more freedom related to
how much effort they choose to expend,
how often they fail and try again (Osterweil,
2014) (real life)
11. Assessment is a process of reasoning
from evidence. Therefore, an
assessment is a tool designed to
observe students’ behavior and
produce data that can be used to draw
reasonable inferences about what
students know.
~ Bob Mislevy
12. Why Games for Assessment?
● Games incorporate multiple pathways to solution(s) where learners can make
meaningful choices and demonstrate multiple ways of solving problems
● Use complex and authentic problems → hard-to-measure constructs
o We need to assess 21st century skills
● Games are motivating and engaging → accurate assessment (Sundre &
Wise, 2003)
● It doesn’t feel like assessment (i.e. stealth assessment)
o Less stresful situations for students
14. The Broad view of Learning Analytics
…collection, analysis and reporting of data about learners and
their contexts, for purposes of understanding and optimising
learning and the environments in which it occurs…
Source: First Learning Analytics
and Knowledge Conference
15. The Learning Analytics data-driven Process
Raw data
generation
Feature
engineering
Visualizations
Recommendation
Report generator
Meaningful features
Which raw data is
necessary?
What to do with the processed
data?
What to obtain and How
to do it?
Technology as an engine to enhance learning
Exploration,
Correlation,
clustering,
prediction,
causes…
Learning
environments
Conclusions generate feedback and close the LA loop
18. Design
● Design and implementation of game system
○ Game mechanics that can generate evidence
from the constructs and a data infrastructure that
effectively stores that evidence
○ The most iterative step of the process with very
frequent playtesting
1. Start with paper prototypes
2. Move to drafty digital prototypes
3. End with advanced digital prototypes
● Data collection
○ Diverse audiences and contexts
○ Very important for game mechanics and tech side
○ Face-to-face playtesting
○ Amazon MTurk
24. Model development
● Implementation of the assessment machinery:
○ Process of turning evidence into constructs
○ Content knowledge assessment: Following a
traditional Evidence-centered Design
○ Cognitive and behavioral assessment: Combining
knowledge engineering process and ML with expert
labelling
● Data collection:
○ Same high school context, age, and settings
○ Two sessions of one hour each
○ Around 10 US high school classes and more than 200
hundred students
27. Common Core Geometry Standards
● Competency model: We focus on the common core geometry standards
o MG.A.1: Use geometric shapes, their measures, and their properties to describe
objects (e.g., modeling a tree trunk or a human torso as a cylinder)
o GMD.B.4: Identify the shapes of two-dimensional cross-sections of three-
dimensional objects, and identify three-dimensional objects generated by rotations
of two-dimensional objects
o CO.A.5: Given a geometric figure and a rotation, reflection, or translation, draw the
transformed figure
o CO.B.6: Use geometric descriptions of rigid motions to transform figures and to
predict the effect of a given rigid motion on a given figure
28. ECD Summary for Geometry Common Standards Assessmement
● Collaboration with geometry specialist, game designer and assessment designer
○ Evidence model: We generate puzzles that generate evidence from the Geometry Common Standards
○ Task model: We map the relationship (none, weak or strong) of each puzzle with the common standard
○ Assembly model: We put all the evidence from a student together to assess their content knowledge
○ Presentation & Delivery model: Reports and dashboards by student/standard. Difficulty by exercise
Puzzle MG.A.1 GMD.B.4 …
Puzzle 1 Weak Weak …
Puzzle 2 None None …
… … … …
Student Puzzle 1 Puzzle 2 …
Student 1
OK, # 1
attempt
OK, # 3
attempts
…
Student 1 NA
Fail, # 5
attempt
…
… … … …
31. Knowledge Engineering Process
● We acquire knowledge about the construct that we want to measure
1. Reading about the construct
2. Conducting interview with experts
3. Reviewing related scientific literature
● We algorithmically implement features that use the data/evidence that can inform the
construct that we want to measure
32. Our simplified case scenario now updates to:
Evidence Constructs
map
Data Features
data schema inform
algorithms
33. Efficiency construct
- Efficiency is the ability to do things well, successfully, and without waste. It
often specifically comprises the capability of a specific application of effort
to produce a specific outcome with a minimum amount or quantity of
waste, expense, or unnecessary effort (Wikipedia)
34. Evidence in Shadowspect related to efficiency
● Ability to do things well:
○ Solving puzzles correctly
● Expense or effort:
○ Time invested
○ Number of attempts to solve a problem
35. Mapping evidence into necessary data in Shadowspect
● We need: puzzles solved correctly, time invested and attempts
○ Necessary types of events for that:
■ puzzle_start (timestamp, student, puzzle_id)
■ leave_to_menu (timestamp, student, puzzle_id)
■ puzzle_attempt (timestamp, student, puzzle_id, correct)
37. Algorithm to compute features from data (pseudo-code)
# note this is a VERY simplified version that do not aim to be the most effective implementation of this algorithm
computeEfficiencyFeatures(student):
student_events = getStudentEvents(student)
correct_exercises_list = list(); number_attempts = 0; total_time = 0; puzzle_started_event = None
for event in student_events:
if(event[‘type’] == ‘puzzle_started’) then
puzzle_started_event = event
elif(event[‘type’] == ‘leave_to_menu’) then
total_time += (event[‘timestamp’] - puzzle_started_event[‘timestamp’])
puzzle_started_event = None
elif(event[‘type’] == ‘puzzle_attempt’):
number_attempts += 1
if(event[‘correct’] == True) then
correct_exercises_list.add(event[‘puzzle_id’])
attempts_per_correct_problem = length(unique(correct_exercises_list))/number_attempts
time_per_correct_problem = length(unique(correct_exercises_list))/total_time
return(attempts_per_correct_problem, time_per_correct_problem)
38. The previous general scenario
Evidence Constructs
map
Data Features
data schema inform
algorithms
39. Model for efficiency in Shadowspect
Evidence
● Correct puzzles
● Time
● Number attempts
Data
● puzzle_start
● leave_to_menu
● puzzle_attempt
data schema inform
computeEfficiency
Features(student)
Construct
Efficiency
Features
attempts_per_correct_problem
time_per_correct_problem
map
41. Expert Labelling and Machine Learning Process
● Two or more experts label text or video replays that can be visually assessed
○ We divide all level interactions in replays that can be labelled
○ Experts review replays and label them for each construct that we want to measure
■ They might use rubrics and we are looking for expert inter-agreement (Cohen’s kappa)
○ We implement a supervised machine learning assessment model based on these labels
● Challenges here include achieving good inter-agreement, technical logistics, replay
resolution and final implementation of the ML model
Example of simplified text replay: 1. Start puzzle – 2. Create shape square – 3. Move square – 4. Create cone
5. Rotate cone – 6. Change perspective – 7. Snapshot – 8. Move cone – 9. Submit – 10 Puzzle correct
42. Expert Labelling and Machine Learning Process
Evidence
Constructsmap
Data Features
data schema
inform
algorithms
expert
assessment
ML/AI
43. Evaluation
● We are not here yet! Future plans:
● Data collection:
○ Implementation as part of the curriculum in high
school classes
○ Demographic and school data with external measures
● Game analytics: How is the game being used by
students? Improvements, enjoyment…
● Model performance evaluation: How are the
models working? What do teachers think about
models?
● Psychometric evaluation: Are our models
correlated to other external tests, e.g. geometry
traditional tests or spatial reasoning validated
instruments
44. It’s time to say goodbye
But let’s conclude before that
45. Conclusions
● Alternative assessment method with great potential
○ Focus on complex constructs, can focus on the process (on only outcomes), is less stressful
and more enjoyable for students
● Highly challenging and multidisciplinary field, main problems:
○ Cost, scalability and generalization across GBA tools, model validity, trustworthiness, and
teacher literacy
● Some companies are already using GBA as part pre-hiring
● Difference between Assessment and assessment
● Opportunities for collaboration!
begins by identifying what should be assessed in terms of knowledge, skills, or other learner attributes. These variables cannot be observed directly, so behaviors and performances that demonstrate these variables need to be identified instead. The next step is determining the types of tasks or situations that would draw out such behaviors or performances.
Example around simple math knowledge in a game: