Adaptive navigation support systems can increase student motivation and learning outcomes when used in online educational systems. Three key findings from the document:
1. Studies found adaptive navigation support significantly increased student activity, persistence, breadth of exploration, and learning gains compared to non-adaptive systems across different domains like programming, hypermedia, and examples.
2. Adaptive navigation was particularly effective for easier content, increasing success rates and attempts per question. For complex content, prerequisite-based guidance helped prepare students.
3. Social navigation using progress of peer students was also effective at increasing motivation and learning, replacing the need for extensive knowledge modeling in adaptive systems.
Stereotype Modeling for Problem-Solving Performance Predictions in MOOCs and ...Peter Brusilovsky
This study examined whether student stereotypes could be used to improve predictions of problem-solving performance in MOOCs. The researchers tested simple stereotypes based on demographics and performance, but found they did not yield accurate models of different student groups' learning. More advanced stereotypes based on patterns of problem-solving behavior also did not distinguish groups with significantly different learning models. The findings suggest stereotypes may not effectively represent the finer-grained differences in how students approach learning. Overall, the study did not find evidence that stereotype models can improve predictions of problem-solving over alternative models of individual student learning.
Human-Centered AI in AI-ED - Keynote at AAAI 2022 AI for Education workshopPeter Brusilovsky
This document summarizes research on human-centered AI in AI education (AIED) systems. It discusses the need for transparency, interactivity, and collaboration between humans and AI in AIED. Some key points:
1) Early "expert systems" lacked transparency and trust, motivating research on explainable, transparent, and human-centered AI.
2) Modern research aims to make learner and content models visible, allow user control of AI parameters and recommendations, and provide explanations for AI decisions.
3) Several AIED systems are discussed that collaborate with users, visualize models, and give users control over navigation, rankings, and social comparisons to improve learning outcomes.
Data-Driven Education 2020: Using Big Educational Data to Improve Teaching an...Peter Brusilovsky
Modern educational settings from regular classrooms to MOOCs produce a a rapidly increasing volume of data that captures individual learning progress of millions of students at different level of granularity. This presence of this data opens a unique opportunity to re-engineer traditional education and build and develop a range of efficient data-driven approaches to support teaching and learning. In my talk, I will present several ways to use big educational data explored in our lab. The focus will be on open social learning modeling and identifying individual differences through sequential pattern mining, but several other approaches will be mentioned. Open social learning modeling and sequential pattern mining provides two considerably different examples on using educational data. One offers an immediate use of class interaction history to develop more engaging content access while another shows how big data could be used to uncover important individual differences that could be used to optimize the process for individual leaners.
Iui2015: Personalized Search: Reconsidering the Value of Open User ModelsPeter Brusilovsky
IUI 2015 talk slides: Ahn, J., Brusilovsky, P., and Han, S. (2015) Personalized Search: Reconsidering the Value of Open User Models. In: Proceedings of Proceedings of the 20th International Conference on Intelligent User Interfaces, Atlanta, Georgia, USA, ACM, pp. 202-212
Human Interfaces to Artificial Intelligence in EducationPeter Brusilovsky
Human Interfaces to Artificial Intelligence in Education discusses:
1) The need for transparency, interactivity, and human-centered design when developing AI systems for education to address issues like lack of ability to inspect AI decisions and lack of trust in AI recommendations.
2) Approaches like explainable AI, visualizing learner models and domain models, and natural communication with AI systems to provide transparency and user control.
3) Examples of open learner model visualizations and explanatory recommendations that make learner knowledge and AI recommendations more transparent.
User Control in AIED (Artificial Intelligence in Education)Peter Brusilovsky
This document summarizes research on improving user control and personalization in artificial intelligence for education (AIED) systems. It discusses several AIED systems that provide adaptive navigation support and annotation based on user models while allowing user control over sequencing and navigation. Evaluation of these systems found they can reduce effort, encourage exploration, and increase learning outcomes when users are able to follow or override advice. The document also presents approaches that improve transparency and control through open learner models, controllable ranking, visualization of recommendation models, and balancing adaptation with user exploration.
Interfaces for User-Controlled and Transparent RecommendationsPeter Brusilovsky
This document discusses interfaces for providing transparent and user-controlled recommendations. It outlines problems with single ranked lists and proposes solutions like explaining recommendations, visualizing relevance processes, and allowing users to explore and control personalization. Specific solutions discussed include visualization tools, open learner/user models, and interfaces that combine relevance from multiple sources and allow controlling factors. Studies found that explorable and controllable recommendations better support understanding relevance and finding relevant items.
Personalization in the Context of Relevance-Based VisualizationPeter Brusilovsky
In this talk, I will review our research attempts to
implement different kinds of personalization in the context
of relevance-based visualization. The goal of this research
stream is to make relevance-based visualization adaptive to
user long-term goals, interests, or prospects rather just
responsive to short term immediate needs such as query
terms. I will present four personalized relevance-based
visualization systems: Adaptive VIBE, TalkExplorer,
SetFusion, and IntersectionExplorer, For each system, I
will present its idea, some evaluation results, and
lessons learned.
https://doi.org/10.1145/3038462.3038474
Stereotype Modeling for Problem-Solving Performance Predictions in MOOCs and ...Peter Brusilovsky
This study examined whether student stereotypes could be used to improve predictions of problem-solving performance in MOOCs. The researchers tested simple stereotypes based on demographics and performance, but found they did not yield accurate models of different student groups' learning. More advanced stereotypes based on patterns of problem-solving behavior also did not distinguish groups with significantly different learning models. The findings suggest stereotypes may not effectively represent the finer-grained differences in how students approach learning. Overall, the study did not find evidence that stereotype models can improve predictions of problem-solving over alternative models of individual student learning.
Human-Centered AI in AI-ED - Keynote at AAAI 2022 AI for Education workshopPeter Brusilovsky
This document summarizes research on human-centered AI in AI education (AIED) systems. It discusses the need for transparency, interactivity, and collaboration between humans and AI in AIED. Some key points:
1) Early "expert systems" lacked transparency and trust, motivating research on explainable, transparent, and human-centered AI.
2) Modern research aims to make learner and content models visible, allow user control of AI parameters and recommendations, and provide explanations for AI decisions.
3) Several AIED systems are discussed that collaborate with users, visualize models, and give users control over navigation, rankings, and social comparisons to improve learning outcomes.
Data-Driven Education 2020: Using Big Educational Data to Improve Teaching an...Peter Brusilovsky
Modern educational settings from regular classrooms to MOOCs produce a a rapidly increasing volume of data that captures individual learning progress of millions of students at different level of granularity. This presence of this data opens a unique opportunity to re-engineer traditional education and build and develop a range of efficient data-driven approaches to support teaching and learning. In my talk, I will present several ways to use big educational data explored in our lab. The focus will be on open social learning modeling and identifying individual differences through sequential pattern mining, but several other approaches will be mentioned. Open social learning modeling and sequential pattern mining provides two considerably different examples on using educational data. One offers an immediate use of class interaction history to develop more engaging content access while another shows how big data could be used to uncover important individual differences that could be used to optimize the process for individual leaners.
Iui2015: Personalized Search: Reconsidering the Value of Open User ModelsPeter Brusilovsky
IUI 2015 talk slides: Ahn, J., Brusilovsky, P., and Han, S. (2015) Personalized Search: Reconsidering the Value of Open User Models. In: Proceedings of Proceedings of the 20th International Conference on Intelligent User Interfaces, Atlanta, Georgia, USA, ACM, pp. 202-212
Human Interfaces to Artificial Intelligence in EducationPeter Brusilovsky
Human Interfaces to Artificial Intelligence in Education discusses:
1) The need for transparency, interactivity, and human-centered design when developing AI systems for education to address issues like lack of ability to inspect AI decisions and lack of trust in AI recommendations.
2) Approaches like explainable AI, visualizing learner models and domain models, and natural communication with AI systems to provide transparency and user control.
3) Examples of open learner model visualizations and explanatory recommendations that make learner knowledge and AI recommendations more transparent.
User Control in AIED (Artificial Intelligence in Education)Peter Brusilovsky
This document summarizes research on improving user control and personalization in artificial intelligence for education (AIED) systems. It discusses several AIED systems that provide adaptive navigation support and annotation based on user models while allowing user control over sequencing and navigation. Evaluation of these systems found they can reduce effort, encourage exploration, and increase learning outcomes when users are able to follow or override advice. The document also presents approaches that improve transparency and control through open learner models, controllable ranking, visualization of recommendation models, and balancing adaptation with user exploration.
Interfaces for User-Controlled and Transparent RecommendationsPeter Brusilovsky
This document discusses interfaces for providing transparent and user-controlled recommendations. It outlines problems with single ranked lists and proposes solutions like explaining recommendations, visualizing relevance processes, and allowing users to explore and control personalization. Specific solutions discussed include visualization tools, open learner/user models, and interfaces that combine relevance from multiple sources and allow controlling factors. Studies found that explorable and controllable recommendations better support understanding relevance and finding relevant items.
Personalization in the Context of Relevance-Based VisualizationPeter Brusilovsky
In this talk, I will review our research attempts to
implement different kinds of personalization in the context
of relevance-based visualization. The goal of this research
stream is to make relevance-based visualization adaptive to
user long-term goals, interests, or prospects rather just
responsive to short term immediate needs such as query
terms. I will present four personalized relevance-based
visualization systems: Adaptive VIBE, TalkExplorer,
SetFusion, and IntersectionExplorer, For each system, I
will present its idea, some evaluation results, and
lessons learned.
https://doi.org/10.1145/3038462.3038474
What Should I Do Next? Adaptive Sequencing in the Context of Open Social Stu...Peter Brusilovsky
This document summarizes research on adaptive sequencing in open social student modeling. It describes how combining knowledge-based guidance with social guidance can encourage non-sequential navigation, increase learning speed for strong students, and positively relate to student performance. A classroom study found that adaptive sequencing increased learning speed and the odds of correct problem solving. Students also provided positive subjective feedback about recommendations. Future work aims to explore alternative visualization and awareness techniques.
From Expert-Driven to Data-Driven Adaptive LearningPeter Brusilovsky
Keynote slides for the Workshop on Advancing Education with Data at the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, Aug 14, 2017
The document discusses domain modeling for personalized learning. It defines a domain model as representing domain knowledge through concepts and their relationships. Domain models serve as the basis for individual student models and for indexing and classifying learning content. They can be used to model student knowledge and decide on appropriate next steps for learning. The document describes different types of domain models, including vector, network, conceptual, and procedural models. It also discusses using ontologies and different aspects in domain modeling and applying domain models to student modeling, content indexing, and personalized guidance.
Two Brains are Better than One: User Control in Adaptive Information AccessPeter Brusilovsky
In recent years, the use of Artificial Intelligence (AI) technologies expanded to many areas where they directly affect the lives of many people. AI-based approaches advise human decision-makers who should be released on bail, whether it is a good time to discharge a patient from a hospital and whether a specific student is at risk to fail a course. Such an extensive use in AI in decision making came with a range of protentional problems that have been extensively studied over the last few years. Recognition of these problems motivated a rapid rise of research on “human-centered AI”, which attempted to address and minimize the negative effects of using AI technologies. Among the ideas of human-centered AI is user control - engaging users in affecting AI decision making to prevent possible errors and biases. In my talk, I will focus on the application of user control in one popular area of AI application, adaptive information access. Adaptive information access systems such as personalized search and recommender systems attempt to model their users to help them in finding the most relevant information. Yet, user modeling and personalization mechanisms might not always work as expected resulting in errors, biases, and suboptimal behavior. Combining the decision power or AI with the ability of the user to guide and control it brings together the strong sides of artificial and human intelligence and could lead to better results. In my talk, I review several projects focused on user control in adaptive information access systems and discuss the benefits and challenges of this approach.
Personalized Online Practice Systems for Learning ProgrammingPeter Brusilovsky
An adaptive learning system called Mastery Grids was created to increase student engagement with online educational content by incorporating personalized and social adaptive features. Mastery Grids uses open learner modeling to display a student's knowledge progress compared to their peers, adaptive navigation support to guide students to relevant activities, and concept-based recommendations of content. A study found that Mastery Grids significantly increased student success rates, time spent engaging with content, and learning compared to non-adaptive systems. Further research added direct recommendations to Mastery Grids and found they increased transparency and led to more efficient learning when explanations of recommendations were provided through the open learner model visualizations.
Data-Driven Education: Using Big Educational Data to Improve Teaching and Learning. Keynote slides for 15th International Conference on Web-Based Learning, ICWL 2016, Rome, Italy, October 26–29.
The Value of Social: Comparing Open Student Modeling and Open Social Student ...Peter Brusilovsky
Brusilovsky, P., Somyurek, S., Guerra, J., Hosseini, R., and Zadorozhny, V. (2015) The Value of Social: Comparing Open Student Modeling and Open Social Student Modeling. In: F. Ricci, K. Bontcheva, O. Conlan and S. Lawless (eds.) Proceedings of 23nd Conference on User Modeling, Adaptation and Personalization (UMAP 2015), Dublin, Ireland, , June 29 - July 3, 2015, Springer Verlag, pp. 44-55, also available at http://link.springer.com/chapter/10.1007/978-3-319-20267-9_4.
Learning analytics are more than a technologyDragan Gasevic
Learning analytics aim to optimize learning through measurement, collection, analysis and reporting of student data. While interest is growing, few institutions have fully adopted analytics. Challenges include a lack of data-informed culture, focusing on solutions over research, and privacy concerns. Fully realizing analytics potential requires multidisciplinary teams, addressing complex educational systems, and developing an analytics-focused culture.
This document proposes an ontological model for representing rubrics digitally using Semantic Web standards like RDF and OWL. Currently, most rubrics shared online are in static, non-machine readable formats like Word documents or proprietary learning management systems. The proposed model aims to make rubrics sharable and reusable across different systems on the web by representing them semantically. It discusses how rubrics benefit both students and teachers by providing clear evaluation criteria and allowing for consistent grading. However, existing rubrics online often lack specificity and are not in open, transferable formats between different tools and systems.
Mastery Grids: An Open Source Social Educational Progress VisualizationPeter Brusilovsky
Presentation for EC-TEL 2015 paper:
Loboda, T., Guerra, J., Hosseini, R., and Brusilovsky, P. (2014) Mastery Grids: An Open Source Social Educational Progress Visualization. In: S. de Freitas, C. Rensing, P. J. Muñoz Merino and T. Ley (eds.) Proceedings of 9th European Conference on Technology Enhanced Learning (EC-TEL 2014), Graz, Austria, September 16-19, 2014, pp. 235-248.
Exploring Tools for Promoting Teacher Efficacy with mLearning (mlearn 2014 Pr...Robert Power
Slides for my presentation with Dean Cristol and Belinda Gimbert of Ohio State University at mLearn 2014, November 4, 2014, at Kadir-Has University in Istanbul, Turkey.
Adaptive Navigation Support and Open Social Learner Modeling for PALPeter Brusilovsky
This presentation is an overview of Open Social Learner Modeling project. It presents Mastery Grids interface, distributed personalized learning architecture Aggregate, and smart content for Java, Python, and SQL
This document summarizes a presentation on learning analytics given by Simon Buckingham Shum. Some key points:
- Learning analytics aims to unlock student data to improve 21st century learning by analyzing patterns in data to better understand learning processes and identify students who may need help.
- Examples discussed include Purdue University's predictive model that identified 66-80% of struggling students and a system that provides real-time feedback to students.
- Analytics can look beyond grades and course performance to capture data on learning dispositions, engagement, curriculum mastery, and student discourse to provide a more holistic view of the learning process.
- Challenges include ensuring analytics are used ethically and to improve learning rather than
Learning with me Mate: Analytics of Social Networks in Higher EducationDragan Gasevic
Effects of social interactions are reported in research on higher education to lead to positive outcomes such as higher levels of internalization, sense of community, academic achievement, metacognition, and student retention. The role of social networks has especially been emphasized in research due to the availability of theoretical foundations and analytic methods to investigate their effects in higher education. The increased use of technologies in education allows for the collection of large and rich datasets about social networks which call for the use of novel analytics methods. This talk will first give a brief overview of the existing work on and lessons learned from some well-known studies on social networks in higher education in diverse situations from face-to-face to massive open online courses. The talk will then identify critical challenges that require immediate attention in order for the study of social networks to make a sustainable impact on learning and teaching. The most important take away from the talk will be that
- computational aspects of the study of social networks need to be integrated deeply with theory, research and practice,
- novel methods for the study of critical dimensions (discourse, structure and dynamics) that shape network formation and network effects are necessary, and
- innovative instructional approaches are essential to address the changing conditions created by contemporary educational and technological contexts.
Tools and Evaluation Techniques to Support Social Awareness in CSCeL: The AVA...EuroCAT CSCL
1. The document describes a 7-step process for designing participation avatars to increase engagement in an online teacher training program in Greece.
2. User evaluations found that calculating participation levels based on the highest posting student and using varied colors improved understanding and motivation.
3. A final study showed the avatars improved active participation and satisfaction with the program compared to the control group without avatars.
This document discusses the evaluation of TOIA, a free online assessment tool. It aimed to test the functionality of TOIA, identify usability issues, and understand how it would be used. The evaluation found that TOIA was easy to use and provided a comprehensive set of assessment tools. However, users noted a lack of question types and concerns about long-term maintenance as a free software. Overall the evaluation helped improve TOIA and provided insights into effective online assessment.
Empirical studies of adaptive annotation in the educational context have demonstrated that it can help students to acquire knowledge faster, improve learning outcomes, reduce navigational overhead, and encourage non-sequential navigation. Over the last 8 years we have explored a lesser known effect of adaptive annotation – its ability to significantly increase student engagement in working with non-mandatory educational content. In the presence of adaptive link annotation, students tend to access significantly more learning content; they stay with it longer, return to it more often and explore a wider variety of learning resources. This talk will present an overview of our exploration of the addictive links effect in many course-long studies, which we ran in several domains (C, SQL and Java programming), for several types of learning content (quizzes, problems, interactive examples). The first part of the talk will review our exploration of a more traditional knowledge-based personalization approach and the second part will focus on more recent studies of social navigation and open social student modeling
Personalized Learning: Expanding the Social Impact of AIPeter Brusilovsky
Slide of my keynote talk at SIAIA '23 workshop held at AAAI 2023:
The use of AI in Education could be traced to the early days of AI. While the publicity associated with the most recent wave of AI applications rarely mentions education, it is through the improvement in education AI could achieve an impressive social impact. In particular, the AI ability to personalize the learning process could make a large difference in a context where learners' knowledge could be radically different from learner to learner. Modern computer and internet technologies can now bring the power of learning in the forms of MOOCs, online textbooks, and zoom courses truly worldwide. Yet, without personalization, the potential of these technologies is not fully leveraged. In this talk, I will review several generations of research on personalized learning and discuss tools, technologies, and infrastructures for personalized learning that we are currently exploring.
What Should I Do Next? Adaptive Sequencing in the Context of Open Social Stu...Peter Brusilovsky
This document summarizes research on adaptive sequencing in open social student modeling. It describes how combining knowledge-based guidance with social guidance can encourage non-sequential navigation, increase learning speed for strong students, and positively relate to student performance. A classroom study found that adaptive sequencing increased learning speed and the odds of correct problem solving. Students also provided positive subjective feedback about recommendations. Future work aims to explore alternative visualization and awareness techniques.
From Expert-Driven to Data-Driven Adaptive LearningPeter Brusilovsky
Keynote slides for the Workshop on Advancing Education with Data at the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, Aug 14, 2017
The document discusses domain modeling for personalized learning. It defines a domain model as representing domain knowledge through concepts and their relationships. Domain models serve as the basis for individual student models and for indexing and classifying learning content. They can be used to model student knowledge and decide on appropriate next steps for learning. The document describes different types of domain models, including vector, network, conceptual, and procedural models. It also discusses using ontologies and different aspects in domain modeling and applying domain models to student modeling, content indexing, and personalized guidance.
Two Brains are Better than One: User Control in Adaptive Information AccessPeter Brusilovsky
In recent years, the use of Artificial Intelligence (AI) technologies expanded to many areas where they directly affect the lives of many people. AI-based approaches advise human decision-makers who should be released on bail, whether it is a good time to discharge a patient from a hospital and whether a specific student is at risk to fail a course. Such an extensive use in AI in decision making came with a range of protentional problems that have been extensively studied over the last few years. Recognition of these problems motivated a rapid rise of research on “human-centered AI”, which attempted to address and minimize the negative effects of using AI technologies. Among the ideas of human-centered AI is user control - engaging users in affecting AI decision making to prevent possible errors and biases. In my talk, I will focus on the application of user control in one popular area of AI application, adaptive information access. Adaptive information access systems such as personalized search and recommender systems attempt to model their users to help them in finding the most relevant information. Yet, user modeling and personalization mechanisms might not always work as expected resulting in errors, biases, and suboptimal behavior. Combining the decision power or AI with the ability of the user to guide and control it brings together the strong sides of artificial and human intelligence and could lead to better results. In my talk, I review several projects focused on user control in adaptive information access systems and discuss the benefits and challenges of this approach.
Personalized Online Practice Systems for Learning ProgrammingPeter Brusilovsky
An adaptive learning system called Mastery Grids was created to increase student engagement with online educational content by incorporating personalized and social adaptive features. Mastery Grids uses open learner modeling to display a student's knowledge progress compared to their peers, adaptive navigation support to guide students to relevant activities, and concept-based recommendations of content. A study found that Mastery Grids significantly increased student success rates, time spent engaging with content, and learning compared to non-adaptive systems. Further research added direct recommendations to Mastery Grids and found they increased transparency and led to more efficient learning when explanations of recommendations were provided through the open learner model visualizations.
Data-Driven Education: Using Big Educational Data to Improve Teaching and Learning. Keynote slides for 15th International Conference on Web-Based Learning, ICWL 2016, Rome, Italy, October 26–29.
The Value of Social: Comparing Open Student Modeling and Open Social Student ...Peter Brusilovsky
Brusilovsky, P., Somyurek, S., Guerra, J., Hosseini, R., and Zadorozhny, V. (2015) The Value of Social: Comparing Open Student Modeling and Open Social Student Modeling. In: F. Ricci, K. Bontcheva, O. Conlan and S. Lawless (eds.) Proceedings of 23nd Conference on User Modeling, Adaptation and Personalization (UMAP 2015), Dublin, Ireland, , June 29 - July 3, 2015, Springer Verlag, pp. 44-55, also available at http://link.springer.com/chapter/10.1007/978-3-319-20267-9_4.
Learning analytics are more than a technologyDragan Gasevic
Learning analytics aim to optimize learning through measurement, collection, analysis and reporting of student data. While interest is growing, few institutions have fully adopted analytics. Challenges include a lack of data-informed culture, focusing on solutions over research, and privacy concerns. Fully realizing analytics potential requires multidisciplinary teams, addressing complex educational systems, and developing an analytics-focused culture.
This document proposes an ontological model for representing rubrics digitally using Semantic Web standards like RDF and OWL. Currently, most rubrics shared online are in static, non-machine readable formats like Word documents or proprietary learning management systems. The proposed model aims to make rubrics sharable and reusable across different systems on the web by representing them semantically. It discusses how rubrics benefit both students and teachers by providing clear evaluation criteria and allowing for consistent grading. However, existing rubrics online often lack specificity and are not in open, transferable formats between different tools and systems.
Mastery Grids: An Open Source Social Educational Progress VisualizationPeter Brusilovsky
Presentation for EC-TEL 2015 paper:
Loboda, T., Guerra, J., Hosseini, R., and Brusilovsky, P. (2014) Mastery Grids: An Open Source Social Educational Progress Visualization. In: S. de Freitas, C. Rensing, P. J. Muñoz Merino and T. Ley (eds.) Proceedings of 9th European Conference on Technology Enhanced Learning (EC-TEL 2014), Graz, Austria, September 16-19, 2014, pp. 235-248.
Exploring Tools for Promoting Teacher Efficacy with mLearning (mlearn 2014 Pr...Robert Power
Slides for my presentation with Dean Cristol and Belinda Gimbert of Ohio State University at mLearn 2014, November 4, 2014, at Kadir-Has University in Istanbul, Turkey.
Adaptive Navigation Support and Open Social Learner Modeling for PALPeter Brusilovsky
This presentation is an overview of Open Social Learner Modeling project. It presents Mastery Grids interface, distributed personalized learning architecture Aggregate, and smart content for Java, Python, and SQL
This document summarizes a presentation on learning analytics given by Simon Buckingham Shum. Some key points:
- Learning analytics aims to unlock student data to improve 21st century learning by analyzing patterns in data to better understand learning processes and identify students who may need help.
- Examples discussed include Purdue University's predictive model that identified 66-80% of struggling students and a system that provides real-time feedback to students.
- Analytics can look beyond grades and course performance to capture data on learning dispositions, engagement, curriculum mastery, and student discourse to provide a more holistic view of the learning process.
- Challenges include ensuring analytics are used ethically and to improve learning rather than
Learning with me Mate: Analytics of Social Networks in Higher EducationDragan Gasevic
Effects of social interactions are reported in research on higher education to lead to positive outcomes such as higher levels of internalization, sense of community, academic achievement, metacognition, and student retention. The role of social networks has especially been emphasized in research due to the availability of theoretical foundations and analytic methods to investigate their effects in higher education. The increased use of technologies in education allows for the collection of large and rich datasets about social networks which call for the use of novel analytics methods. This talk will first give a brief overview of the existing work on and lessons learned from some well-known studies on social networks in higher education in diverse situations from face-to-face to massive open online courses. The talk will then identify critical challenges that require immediate attention in order for the study of social networks to make a sustainable impact on learning and teaching. The most important take away from the talk will be that
- computational aspects of the study of social networks need to be integrated deeply with theory, research and practice,
- novel methods for the study of critical dimensions (discourse, structure and dynamics) that shape network formation and network effects are necessary, and
- innovative instructional approaches are essential to address the changing conditions created by contemporary educational and technological contexts.
Tools and Evaluation Techniques to Support Social Awareness in CSCeL: The AVA...EuroCAT CSCL
1. The document describes a 7-step process for designing participation avatars to increase engagement in an online teacher training program in Greece.
2. User evaluations found that calculating participation levels based on the highest posting student and using varied colors improved understanding and motivation.
3. A final study showed the avatars improved active participation and satisfaction with the program compared to the control group without avatars.
This document discusses the evaluation of TOIA, a free online assessment tool. It aimed to test the functionality of TOIA, identify usability issues, and understand how it would be used. The evaluation found that TOIA was easy to use and provided a comprehensive set of assessment tools. However, users noted a lack of question types and concerns about long-term maintenance as a free software. Overall the evaluation helped improve TOIA and provided insights into effective online assessment.
Empirical studies of adaptive annotation in the educational context have demonstrated that it can help students to acquire knowledge faster, improve learning outcomes, reduce navigational overhead, and encourage non-sequential navigation. Over the last 8 years we have explored a lesser known effect of adaptive annotation – its ability to significantly increase student engagement in working with non-mandatory educational content. In the presence of adaptive link annotation, students tend to access significantly more learning content; they stay with it longer, return to it more often and explore a wider variety of learning resources. This talk will present an overview of our exploration of the addictive links effect in many course-long studies, which we ran in several domains (C, SQL and Java programming), for several types of learning content (quizzes, problems, interactive examples). The first part of the talk will review our exploration of a more traditional knowledge-based personalization approach and the second part will focus on more recent studies of social navigation and open social student modeling
Personalized Learning: Expanding the Social Impact of AIPeter Brusilovsky
Slide of my keynote talk at SIAIA '23 workshop held at AAAI 2023:
The use of AI in Education could be traced to the early days of AI. While the publicity associated with the most recent wave of AI applications rarely mentions education, it is through the improvement in education AI could achieve an impressive social impact. In particular, the AI ability to personalize the learning process could make a large difference in a context where learners' knowledge could be radically different from learner to learner. Modern computer and internet technologies can now bring the power of learning in the forms of MOOCs, online textbooks, and zoom courses truly worldwide. Yet, without personalization, the potential of these technologies is not fully leveraged. In this talk, I will review several generations of research on personalized learning and discuss tools, technologies, and infrastructures for personalized learning that we are currently exploring.
CHECO Retreat - Changing landscape of teachingJeff Loats
Dr. Jeff Loats presented on blended learning initiatives and evidence-based teaching techniques involving technology. He discussed the blended learning initiative at MSU Denver which focuses on introductory courses and provides sustained support for instructors. Three key techniques were covered: Just-in-Time Teaching using pre-class assignments, classroom response systems like clickers, and flipped teaching with videos assigned as homework. The presentation emphasized combining techniques and adopting practices supported by research evidence to improve student learning over traditional lecture-based methods.
This document discusses issue-based metrics and self-reported metrics for measuring user experience. It describes issue-based metrics as involving qualitative data about usability issues identified during user studies, including severity ratings of issues. Self-reported metrics involve subjective data collected through questionnaires and interviews using rating scales, the System Usability Scale, and other methods. Key considerations for both include identifying and analyzing patterns in issues and responses to focus design improvements.
This document describes KnowledgeZoom, a concept-based exam study tool with zoomable open student modeling to help students efficiently prepare for final exams. It integrates fine-grained concept-based guidance, an open student model that progressively zooms between topics and sub-topics, and adaptive problem sequencing. The tool includes a Knowledge Explorer that visualizes students' knowledge gaps through a zoomable treemap, and a Knowledge Maximizer that sequences practice problems based on students' knowledge levels and impact. An evaluation with 14 students found the tool helped identify knowledge weaknesses and generated quizzes covering multiple concepts, though better integration between its components could further improve the student experience.
Classsourcing: Crowd-Based Validation of Question-Answer Learning Objects @ I...Jakub Šimko
A simple approach for assessing answer validity information from a student crowd in an online learning scenario context. Raises the questions about using of the student crowds for enhancing learning content and online student collaboration.
KnowledgeZoom for Java: A Concept-Based Exam Study Tool Michelle Liang
KnowledgeZoom is a concept-based exam study tool that integrates fine-grained concept guidance, zoomable open student modeling, and adaptive problem sequencing to help students efficiently prepare for final exams. It includes the Knowledge Explorer, which visualizes students' knowledge of Java concepts in a zoomable treemap, and the Knowledge Maximizer, which sequences practice problems based on students' knowledge levels. An evaluation with 14 students found they attempted more questions and achieved higher success rates with KnowledgeZoom than other tools, and most found its identification of weak concepts and coverage of multiple concepts to be helpful for exam preparation. Future work will improve the integration between the tools and further investigate their effects on student learning.
Interactions of reading and assessment activitiesSergey Sosnovsky
Reading and assessment are elementary activities for knowl- edge acquisition in online learning. Assessments represented as quizzes can help learners to identify gaps in their knowledge and understanding, which they can then overcome by reading the corresponding text-based course material. Reversely, quizzes can be used to evaluate reading com- prehension. The predominantly self-regulated interaction of reading and quiz activities in learning systems used in higher education has been little studied. In this paper, we examine this interaction using scroll and log data from an online undergraduate course (N=142). By analyzing pro- cesses and sequential patterns in user sessions, we identified six session clusters for characteristic reading and quiz patterns potentially relevant for adaptive learning support. These clusters showed that individual user sessions included either mainly reading or quizzes, but rarely both.
IUI2017 SmartLearn keynote: Intelligent Interfaces for Open Social Student M...Peter Brusilovsky
In this talk I will introduce the emerging technology of
Open Social Student Modeling (OSSM) and review several
projects performed in our research lab to investigate the
potential of OSSM.
OSSM is a recent extension of Open Student Modeling
(OSM), a popular technology in the area of personalized
learning systems. While in traditional personalized systems,
student models were hidden “under the hood” and used to
personalize the educational process; open student modeling
introduced the ability to view and modify the state of
students’ own knowledge to support reflection, selforganized
learning, and system transparency. Open Social
Student Modeling takes this idea one step further by
allowing students to explore each other’s models or an
aggregated model of the class. The idea to make OSM
social was originally suggested and explored by Bull [1; 2].
Over the last few years, our team explored several
approaches to present OSSM in a highly visual form and
evaluated these approaches in a sequence of classroom and
lab studies. I will present a summary of this work
introducing such systems as QuizMap [3], Progressor [4],
and Mastery Grids [5] and reviewing most interesting
research evidence collected by the studies.
VII Jornadas eMadrid "Education in exponential times". Erkan Er: "Predicting ...eMadrid network
VII Jornadas eMadrid "Education in exponential times". Erkan Er: "Predicting Peer-Review Participation at Large Scale Using an Ensemble Learning Method". 04/07/2017.
The document discusses online assessment in education. It defines formative and summative assessment and their purposes. It describes various formative assessment techniques that can be used online like polls, discussions, blogs, e-portfolios, and rubrics. It also discusses principles of effective online assessment and types of assessments like tests and quizzes that can be created using tools like Hot Potatoes and Google Forms. Summative assessment is used to evaluate learning while formative assessment provides feedback to improve learning.
This document outlines a workshop on assessment and feedback approaches. It discusses challenges with current assessment practices, such as an over-reliance on summative assessments and a lack of formative feedback. It then presents the TESTA (Transforming the Experience of Students Through Assessment) approach, which aims to address these issues by taking a whole-program approach to balancing summative and low-stakes formative assessments and improving feedback practices. The workshop involves examining assessment data, discussing challenges, and learning TESTA principles for improving assessment design and student experience.
Learning-Based Evaluation of Visual Analytic Systems.BELIV Workshop
This document proposes a "learning-based evaluation" method for assessing visual analytics systems. The goal is to directly measure how much knowledge users gain from interacting with a system. It involves giving users a new but related task after use and testing their proficiency. This allows evaluating if the system helped users learn about the interface, data or problem domain. The document outlines applying this to the VAST Challenge and integrating it with other evaluation methods. It argues this approach matches what clients like agencies need to see to adopt systems.
This document discusses instructional design and planning, specifically regarding systems theory and its relation to instructional systems. It covers key concepts of systems theory including that components respond when manipulated and that there is interdependence and feedback. An instructional process is viewed as a system with the purpose of bringing about learning. If a learner's performance is unsatisfactory after instruction, changes are needed to provide more effective instruction. Instructional design is defined as the process of solving instructional problems through systematic analysis of learning conditions. Methods for selecting methods, media, and materials in instructional design are discussed, including choosing instructional strategies and creating new instructional materials by considering learning objectives and putting oneself in the learner's shoes.
This document discusses classroom assessment techniques (CATs), which are simple, non-graded activities used to provide instructors feedback on student learning. It provides examples of CATs like the Muddiest Point and Minute Paper. The document emphasizes that CATs help instructors improve their teaching by gaining insights into student understanding. It also notes that CATs benefit students by encouraging self-assessment and engagement. Overall, the document promotes the use of CATs to enhance the teaching and learning process through frequent formative assessment.
This document defines and discusses different types of educational software: drill-and-practice, tutorials, simulations, instructional games, and problem-solving software. It provides examples of each type, along with their benefits and limitations. Guidelines are suggested for effective use of each type in the classroom to support learning.
This document summarizes research on adaptive sequencing in open social student modeling. It describes how combining knowledge-based guidance with social guidance can encourage non-sequential navigation, increase learning speed for strong students, and positively relate to student performance. A classroom study found that adaptive sequencing increased learning speed and the odds of correct problem solving. Students also provided positive subjective feedback about recommendations. Future work aims to explore alternative visualization and awareness techniques.
Similar to Addictive links: Adaptive Navigation Support in College-Level Courses (20)
This document contains a transcript of the doctoral defense of Isabella M. Venter from the University of the Western Cape, South Africa. During the defense, the candidate was asked questions by the committee about their research topic, literature review, research design, methodology, and knowledge contribution. The research examined how participatory design and digital storytelling could be used to create an HIV/AIDS education environment in Sub-Saharan Africa. The committee inquired about how the research differed from existing literature, why not all key concepts were reviewed, and how design science research was applied through multiple contexts in East and South Africa.
This document summarizes a study on developing skills for designing mobile interfaces for diverse users through a situated learning assignment. Students interviewed senior mobile phone users and assisted them with a function to understand their needs. They analyzed the results through frameworks like Habermas' knowledge interests. The assignment helped students improve communication, teamwork, research, and consideration of diverse perspectives. The multi-year study yielded publications and showed promise for preparing students for real-world challenges through similar experiential learning projects.
This document discusses the potential for "Big Theology" using big data approaches to analyze and understand theological data on a large scale. It begins by defining big data and explaining how digital theology analyzes theologically relevant data through digital platforms and tools. The document then proposes that big theology could represent, store, retrieve, analyze, predict, and visualize theological phenomena based on a global collection of digital religious expressions and narratives. Finally, it provides examples of potential applications of big theology, such as a mobile catechism, using data to promote peace, technologies for sacred spaces, and assisting church leadership. It concludes by discussing challenges and opportunities for international collaboration on big theology projects.
This document discusses the successes and challenges of using information and communication technologies (ICTs) in Africa to promote development. Some key successes highlighted include the rapid expansion of mobile phone access across Africa, the transformative impact of M-Pesa mobile banking in Kenya, and the use of mobile phones to support activism campaigns. However, the document also notes challenges such as ICTs only being part of the solution and not addressing other needs like capacity building; institutional contexts sometimes constraining what can be achieved; and systems not benefiting the poorest groups. The implications discussed for the ICT4D community include the need for multidisciplinarity, having a strategic agenda, and adopting strong ethical norms.
Presentation given by Dr. Jarkko Suhonen at the launch event of the IMPDET-LE doctoral training hub at College of Business Education, Tanzania, June 25, 2014
The document summarizes a symposium on the role of information and communication technologies (ICTs) in addressing societal challenges from an education perspective in the context of European Union and Latin American collaboration. It provides details on two presentations: one on 21st century education and the role of ICT, and another on using ICT to make a difference in education. The objectives of the symposium are outlined as promoting understanding of education challenges and solutions, practical implementation of technology tools for sustainable development, and creating opportunities for knowledge sharing through EU-LAC collaboration.
Presentation slides of Dr. Jarkko Suhonen at X International Interdisciplinary Scientific Research Congress (X CIC), June 12-14, 2014, Santo Domingo, Dominican Republic.
The document discusses a project to develop a research culture at the College of Business Education in Tanzania through establishing a PhD school. The project aims to provide on-site PhD training to CBE faculty to form a research group focused on areas like education technology. This is intended to address the lack of research culture and outputs at Tanzanian universities by developing local capacity rather than sending faculty abroad. The expected results include a functional research group undertaking relevant projects and increasing academic and industry connections both locally and internationally.
This document contains questions posed to Erkki Sutinen regarding his research on engaging HE/UH teachers in technology. The questions probe Sutinen's key claims, theoretical framework, methodology, findings on teacher learning, views on organizational change and innovation, context of the University of Helsinki, and visions for the future of educational technology. Sutinen is asked to reflect on the challenges of his research process and next steps in studying how technology can help higher education institutions cope with ongoing changes.
This document provides information about the IMPDET-LE Preparatory Workshop hosted by Jarkko Suhonen from the University of Eastern Finland's edTechΔ research group. The edTechΔ group was established in the early 2000s and focuses on several areas of research including open and distance learning, educational games, sentiment analysis, and more. The workshop discussed the main research interests of the group which include innovations in education technology, digital storytelling, and learning analytics. It also outlined the study requirements for a doctoral dissertation through the edTechΔ group, including publishing papers and completing coursework.
This document summarizes the questions and observations from the public defense of Ms. Yue Dai's doctoral dissertation. The opponent, Dr. Heikki Topi, asks questions about the research context, goals, epistemology, design science research approach, validity of results, evaluation methods, data sources, usability, theoretical contributions, implementation of models, and outcomes. Topics include the disciplines the work contributes to, challenges of multidisciplinary research, how goals were addressed, the influence of design science literature, validity concerns, lessons learned between models, and plans for further testing and application of the models in practice.
This document summarizes the edTechΔ Research Group at the University of Eastern Finland. It notes that the group has 5 senior researchers and over 20 PhD students. It has received over 8 million euros in external funding since 2001 and publishes an average of 23 peer-reviewed publications per year. The research areas of the group include open and distance learning, educational games, computer science education, and more. Specific projects mentioned include SURA YA UKMWI on HIV/AIDS education in Uganda, the Jeliot mobile games, and tools for special education children through the CASCATE project.
This document summarizes the research focus areas and activities of the School of Computing at the University of Eastern Finland (UEF). It discusses four main areas of focus: 1) educational technology, 2) interactive technologies, 3) natural language processing and text analysis, and 4) inclusive technologies for specific user groups. For each area, it provides 1-2 sentences about the lead researchers and example projects. It also summarizes outreach activities including an annual science festival and technology clubs for children.
This document summarizes a research project called CASCATE that studied technologies to support children with autism spectrum disorders. The project involved 4 children with autism who participated in weekly hour-long "action group" sessions over 2 years using 4 technology-based workstations. The sessions focused on creativity, strengths, modifiable technology, and transforming the children's role from passive users to active creators. The goals were to include these children in the research, focus on their strengths rather than disabilities, and develop a flexible technology-enhanced learning environment tailored to each child.
Presentation from the Digital Storytelling, Crowdsourcing and Science Education for International Research and Training symposium, August 2013, School of Computing, University of Eastern Finland.
The document describes a research project aimed at improving the usability of enterprise resource planning (ERP) systems through increased collaboration between users and the system. The researchers apply principles of human-computer collaboration theory to the design and evaluation of ERP user interfaces. They conducted field studies to identify usability issues and developed design principles to guide more collaborative interface design. Prototypes were created and evaluated in lab studies. The goal is to make ERP systems more usable by strengthening their collaborative capabilities.
Behzad Joze Hashemian's presentation Digital Storytelling, Crowdsourcing and Science Education for International Research and Training -symposium, August 29, 2013.
Jaakko Helminen's presentation in the Digital Storytelling, Crowdsourcing and Science Education for International Research and Training -symposium, August 29, 2013.
More from University of Eastern Finland, IMPDET-LE (20)
A workshop hosted by the South African Journal of Science aimed at postgraduate students and early career researchers with little or no experience in writing and publishing journal articles.
This presentation includes basic of PCOS their pathology and treatment and also Ayurveda correlation of PCOS and Ayurvedic line of treatment mentioned in classics.
How to Manage Your Lost Opportunities in Odoo 17 CRMCeline George
Odoo 17 CRM allows us to track why we lose sales opportunities with "Lost Reasons." This helps analyze our sales process and identify areas for improvement. Here's how to configure lost reasons in Odoo 17 CRM
বাংলাদেশের অর্থনৈতিক সমীক্ষা ২০২৪ [Bangladesh Economic Review 2024 Bangla.pdf] কম্পিউটার , ট্যাব ও স্মার্ট ফোন ভার্সন সহ সম্পূর্ণ বাংলা ই-বুক বা pdf বই " সুচিপত্র ...বুকমার্ক মেনু 🔖 ও হাইপার লিংক মেনু 📝👆 যুক্ত ..
আমাদের সবার জন্য খুব খুব গুরুত্বপূর্ণ একটি বই ..বিসিএস, ব্যাংক, ইউনিভার্সিটি ভর্তি ও যে কোন প্রতিযোগিতা মূলক পরীক্ষার জন্য এর খুব ইম্পরট্যান্ট একটি বিষয় ...তাছাড়া বাংলাদেশের সাম্প্রতিক যে কোন ডাটা বা তথ্য এই বইতে পাবেন ...
তাই একজন নাগরিক হিসাবে এই তথ্য গুলো আপনার জানা প্রয়োজন ...।
বিসিএস ও ব্যাংক এর লিখিত পরীক্ষা ...+এছাড়া মাধ্যমিক ও উচ্চমাধ্যমিকের স্টুডেন্টদের জন্য অনেক কাজে আসবে ...
Executive Directors Chat Leveraging AI for Diversity, Equity, and InclusionTechSoup
Let’s explore the intersection of technology and equity in the final session of our DEI series. Discover how AI tools, like ChatGPT, can be used to support and enhance your nonprofit's DEI initiatives. Participants will gain insights into practical AI applications and get tips for leveraging technology to advance their DEI goals.
it describes the bony anatomy including the femoral head , acetabulum, labrum . also discusses the capsule , ligaments . muscle that act on the hip joint and the range of motion are outlined. factors affecting hip joint stability and weight transmission through the joint are summarized.
A Strategic Approach: GenAI in EducationPeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
Strategies for Effective Upskilling is a presentation by Chinwendu Peace in a Your Skill Boost Masterclass organisation by the Excellence Foundation for South Sudan on 08th and 09th June 2024 from 1 PM to 3 PM on each day.
Pride Month Slides 2024 David Douglas School District
Addictive links: Adaptive Navigation Support in College-Level Courses
1. Addictive Links:
Adaptive Navigation Support
for College-Level Courses
Peter Brusilovsky with:
Sergey Sosnovsky, Michael Yudelson, Sharon Hsiao
School of Information Sciences,
University of Pittsburgh
2. User Model
Collects information
about individual user
Provides
adaptation effect
Adaptive
System
User Modeling side
Adaptation side
User-Adaptive Systems
Classic loop user modeling - adaptation in adaptive systems
3. Adaptive Software Systems
• Intelligent Tutoring Systems
– adaptive course sequencing
– adaptive . . .
• Adaptive Hypermedia Systems
– adaptive presentation
– adaptive navigation support
• Adaptive Help Systems
• Adaptive . . .
4. Adaptive Hypermedia
• Hypermedia systems = Pages + Links
• Adaptive presentation
– content adaptation
• Adaptive navigation support
– link adaptation
5. • Direct guidance
• Hiding, restricting, disabling
• Generation
• Ordering
• Annotation
• Map adaptation
Adaptive Navigation Support
7. Adaptive Link Annotation: InterBook
1. Concept role
2. Current concept state
3. Current section state
4. Linked sections state
4
3
2
1
√"
Metadata-based mechanism
11. The Value of ANS
• Lower navigation overhead
– Access the content at the right time
– Find relevant information faster
• Better learning outcomes
– Achieve the same level of knowledge faster
– Better results with fixed time
• Encourages non-sequential navigation
12. The Case of QuizPACK
• QuizPACK: Quizzes for
Parameterized Assessment of
C Knowledge
• Each question is a pattern of a
simple C program. When it is
delivered to a student the
special parameter is
dynamically instantiated by a
random value within the pre-
assigned borders.
• Used mostly as a self-
assessment tool in two C-
programming courses
13. QuizPACK: Value and Problems
• Good news:
– activity with QuizPACK significantly correlated with
student performance in classroom quizzes
– Knowledge gain rose from 1.94 to 5.37
• But:
– Low success rate - below 40%
– The system is under-used (used less than it deserves)
• Less than 10 sessions at average
• Average Course Coverage below 40%
14. Adding Motivation (2003)
• Students need some better motivation to work with non-
mandatory educational content…
• Added classroom quizzes:
– Five randomly initialized questions out of 20-30 questions
assigned each week
• Good results - activity, percentage of active questions,
course coverage - all increased 2-3 times! But still not as
much as we want. Could we do better?
• Maybe students bump into wrong questions? Too easy?
Too complicated? Discouraging…
• Let’s try something that worked in the past
15. Questions of
the current
quiz, served
by QuizPACK
List of annotated
links to all quizzes
available for a
student in the
current course
Refresh
and help
icons
QuizGuide = QuizPACK+ANS
16. Demo: QuizPack
• KT Portal - http://adapt2.sis.pitt.edu/cbum/
• Create your account or use test / test
18. QuizGuide: Adaptive Annotations
• Target-arrow abstraction:
– Number of arrows – level of
knowledge for the specific
topic (from 0 to 3).
Individual, event-based
adaptation.
– Color Intensity – learning
goal (current, prerequisite
for current, not-relevant,
not-ready). Group, time-
based adaptation.
n Topic–quiz organization:
20. QuizGuide: Motivation
• Adaptive navigation support increased student's
activity and persistence of using the system
Average activity
0
50
100
150
200
250
300
2002 2003 2004
Average num. of
sessions
0
5
10
15
20
2002 2003 2004
Average course
coverage
0%
10%
20%
30%
40%
50%
60%
2002 2003 2004
Active students
0%
20%
40%
60%
80%
100%
2002 2003 2004
n Within the same class QuizGuide session were much longer than
QuizPACK sessions: 24 vs. 14 question attempts at average.
n Average Knowledge Gain for the class rose from 5.1 to 6.5
21. QuizGuide: Success Rate?
n Well, that worked too, but
the scale is not comparable
n One-way ANOVA shows
that mean success value for
QuizGuide is significantly
larger then the one for
QuizPACK:
F(1, 43) = 5.07
(p-value = 0.03).
22. A new value of ANS?
• The scale of the effect is too large…
May be just a good luck?
• New effect after 15 years of research?
• Not quite new, rather ignored and
forgotten - ELM-ART data
24. Round 2: Let’s Try it Again…
• Maybe the effect could only be discovered in
full-scale classroom studies – while past studies
were lab-based?
• Another study with the same system
– QuizGuide+QuizPACK vs. QuizPACK
• A study with another system using similar kinds
of adaptive navigation support
– NavEx+WebEx vs. WebEx
• NavEx - a value-added ANS front-end for
WebEx - interactive example exploration system
26. Concept-based student modeling
Example 2
Example M
Example 1
Problem 1
Problem 2
Problem K
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N
Examples
Problems
Concepts
28. Does it work?
• The increase of the amount of work for the
course
Clicks - Overall
0
50
100
150
200
250
300
Non-adaptive Adaptive
Examples
Quizzes
Lectures - Overall
0
2
4
6
8
10
12
Non-adaptive Adaptive
Examples
Quizzes
Learning Objects - Overall
0
5
10
15
20
25
30
Non-adaptive Adaptive
Examples
Quizzes
29. Is It Really Addictive?
• Are they coming more often? Mostly, but there
is no stable effect
• But when they come, they stay… like with an
addictive game
Clicks - Per Session
0
5
10
15
20
Non-adaptive Adaptive
Examples
Quizzes
Learning Objects - Per
Session
0
1
2
3
4
Non-adaptive Adaptive
Examples
Quizzes
30. Why It Is Working?
• Progress-based annotation
– Displays the progress achieved so far
– Does it work as a reward mechanism?
– Open Student Modeling
• State-based annotation
– Not useful, ready, not ready
– Access activities in the right time
– Appropriate difficulty, keep motivation
32. The Diversity of Work
• C-Ratio: Measures the breadth of exploration
• Goal distance: Measures the depth
Self-motivated Work - C-Ratio
(%)
0
0.2
0.4
0.6
Non-adaptive Adaptive
Quizzes
Examples
Self-motivated Work - Goal
Distance (LO's)
0
5
10
15
20
Non-adaptive Adaptive
Quizzes
Examples
33. Round 3: Trying another domain…
• Is it something relevant to C programming or to
simple kind of content?
• New changes:
– SQL Programming instead of C
– Programming problems (code writing) instead of
questions (code evaluation)
– Comparison of concept-based and topic-based
mechanisms in the same domain and with the same
kind of content
34. • SQL-KnoT delivers online SQL problems, checks student’s
answers and provides a corrective feedback
• Every problem is dynamically generated using a template
and a set of
databases
• All problems have
been assigned to 1
of the course
topics and
indexed with
concepts from the
SQL ontology
SQL Knowledge Tester
35. • To investigate possible influence of concept-based
adaptation in the present of topic-based adaptation we
developed two versions of QuizGuide:
Topic-based Topic-based+Concept-Based
Concept-based vs Topic-based ANS
36. • Two Database Courses (Fall 2007):
§ Undergraduate (36 students)
§ Graduate (38 students)
• Each course divided into two groups:
§ Topic-based navigation
§ Topic-based + Concept-Based Navigation
• All students had access to the same set of SQL-
KnoT problems available in adaptive
(QuizGuide) and in non-adaptive mode (Portal)
Study Design
37. • Total number of attempts made by all students:
in adaptive mode (4081), in non-adaptive mode (1218)
• Students in general were much more willing to access
the adaptive version of the system, explored more
content with it and to stayed with it longer:
Questions
0
25
50
75
100
Quizzes
0
5
10
15
20
25
Topics
0
1
2
3
4
5
6
Sessions
0
1
2
3
4
5 Session Length
0
5
10
15
20
25
Adaptive
Non-adaptive
It works again! Like magic…
38. • Did concept-based adaptation increase the
magnitude of the motivational effect?
§ No significant difference in the average numbers of attempts,
problems answered, topics explored
§ No significant difference in the session length
• Was there any other observable difference?
§ Average number of attempts per question
§ Resulting Knowledge Level
(averaged across all concepts)
Attempts per
question
0
2
4
6
Knowledge level
0
0.2
0.4
0.6
Combined
Topic-based
Concept-based ANS: Added Value?
42. • Difference in the ratio of Repetition1 pattern
explains:
§ difference in the average number of attempts per question
§ difference in the cumulative resulting knowledge level
§ Students repeat the same question again and
again:
§ They “get addicted“ to the concept-based icons
§ Is it a good thing for us?
− YES – they react to the navigational cues, they work more
− NO – we expect them to concentrate on those questions where they have
smaller progress instead of drilling in the same question
Discussion
43. Round 4: The Issue of Complexity
• Let’s now try it for Java…
• What is the research goal?
• Java is a more sophisticated domain than C
– OOP versus Procedural
– Higher complexity
• Will it work for complex
questions?
• Will it work similarly? 0% 20% 40% 60% 80% 100%
C
Java
language complexity
Easy
Moderate
Hard
47. !! !!
JavaGuide
(Fall 2008)
QuizJET
(Spring 2008)
!! parameters (n=22) (n=31)
Overall User
Statistics
Attempts 125.50 41.71
Success Rate 58.31% 42.63%
Distinct Topics 11.77 4.94
Distinct Questions 46.18 17.23
Average
User Session
Statistics
Attempts 30.34 21.50
Distinct Topics 2.85 2.55
Distinct Questions 11.16 8.88
Magic… Here We Go Again!
48. • Significantly more Attempts
on the easy questions in
JavaGuide than in QuizJET
• F(1, 153) = 7.081, p = .009, partial η2 = .044
The Effect Depends on Complexity
49. • Significant higher
Success Rate
• F(1, 153) = 40.593, p .001, partial η2 = .210
• On average 2.5 times
more likely to answer a
quiz correctly with
adaptive navigation
support
68.73%
67.00%
39.32%38.00%
28.23%
11.90%
0%
20%
40%
60%
80%
100%
Easy Moderate Hard
SuccessRate
Complexity Level
Success Rate
JavaGuide
QuizJET
Different Pattern For Success Rate
50. 0.94
0.61
0.29
1.85
1.01
0.44
0
0.5
1
1.5
2
Easy Moderate Hard
Attempts/Question
Complexity Level
Average Attempts per
Question
68.73%
67.00%
39.32%38.00%
28.23%
11.90%
0%
20%
40%
60%
80%
100%
Easy Moderate Hard
SuccessRate
Complexity Level
Success Rate
JavaGuide
QuizJET
Prerequisite-based guidance prepared students to attempt complex questions after exploring easier ones
Putting it Together…
51. Round 5: Social Navigation
• Concept-based and topic-based navigation support
work well to increase success and motivation
• Knowledge-based approaches require some
knowledge engineering – concept/topic models,
prerequisites, time schedule
• In our past work we learned that social navigation –
guidance extracted from the work of a community of
learners – might replace knowledge-based guidance
• Social wisdom vs. knowledge engineering
52. Open Social Student Modeling
• Key ideas
– Assume simple topic-based design
– No prerequsites or concept modeling
– Show topic- and content- level knowledge progress of
a student in contrast to the same progress of the class
• Main challenge
– How to design the interface to show student and class
progress over topics?
– We went through several attempts
56. Class vs. Peers
• Peer progress was important, students
frequently accessed content using peer models
• The more the students compared to their peers,
the higher post-quiz scores they received (r=
0.34 p=0.004)
• Parallel IV didn’t allow to recognized good peers
before opening the model
• Progressor added clear peer progress
58. Why It Is Important?
• Many systems demonstrated their educational
effectiveness in a lab-like settings: once the students
are pushed to use it - it benefits their learning
• However, once released to real classes, these systems
are under-used - most of them offer additional non-
mandatory learning opportunities
• “Students are only interested in points and grades”
• Convert all tools into credit-bearing activities?
• Or use alternative approaches to increase motivation
62. Take-home messages
• A combination of progress-based and state-
based adaptive link annotation increases the
amount and the diversity of student work with
non-mandatory educational content
• The effect is stable and the scale of it is quite
large
• Properly organized Social Navigation might be
at least as successful as the knowledge-based
• Requires a long-term classroom study to
observe
63. What we are doing now?
• Exploring new generation of open social
modeling tools in wide variety if classes and
domains from US to Nigeria
– Interested to be a pilot site?
• Exploring more advanced guidance and
modeling approaches based on large volume of
social data
• Applying open social modeling to motivate
readings
64. Acknowledgements
• Joint work with
– Sergey Sosnovsky
– Michael Yudelson
– Sharon Hsiao
• Pitt “Innovation in Education” grant
• NSF Grants
– EHR 0310576
– IIS 0426021
– CAREER 0447083
65. Try It!
• http://adapt2.sis.pitt.edu/kt/
• Brusilovsky, P., Sosnovsky, S., and Yudelson, M. (2009)
Addictive links: The motivational value of adaptive link annotation.
New Review of Hypermedia and Multimedia 15 (1), 97-118.
• Hsiao, I.-H., Sosnovsky, S., and Brusilovsky, P. (2010) Guiding
students to the right questions: adaptive navigation support in an E-
Learning system for Java programming. Journal of Computer Assisted
Learning 26 (4), 270-283.
• Hsiao, I.-H., Bakalov, F., Brusilovsky, P., and König-Ries, B. (2013)
Progressor: social navigation support through open social student
modeling. New Review of Hypermedia and Multimedia [PDF]
Read About It!