SlideShare a Scribd company logo
1 of 17
Download to read offline
The RISE Framework: Using learning analytics
for the continuous improvement of open
educational resources
Robert Bodily, Rob Nyland, and David Wiley
Project Background
◈ Bob and Rob were members of John Hilton’s OER Research
Class at BYU
◈ In the class we were using the COUP framework for OER
Research
◆ Cost
◆ Outcomes
◆ Use
◆ Perceptions
◈ We wanted to understand what learning analytics could bring to
the conversation around OER
The 5 R’s of OER
Retain
Reuse
Revise
Remix
Redistribute
“Because the Open Curriculum is licensed in such a way that we can
revise materials directly, OHSU is able to engage in a highly
data-driven curriculum improvement process. As students go
through the online courses, their clicks and learning paths can be
tracked and recorded. This information can be combined with item
response theory and learning outcome analysis data to set priorities
for curriculum or assessment revision empirically. This data-driven
process of curriculum improvement should allow the Open
Curriculum to reach a very high level of quality very quickly.”
- David Wiley, 2009
Continuous Improvement
Learning
Analytics
OER
Data-Driven
Continuous
Improvement
Continuous Improvement at OLI
“A key attribute of the OLI
environment is that while students
are working through the course, we
are collecting analytics data and
using those data to drive multiple
feedback loops”
-Strader & Thille (2012)
Resource Inspection, Selection, and Enhancement
(RISE)
Inspection: Analyze course resources using a variety of automated
methods
Selection: Identify resources that need additional evaluation
Enhancement: Facilitate enhancement through an automated
inspection and selection process (enhancement is usually done
manually by an instructional designer).
Four Quadrant Evaluation Structure
High
Grades
High student prior knowledge,
inherently easy learning outcome,
highly effective content, poorly written
assessment
Effective resources, effective
assessment, strong outcome
alignment
Low
Grades
Low motivation or high life distraction of
students, too much material, technical
or other difficulties accessing resources
Poorly designed resources,
poorly written assessments,
poor outcome alignment,
difficult learning outcome
Low Use High Use
Our Approach: Case Study
◈ Fall 2015 Introduction to Business Course (N = 1002)
◈ OER courseware platform
◆ OER online text
◆ Videos
◆ Simulations
◆ Formative assessments
◆ Practice assessments
◆ Summative assessments
◈ Assessments and resources are all linked to course outcomes
Data Tracking
Resource Use Data
◈ Tracked with Google Analytics
◈ Defined as pageviews (other options include time spent and ratings)
Assessment Data
◈ Tracked with Open Assessments
◈ Defined as grade on summative assessments
Types of Analyses
1. Outer quadrant analysis
2. Page type analysis
3. Extreme outlier analysis
Outer Quadrant Analysis
◈ Blue dotted line is the
additive z-score
threshold.
◈ Black lines are the
median values for
pageviews and grades
Outer Quadrant Analysis
Page Title Grade PV/student A.Z-Score Quadrant
Reading: The Four Ps of Marketing 0.81 1.64 3.65 1
Outcome: Recruiting Employees 0.88 0.53 3.4 2
Video: Legal Rights Under Implied
Warranties
0.53 0.57 3.6 3
Reading: The Organization Chart and
Reporting Structure
0.64 1.89 4.56 4
Page Type Analysis
Reading Pages Self-Check Pages Outcome Pages
Video Pages
Simulation
Pages
◈ External search traffic: 1500 pageviews
◈ Course search traffic: 1626 pageviews
◈ Once external search traffic was removed, it
was no longer an outlier.
Extreme Outlier Analysis
Page Title Grade PV/student A.Z-Score
Reading: Stages of an
Economy
0.83 2.82 7.79
Enhancement
◈ Where the continuous improvement “rubber meets the road”
◈ Start with a theory of the domain
◈ Examine assessments
◆ Evaluate student response patterns / IRT / CFA
◆ Review assessments for design problems (negative language)
◆ Redesign or replace assessments
◈ Move on to content
◆ Examine content for assessment alignment (outcome, words)
◆ Examine content for micro-design problems (mismatch)
◆ Redesign or replace content
Questions?

More Related Content

What's hot

Tips for Assessing Student Learning Using Blackboard
Tips for Assessing Student Learning Using BlackboardTips for Assessing Student Learning Using Blackboard
Tips for Assessing Student Learning Using BlackboardJason Rhode
 
Institutional roll out of submission and marking ss
Institutional roll out of submission and marking ssInstitutional roll out of submission and marking ss
Institutional roll out of submission and marking ssLTatUoN
 
Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...
Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...
Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...Blackboard APAC
 
iGeneration Conference
iGeneration ConferenceiGeneration Conference
iGeneration Conferencenix1
 
Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...
Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...
Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...Blackboard APAC
 
Gavin henrick learning_analytics-workshop_questions
Gavin henrick learning_analytics-workshop_questionsGavin henrick learning_analytics-workshop_questions
Gavin henrick learning_analytics-workshop_questionsGavin Henrick
 
Preparing Students
Preparing StudentsPreparing Students
Preparing StudentsKatie Turner
 
Blackboard Opening Keynote | Katie Blot - Blackboard | TLCANZ17
Blackboard Opening Keynote | Katie Blot - Blackboard | TLCANZ17Blackboard Opening Keynote | Katie Blot - Blackboard | TLCANZ17
Blackboard Opening Keynote | Katie Blot - Blackboard | TLCANZ17Blackboard APAC
 
Improving student engagement with the assessment process in undergraduate mic...
Improving student engagement with the assessment process in undergraduate mic...Improving student engagement with the assessment process in undergraduate mic...
Improving student engagement with the assessment process in undergraduate mic...Alison Graham
 
Effective Teaching with Learn@UW
Effective Teaching with Learn@UWEffective Teaching with Learn@UW
Effective Teaching with Learn@UWjbohrer
 
Supporting Faculty in the Virtual Classroom
Supporting Faculty in the Virtual ClassroomSupporting Faculty in the Virtual Classroom
Supporting Faculty in the Virtual ClassroomMelanie
 
E-Learning in Newborn Health A paradigm shift in continuing professional deve...
E-Learning in Newborn HealthA paradigm shift in continuing professional deve...E-Learning in Newborn HealthA paradigm shift in continuing professional deve...
E-Learning in Newborn Health A paradigm shift in continuing professional deve...nisaiims
 
Supporting Faculty In The Angel Virtual Classroom
Supporting Faculty In The Angel Virtual ClassroomSupporting Faculty In The Angel Virtual Classroom
Supporting Faculty In The Angel Virtual ClassroomMelanie
 
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...EDEN Digital Learning Europe
 
Extending Moodle - Moodlemoot Romania 2013
Extending Moodle - Moodlemoot Romania 2013Extending Moodle - Moodlemoot Romania 2013
Extending Moodle - Moodlemoot Romania 2013Gavin Henrick
 
Can electronic marking help engage students with assessment and feedback
Can electronic marking help engage students with assessment and feedbackCan electronic marking help engage students with assessment and feedback
Can electronic marking help engage students with assessment and feedbackSara Marsham
 
Footholds and Foundations: Setting Freshmen on the Path to Lifelong Learning
Footholds and Foundations: Setting Freshmen on the Path to Lifelong LearningFootholds and Foundations: Setting Freshmen on the Path to Lifelong Learning
Footholds and Foundations: Setting Freshmen on the Path to Lifelong Learningannielibrarian
 
Can electronic marking help to engage students with assessment and feedback?
Can electronic marking help to engage students with assessment and feedback?Can electronic marking help to engage students with assessment and feedback?
Can electronic marking help to engage students with assessment and feedback?Alison Graham
 
Faculty Mentoring Faculty
Faculty Mentoring FacultyFaculty Mentoring Faculty
Faculty Mentoring FacultyStaci Trekles
 

What's hot (20)

Tips for Assessing Student Learning Using Blackboard
Tips for Assessing Student Learning Using BlackboardTips for Assessing Student Learning Using Blackboard
Tips for Assessing Student Learning Using Blackboard
 
Institutional roll out of submission and marking ss
Institutional roll out of submission and marking ssInstitutional roll out of submission and marking ss
Institutional roll out of submission and marking ss
 
Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...
Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...
Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...
 
iGeneration Conference
iGeneration ConferenceiGeneration Conference
iGeneration Conference
 
Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...
Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...
Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...
 
Gavin henrick learning_analytics-workshop_questions
Gavin henrick learning_analytics-workshop_questionsGavin henrick learning_analytics-workshop_questions
Gavin henrick learning_analytics-workshop_questions
 
Preparing Students
Preparing StudentsPreparing Students
Preparing Students
 
Blackboard Opening Keynote | Katie Blot - Blackboard | TLCANZ17
Blackboard Opening Keynote | Katie Blot - Blackboard | TLCANZ17Blackboard Opening Keynote | Katie Blot - Blackboard | TLCANZ17
Blackboard Opening Keynote | Katie Blot - Blackboard | TLCANZ17
 
Improving student engagement with the assessment process in undergraduate mic...
Improving student engagement with the assessment process in undergraduate mic...Improving student engagement with the assessment process in undergraduate mic...
Improving student engagement with the assessment process in undergraduate mic...
 
Effective Teaching with Learn@UW
Effective Teaching with Learn@UWEffective Teaching with Learn@UW
Effective Teaching with Learn@UW
 
Supporting Faculty in the Virtual Classroom
Supporting Faculty in the Virtual ClassroomSupporting Faculty in the Virtual Classroom
Supporting Faculty in the Virtual Classroom
 
E-Learning in Newborn Health A paradigm shift in continuing professional deve...
E-Learning in Newborn HealthA paradigm shift in continuing professional deve...E-Learning in Newborn HealthA paradigm shift in continuing professional deve...
E-Learning in Newborn Health A paradigm shift in continuing professional deve...
 
Supporting Faculty In The Angel Virtual Classroom
Supporting Faculty In The Angel Virtual ClassroomSupporting Faculty In The Angel Virtual Classroom
Supporting Faculty In The Angel Virtual Classroom
 
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
 
Extending Moodle - Moodlemoot Romania 2013
Extending Moodle - Moodlemoot Romania 2013Extending Moodle - Moodlemoot Romania 2013
Extending Moodle - Moodlemoot Romania 2013
 
CoLT 2017 - Prof. Dr. Vishna Devi Nadarajah
CoLT 2017 - Prof. Dr. Vishna Devi NadarajahCoLT 2017 - Prof. Dr. Vishna Devi Nadarajah
CoLT 2017 - Prof. Dr. Vishna Devi Nadarajah
 
Can electronic marking help engage students with assessment and feedback
Can electronic marking help engage students with assessment and feedbackCan electronic marking help engage students with assessment and feedback
Can electronic marking help engage students with assessment and feedback
 
Footholds and Foundations: Setting Freshmen on the Path to Lifelong Learning
Footholds and Foundations: Setting Freshmen on the Path to Lifelong LearningFootholds and Foundations: Setting Freshmen on the Path to Lifelong Learning
Footholds and Foundations: Setting Freshmen on the Path to Lifelong Learning
 
Can electronic marking help to engage students with assessment and feedback?
Can electronic marking help to engage students with assessment and feedback?Can electronic marking help to engage students with assessment and feedback?
Can electronic marking help to engage students with assessment and feedback?
 
Faculty Mentoring Faculty
Faculty Mentoring FacultyFaculty Mentoring Faculty
Faculty Mentoring Faculty
 

Viewers also liked

Adopting agile via continuous improvement with workshop
Adopting agile via continuous improvement with workshopAdopting agile via continuous improvement with workshop
Adopting agile via continuous improvement with workshopPriyank Shah
 
Software Development Process Models (SCRUM Methodology)
Software Development Process Models (SCRUM Methodology)Software Development Process Models (SCRUM Methodology)
Software Development Process Models (SCRUM Methodology)Muhammad Ahmed
 
Blackboard Learning Analytics Research Update
Blackboard Learning Analytics Research UpdateBlackboard Learning Analytics Research Update
Blackboard Learning Analytics Research UpdateJohn Whitmer, Ed.D.
 
Agile205: Intro to Agile Product Management
Agile205: Intro to Agile Product ManagementAgile205: Intro to Agile Product Management
Agile205: Intro to Agile Product ManagementRich Mironov
 
Product Strategy
Product StrategyProduct Strategy
Product StrategyMatt Hunter
 
Agile Product Manager/Product Owner Dilemma (PMEC)
Agile Product Manager/Product Owner Dilemma (PMEC)Agile Product Manager/Product Owner Dilemma (PMEC)
Agile Product Manager/Product Owner Dilemma (PMEC)Enthiosys Inc
 

Viewers also liked (7)

Adopting agile via continuous improvement with workshop
Adopting agile via continuous improvement with workshopAdopting agile via continuous improvement with workshop
Adopting agile via continuous improvement with workshop
 
Software Development Process Models (SCRUM Methodology)
Software Development Process Models (SCRUM Methodology)Software Development Process Models (SCRUM Methodology)
Software Development Process Models (SCRUM Methodology)
 
Test Automation in Agile
Test Automation in AgileTest Automation in Agile
Test Automation in Agile
 
Blackboard Learning Analytics Research Update
Blackboard Learning Analytics Research UpdateBlackboard Learning Analytics Research Update
Blackboard Learning Analytics Research Update
 
Agile205: Intro to Agile Product Management
Agile205: Intro to Agile Product ManagementAgile205: Intro to Agile Product Management
Agile205: Intro to Agile Product Management
 
Product Strategy
Product StrategyProduct Strategy
Product Strategy
 
Agile Product Manager/Product Owner Dilemma (PMEC)
Agile Product Manager/Product Owner Dilemma (PMEC)Agile Product Manager/Product Owner Dilemma (PMEC)
Agile Product Manager/Product Owner Dilemma (PMEC)
 

Similar to The RISE Framework: Using learning analytics for the continuous improvement of open educational resources

ATD OER Degree Initiative | SRI-rpkGROUP Services
ATD OER Degree Initiative | SRI-rpkGROUP ServicesATD OER Degree Initiative | SRI-rpkGROUP Services
ATD OER Degree Initiative | SRI-rpkGROUP ServicesAchieving the Dream
 
OER Degee Initiative Kickoff | Data & Evaluation Services
OER Degee Initiative Kickoff |  Data & Evaluation ServicesOER Degee Initiative Kickoff |  Data & Evaluation Services
OER Degee Initiative Kickoff | Data & Evaluation ServicesAchieving the Dream
 
SYNERGY Induction to Pedagogy Programme - Evaluation of the Learning Resource...
SYNERGY Induction to Pedagogy Programme - Evaluation of the Learning Resource...SYNERGY Induction to Pedagogy Programme - Evaluation of the Learning Resource...
SYNERGY Induction to Pedagogy Programme - Evaluation of the Learning Resource...Sarah Land
 
An Approach to Skill Mapping in Online Courses
An Approach to Skill Mapping in Online CoursesAn Approach to Skill Mapping in Online Courses
An Approach to Skill Mapping in Online CoursesRoss Strader
 
CCCOER OER Degree Research with Achieving the Dream, SRI Education, and rpk G...
CCCOER OER Degree Research with Achieving the Dream, SRI Education, and rpk G...CCCOER OER Degree Research with Achieving the Dream, SRI Education, and rpk G...
CCCOER OER Degree Research with Achieving the Dream, SRI Education, and rpk G...Una Daly
 
ABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of DerbyABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of DerbyEd Foster
 
LIN3 2013: 5. technical and pedagogical assessment handout
LIN3 2013: 5. technical and pedagogical assessment handout LIN3 2013: 5. technical and pedagogical assessment handout
LIN3 2013: 5. technical and pedagogical assessment handout Anu Ylitalo
 
SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...
SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...
SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...Sarah Land
 
Development of the Open Educational Practices Impact Evaluation Index
Development of the Open Educational Practices Impact Evaluation IndexDevelopment of the Open Educational Practices Impact Evaluation Index
Development of the Open Educational Practices Impact Evaluation IndexROER4D
 
Development of the Open Educational Practices Impact Evaluation Index
Development of the Open Educational Practices Impact Evaluation IndexDevelopment of the Open Educational Practices Impact Evaluation Index
Development of the Open Educational Practices Impact Evaluation IndexOpen Education Consortium
 
OE-Global2017-Developing_OEPIEIndex_Naidu_Karunanayaka
OE-Global2017-Developing_OEPIEIndex_Naidu_KarunanayakaOE-Global2017-Developing_OEPIEIndex_Naidu_Karunanayaka
OE-Global2017-Developing_OEPIEIndex_Naidu_KarunanayakaShironica Karunanayaka
 
Step 1 Project Initiation and get organized Rev1_print.pptx
Step 1 Project Initiation and get organized Rev1_print.pptxStep 1 Project Initiation and get organized Rev1_print.pptx
Step 1 Project Initiation and get organized Rev1_print.pptxARNELUSMAN2
 
PRINCIPLES RELATED TO THE SELECTION - Assessment.pdf
PRINCIPLES RELATED TO THE SELECTION  - Assessment.pdfPRINCIPLES RELATED TO THE SELECTION  - Assessment.pdf
PRINCIPLES RELATED TO THE SELECTION - Assessment.pdfBeulahJayarani
 
Content acquisition process
Content acquisition processContent acquisition process
Content acquisition processM Large
 
OBE-Implementation Plan 17-8-21.pptx
OBE-Implementation Plan 17-8-21.pptxOBE-Implementation Plan 17-8-21.pptx
OBE-Implementation Plan 17-8-21.pptxsyedmoazzamnizami
 
5R Open Course Design Framework, Fall 2015 version
5R Open Course Design Framework, Fall 2015 version5R Open Course Design Framework, Fall 2015 version
5R Open Course Design Framework, Fall 2015 versionDavid Wiley
 
Jisc learning analytics service overview Aug 2018
Jisc learning analytics service overview Aug 2018Jisc learning analytics service overview Aug 2018
Jisc learning analytics service overview Aug 2018Paul Bailey
 
Empowering the Instructor with Learning Analytics
Empowering the Instructor with Learning AnalyticsEmpowering the Instructor with Learning Analytics
Empowering the Instructor with Learning AnalyticsMichael Wilder
 

Similar to The RISE Framework: Using learning analytics for the continuous improvement of open educational resources (20)

ATD OER Degree Initiative | SRI-rpkGROUP Services
ATD OER Degree Initiative | SRI-rpkGROUP ServicesATD OER Degree Initiative | SRI-rpkGROUP Services
ATD OER Degree Initiative | SRI-rpkGROUP Services
 
OER Degee Initiative Kickoff | Data & Evaluation Services
OER Degee Initiative Kickoff |  Data & Evaluation ServicesOER Degee Initiative Kickoff |  Data & Evaluation Services
OER Degee Initiative Kickoff | Data & Evaluation Services
 
SYNERGY Induction to Pedagogy Programme - Evaluation of the Learning Resource...
SYNERGY Induction to Pedagogy Programme - Evaluation of the Learning Resource...SYNERGY Induction to Pedagogy Programme - Evaluation of the Learning Resource...
SYNERGY Induction to Pedagogy Programme - Evaluation of the Learning Resource...
 
An Approach to Skill Mapping in Online Courses
An Approach to Skill Mapping in Online CoursesAn Approach to Skill Mapping in Online Courses
An Approach to Skill Mapping in Online Courses
 
CCCOER OER Degree Research with Achieving the Dream, SRI Education, and rpk G...
CCCOER OER Degree Research with Achieving the Dream, SRI Education, and rpk G...CCCOER OER Degree Research with Achieving the Dream, SRI Education, and rpk G...
CCCOER OER Degree Research with Achieving the Dream, SRI Education, and rpk G...
 
Using Assessment data
Using Assessment dataUsing Assessment data
Using Assessment data
 
ABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of DerbyABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of Derby
 
LIN3 2013: 5. technical and pedagogical assessment handout
LIN3 2013: 5. technical and pedagogical assessment handout LIN3 2013: 5. technical and pedagogical assessment handout
LIN3 2013: 5. technical and pedagogical assessment handout
 
SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...
SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...
SYNERGY Induction to Pedagogy Programme - Evaluation of the Environment (ENGL...
 
Development of the Open Educational Practices Impact Evaluation Index
Development of the Open Educational Practices Impact Evaluation IndexDevelopment of the Open Educational Practices Impact Evaluation Index
Development of the Open Educational Practices Impact Evaluation Index
 
Development of the Open Educational Practices Impact Evaluation Index
Development of the Open Educational Practices Impact Evaluation IndexDevelopment of the Open Educational Practices Impact Evaluation Index
Development of the Open Educational Practices Impact Evaluation Index
 
OE-Global2017-Developing_OEPIEIndex_Naidu_Karunanayaka
OE-Global2017-Developing_OEPIEIndex_Naidu_KarunanayakaOE-Global2017-Developing_OEPIEIndex_Naidu_Karunanayaka
OE-Global2017-Developing_OEPIEIndex_Naidu_Karunanayaka
 
Step 1 Project Initiation and get organized Rev1_print.pptx
Step 1 Project Initiation and get organized Rev1_print.pptxStep 1 Project Initiation and get organized Rev1_print.pptx
Step 1 Project Initiation and get organized Rev1_print.pptx
 
PRINCIPLES RELATED TO THE SELECTION - Assessment.pdf
PRINCIPLES RELATED TO THE SELECTION  - Assessment.pdfPRINCIPLES RELATED TO THE SELECTION  - Assessment.pdf
PRINCIPLES RELATED TO THE SELECTION - Assessment.pdf
 
Content acquisition process
Content acquisition processContent acquisition process
Content acquisition process
 
OBE-Implementation Plan 17-8-21.pptx
OBE-Implementation Plan 17-8-21.pptxOBE-Implementation Plan 17-8-21.pptx
OBE-Implementation Plan 17-8-21.pptx
 
5R Open Course Design Framework, Fall 2015 version
5R Open Course Design Framework, Fall 2015 version5R Open Course Design Framework, Fall 2015 version
5R Open Course Design Framework, Fall 2015 version
 
Jisc learning analytics service overview Aug 2018
Jisc learning analytics service overview Aug 2018Jisc learning analytics service overview Aug 2018
Jisc learning analytics service overview Aug 2018
 
Empowering the Instructor with Learning Analytics
Empowering the Instructor with Learning AnalyticsEmpowering the Instructor with Learning Analytics
Empowering the Instructor with Learning Analytics
 
OBE pdf [Autosaved].ppt
OBE pdf [Autosaved].pptOBE pdf [Autosaved].ppt
OBE pdf [Autosaved].ppt
 

Recently uploaded

DATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etcDATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etclalithasri22
 
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...Dr Arash Najmaei ( Phd., MBA, BSc)
 
Statistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfStatistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfnikeshsingh56
 
Film cover research.pptx for media courseowrk
Film cover research.pptx for media courseowrkFilm cover research.pptx for media courseowrk
Film cover research.pptx for media courseowrk494f574xmv
 
Inference rules in artificial intelligence
Inference rules in artificial intelligenceInference rules in artificial intelligence
Inference rules in artificial intelligencePriyadharshiniG41
 
Adobe Scan 06-Mar-2024 (1).pdf shavashwvw
Adobe Scan 06-Mar-2024 (1).pdf shavashwvwAdobe Scan 06-Mar-2024 (1).pdf shavashwvw
Adobe Scan 06-Mar-2024 (1).pdf shavashwvws73678sri
 
Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...
Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...
Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...ThinkInnovation
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaManalVerma4
 
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdf
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdfNeo4j_Exploring the Impact of Graph Technology on Financial Services.pdf
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdfNeo4j
 
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdf
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdfRabobank_Exploring the Impact of Graph Technology on Financial Services.pdf
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdfNeo4j
 
Data Discovery With Power Query in excel
Data Discovery With Power Query in excelData Discovery With Power Query in excel
Data Discovery With Power Query in excelKapilSidhpuria3
 
prediction of default payment next month using a logistic approach
prediction of default payment next month using a logistic approachprediction of default payment next month using a logistic approach
prediction of default payment next month using a logistic approachAdekunleJoseph4
 
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...Jack Cole
 
Role of Consumer Insights in business transformation
Role of Consumer Insights in business transformationRole of Consumer Insights in business transformation
Role of Consumer Insights in business transformationAnnie Melnic
 
Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfNicoChristianSunaryo
 
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsba
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsbaAdobe Scan 06-Mar-2024 (1).pdfwvsbbsbsba
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsbas73678sri
 
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelDecoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelBoston Institute of Analytics
 
testingsdadadadaaddadadadadadadadaad.pdf
testingsdadadadaaddadadadadadadadaad.pdftestingsdadadadaaddadadadadadadadaad.pdf
testingsdadadadaaddadadadadadadadaad.pdfDSP Mutual Fund
 

Recently uploaded (19)

DATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etcDATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etc
 
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
 
Statistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfStatistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdf
 
Film cover research.pptx for media courseowrk
Film cover research.pptx for media courseowrkFilm cover research.pptx for media courseowrk
Film cover research.pptx for media courseowrk
 
Inference rules in artificial intelligence
Inference rules in artificial intelligenceInference rules in artificial intelligence
Inference rules in artificial intelligence
 
Adobe Scan 06-Mar-2024 (1).pdf shavashwvw
Adobe Scan 06-Mar-2024 (1).pdf shavashwvwAdobe Scan 06-Mar-2024 (1).pdf shavashwvw
Adobe Scan 06-Mar-2024 (1).pdf shavashwvw
 
Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...
Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...
Predictive Analysis - Using Insight-informed Data to Plan Inventory in Next 6...
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in India
 
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdf
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdfNeo4j_Exploring the Impact of Graph Technology on Financial Services.pdf
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdf
 
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdf
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdfRabobank_Exploring the Impact of Graph Technology on Financial Services.pdf
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdf
 
Data Discovery With Power Query in excel
Data Discovery With Power Query in excelData Discovery With Power Query in excel
Data Discovery With Power Query in excel
 
prediction of default payment next month using a logistic approach
prediction of default payment next month using a logistic approachprediction of default payment next month using a logistic approach
prediction of default payment next month using a logistic approach
 
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
 
Role of Consumer Insights in business transformation
Role of Consumer Insights in business transformationRole of Consumer Insights in business transformation
Role of Consumer Insights in business transformation
 
2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use
 
Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdf
 
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsba
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsbaAdobe Scan 06-Mar-2024 (1).pdfwvsbbsbsba
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsba
 
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelDecoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
 
testingsdadadadaaddadadadadadadadaad.pdf
testingsdadadadaaddadadadadadadadaad.pdftestingsdadadadaaddadadadadadadadaad.pdf
testingsdadadadaaddadadadadadadadaad.pdf
 

The RISE Framework: Using learning analytics for the continuous improvement of open educational resources

  • 1. The RISE Framework: Using learning analytics for the continuous improvement of open educational resources Robert Bodily, Rob Nyland, and David Wiley
  • 2. Project Background ◈ Bob and Rob were members of John Hilton’s OER Research Class at BYU ◈ In the class we were using the COUP framework for OER Research ◆ Cost ◆ Outcomes ◆ Use ◆ Perceptions ◈ We wanted to understand what learning analytics could bring to the conversation around OER
  • 3. The 5 R’s of OER Retain Reuse Revise Remix Redistribute
  • 4. “Because the Open Curriculum is licensed in such a way that we can revise materials directly, OHSU is able to engage in a highly data-driven curriculum improvement process. As students go through the online courses, their clicks and learning paths can be tracked and recorded. This information can be combined with item response theory and learning outcome analysis data to set priorities for curriculum or assessment revision empirically. This data-driven process of curriculum improvement should allow the Open Curriculum to reach a very high level of quality very quickly.” - David Wiley, 2009
  • 6. Continuous Improvement at OLI “A key attribute of the OLI environment is that while students are working through the course, we are collecting analytics data and using those data to drive multiple feedback loops” -Strader & Thille (2012)
  • 7. Resource Inspection, Selection, and Enhancement (RISE) Inspection: Analyze course resources using a variety of automated methods Selection: Identify resources that need additional evaluation Enhancement: Facilitate enhancement through an automated inspection and selection process (enhancement is usually done manually by an instructional designer).
  • 8. Four Quadrant Evaluation Structure High Grades High student prior knowledge, inherently easy learning outcome, highly effective content, poorly written assessment Effective resources, effective assessment, strong outcome alignment Low Grades Low motivation or high life distraction of students, too much material, technical or other difficulties accessing resources Poorly designed resources, poorly written assessments, poor outcome alignment, difficult learning outcome Low Use High Use
  • 9. Our Approach: Case Study ◈ Fall 2015 Introduction to Business Course (N = 1002) ◈ OER courseware platform ◆ OER online text ◆ Videos ◆ Simulations ◆ Formative assessments ◆ Practice assessments ◆ Summative assessments ◈ Assessments and resources are all linked to course outcomes
  • 10. Data Tracking Resource Use Data ◈ Tracked with Google Analytics ◈ Defined as pageviews (other options include time spent and ratings) Assessment Data ◈ Tracked with Open Assessments ◈ Defined as grade on summative assessments
  • 11. Types of Analyses 1. Outer quadrant analysis 2. Page type analysis 3. Extreme outlier analysis
  • 12. Outer Quadrant Analysis ◈ Blue dotted line is the additive z-score threshold. ◈ Black lines are the median values for pageviews and grades
  • 13. Outer Quadrant Analysis Page Title Grade PV/student A.Z-Score Quadrant Reading: The Four Ps of Marketing 0.81 1.64 3.65 1 Outcome: Recruiting Employees 0.88 0.53 3.4 2 Video: Legal Rights Under Implied Warranties 0.53 0.57 3.6 3 Reading: The Organization Chart and Reporting Structure 0.64 1.89 4.56 4
  • 14. Page Type Analysis Reading Pages Self-Check Pages Outcome Pages Video Pages Simulation Pages
  • 15. ◈ External search traffic: 1500 pageviews ◈ Course search traffic: 1626 pageviews ◈ Once external search traffic was removed, it was no longer an outlier. Extreme Outlier Analysis Page Title Grade PV/student A.Z-Score Reading: Stages of an Economy 0.83 2.82 7.79
  • 16. Enhancement ◈ Where the continuous improvement “rubber meets the road” ◈ Start with a theory of the domain ◈ Examine assessments ◆ Evaluate student response patterns / IRT / CFA ◆ Review assessments for design problems (negative language) ◆ Redesign or replace assessments ◈ Move on to content ◆ Examine content for assessment alignment (outcome, words) ◆ Examine content for micro-design problems (mismatch) ◆ Redesign or replace content