Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The RISE Framework: Using learning analytics
for the continuous improvement of open
educational resources
Robert Bodily, R...
Project Background
◈ Bob and Rob were members of John Hilton’s OER Research
Class at BYU
◈ In the class we were using the ...
The 5 R’s of OER
Retain
Reuse
Revise
Remix
Redistribute
“Because the Open Curriculum is licensed in such a way that we can
revise materials directly, OHSU is able to engage in a ...
Continuous Improvement
Learning
Analytics
OER
Data-Driven
Continuous
Improvement
Continuous Improvement at OLI
“A key attribute of the OLI
environment is that while students
are working through the cours...
Resource Inspection, Selection, and Enhancement
(RISE)
Inspection: Analyze course resources using a variety of automated
m...
Four Quadrant Evaluation Structure
High
Grades
High student prior knowledge,
inherently easy learning outcome,
highly effe...
Our Approach: Case Study
◈ Fall 2015 Introduction to Business Course (N = 1002)
◈ OER courseware platform
◆ OER online tex...
Data Tracking
Resource Use Data
◈ Tracked with Google Analytics
◈ Defined as pageviews (other options include time spent a...
Types of Analyses
1. Outer quadrant analysis
2. Page type analysis
3. Extreme outlier analysis
Outer Quadrant Analysis
◈ Blue dotted line is the
additive z-score
threshold.
◈ Black lines are the
median values for
page...
Outer Quadrant Analysis
Page Title Grade PV/student A.Z-Score Quadrant
Reading: The Four Ps of Marketing 0.81 1.64 3.65 1
...
Page Type Analysis
Reading Pages Self-Check Pages Outcome Pages
Video Pages
Simulation
Pages
◈ External search traffic: 1500 pageviews
◈ Course search traffic: 1626 pageviews
◈ Once external search traffic was remov...
Enhancement
◈ Where the continuous improvement “rubber meets the road”
◈ Start with a theory of the domain
◈ Examine asses...
Questions?
Upcoming SlideShare
Loading in …5
×

The RISE Framework: Using learning analytics for the continuous improvement of open educational resources

241 views

Published on

We present the Resource Inspection, Selection, and Enhancement (RISE) framework, a learning analytics framework designed to enable teachers to engage in the continuous improvement process. This framework helps identify resources that should be evaluated by a teacher or an instructional designer.

Visit BobBodily.com for more information about my research.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

The RISE Framework: Using learning analytics for the continuous improvement of open educational resources

  1. 1. The RISE Framework: Using learning analytics for the continuous improvement of open educational resources Robert Bodily, Rob Nyland, and David Wiley
  2. 2. Project Background ◈ Bob and Rob were members of John Hilton’s OER Research Class at BYU ◈ In the class we were using the COUP framework for OER Research ◆ Cost ◆ Outcomes ◆ Use ◆ Perceptions ◈ We wanted to understand what learning analytics could bring to the conversation around OER
  3. 3. The 5 R’s of OER Retain Reuse Revise Remix Redistribute
  4. 4. “Because the Open Curriculum is licensed in such a way that we can revise materials directly, OHSU is able to engage in a highly data-driven curriculum improvement process. As students go through the online courses, their clicks and learning paths can be tracked and recorded. This information can be combined with item response theory and learning outcome analysis data to set priorities for curriculum or assessment revision empirically. This data-driven process of curriculum improvement should allow the Open Curriculum to reach a very high level of quality very quickly.” - David Wiley, 2009
  5. 5. Continuous Improvement Learning Analytics OER Data-Driven Continuous Improvement
  6. 6. Continuous Improvement at OLI “A key attribute of the OLI environment is that while students are working through the course, we are collecting analytics data and using those data to drive multiple feedback loops” -Strader & Thille (2012)
  7. 7. Resource Inspection, Selection, and Enhancement (RISE) Inspection: Analyze course resources using a variety of automated methods Selection: Identify resources that need additional evaluation Enhancement: Facilitate enhancement through an automated inspection and selection process (enhancement is usually done manually by an instructional designer).
  8. 8. Four Quadrant Evaluation Structure High Grades High student prior knowledge, inherently easy learning outcome, highly effective content, poorly written assessment Effective resources, effective assessment, strong outcome alignment Low Grades Low motivation or high life distraction of students, too much material, technical or other difficulties accessing resources Poorly designed resources, poorly written assessments, poor outcome alignment, difficult learning outcome Low Use High Use
  9. 9. Our Approach: Case Study ◈ Fall 2015 Introduction to Business Course (N = 1002) ◈ OER courseware platform ◆ OER online text ◆ Videos ◆ Simulations ◆ Formative assessments ◆ Practice assessments ◆ Summative assessments ◈ Assessments and resources are all linked to course outcomes
  10. 10. Data Tracking Resource Use Data ◈ Tracked with Google Analytics ◈ Defined as pageviews (other options include time spent and ratings) Assessment Data ◈ Tracked with Open Assessments ◈ Defined as grade on summative assessments
  11. 11. Types of Analyses 1. Outer quadrant analysis 2. Page type analysis 3. Extreme outlier analysis
  12. 12. Outer Quadrant Analysis ◈ Blue dotted line is the additive z-score threshold. ◈ Black lines are the median values for pageviews and grades
  13. 13. Outer Quadrant Analysis Page Title Grade PV/student A.Z-Score Quadrant Reading: The Four Ps of Marketing 0.81 1.64 3.65 1 Outcome: Recruiting Employees 0.88 0.53 3.4 2 Video: Legal Rights Under Implied Warranties 0.53 0.57 3.6 3 Reading: The Organization Chart and Reporting Structure 0.64 1.89 4.56 4
  14. 14. Page Type Analysis Reading Pages Self-Check Pages Outcome Pages Video Pages Simulation Pages
  15. 15. ◈ External search traffic: 1500 pageviews ◈ Course search traffic: 1626 pageviews ◈ Once external search traffic was removed, it was no longer an outlier. Extreme Outlier Analysis Page Title Grade PV/student A.Z-Score Reading: Stages of an Economy 0.83 2.82 7.79
  16. 16. Enhancement ◈ Where the continuous improvement “rubber meets the road” ◈ Start with a theory of the domain ◈ Examine assessments ◆ Evaluate student response patterns / IRT / CFA ◆ Review assessments for design problems (negative language) ◆ Redesign or replace assessments ◈ Move on to content ◆ Examine content for assessment alignment (outcome, words) ◆ Examine content for micro-design problems (mismatch) ◆ Redesign or replace content
  17. 17. Questions?

×