Penny Ralston-Berg August 7, 2009 Distance Teaching and Learning Madison, WI

Outline Project background Findings Implications for course design Questions

Project background

Findings

Implications for course design

Questions

Project Background Benchmarks in quality Institute for Higher Education Policy (2000) Sloan-C Quality Framework (2002) AECT Best Practices in Instructional Design (2003) Quality Matters (2005-06) E-Learning Maturity Model (2006) NACOL National Standards for Quality Online Teaching (2008) Quality Matters (2008-10)

Benchmarks in quality

Institute for Higher Education Policy (2000)

Sloan-C Quality Framework (2002)

AECT Best Practices in Instructional Design (2003)

Quality Matters (2005-06)

E-Learning Maturity Model (2006)

NACOL National Standards for Quality Online Teaching (2008)

Quality Matters (2008-10)

Project Background Benchmarks in quality Institute for Higher Education Policy (2000) Sloan-C Quality Framework (2002) AECT Best Practices in Instructional Design (2003) Quality Matters (2005-06) E-Learning Maturity Model (2006) NACOL National Standards for Quality Online Teaching (2008) Quality Matters (2008-10)

Benchmarks in quality

Institute for Higher Education Policy (2000)

Sloan-C Quality Framework (2002)

AECT Best Practices in Instructional Design (2003)

Quality Matters (2005-06)

E-Learning Maturity Model (2006)

NACOL National Standards for Quality Online Teaching (2008)

Quality Matters (2008-10)

Project Background Quality Matters (QM) stands out Research refined over time Comprehensive literature review Distilled to one-page rubric Eight categories Forty specific elements ranked as essential, very important, important Subscription service : http://www.qualitymatters.org Reference: © 2008. MarylandOnline, Inc. All rights reserved.

Quality Matters (QM) stands out

Research refined over time

Comprehensive literature review

Distilled to one-page rubric

Eight categories

Forty specific elements ranked as essential, very important, important

Subscription service : http://www.qualitymatters.org

Additional Resources Previous research (Ralston-Berg & Nath) Garrison & Arbaugh (2007) Community of Inquiry Framework Stodel, Thompson, & McDonald (2006) Learners' perspectives on what is missing from online learning Young & Norgard (2006) Quality from the students’ perspective

Previous research (Ralston-Berg & Nath)

Garrison & Arbaugh (2007) Community of Inquiry Framework

Stodel, Thompson, & McDonald (2006) Learners' perspectives on what is missing from online learning

Young & Norgard (2006) Quality from the students’ perspective

Quality: A Definition Quality can be defined as an educational system in which the learner has a reasonable opportunity for success. (Ragan & Sax, 2005) Ask students about quality in terms of which items contribute to their success in an online environment.

Quality can be defined as an educational system in which the learner has a reasonable opportunity for success. (Ragan & Sax, 2005)

Ask students about quality in terms of which items contribute to their success in an online environment.

Key Questions Do students agree with published benchmarks? Are published characteristics of quality perceived as contributing to student success? Is there anything missing? How does this impact our design practices for online learners?

Do students agree with published benchmarks?

Are published characteristics of quality perceived as contributing to student success?

Is there anything missing?

How does this impact our design practices for online learners?

Method Descriptive research Quantitative online survey design Respondents will be at least 18 years of age Completed course or currently enrolled Participation is voluntary, anonymous and confidential.

Descriptive research

Quantitative online survey design

Respondents will be at least 18 years of age

Completed course or currently enrolled

Participation is voluntary, anonymous and confidential.

Method Examine quality elements listed in: Quality Matters rubric standards 2008-2010 edition with assigned point values © 2006 MarylandOnline, Inc. http://www.qualitymatters.org Other sources Convert quality elements to student-centered questions

Examine quality elements listed in: Quality Matters rubric standards 2008-2010 edition with assigned point values © 2006 MarylandOnline, Inc. http://www.qualitymatters.org Other sources

Convert quality elements to student-centered questions

Method Example Original: Instructions make clear how to get started and where to find various course components. Revised: Clear instructions tell me how to get started and how to find various course components.

Example Original: Instructions make clear how to get started and where to find various course components. Revised: Clear instructions tell me how to get started and how to find various course components.

Survey Design Delivery in online format – Survey Monkey 89 items 69 quality indicators 3 open response 17 demographic

Delivery in online format – Survey Monkey

89 items

69 quality indicators

3 open response

17 demographic

Survey Design Rank value on scale of 0 to 3 0=Not at all important - does not contribute to my success 1=Important 2=Very important 3=Essential – could not succeed without it

Rank value on scale of 0 to 3

0=Not at all important - does not contribute to my success

1=Important

2=Very important

3=Essential – could not succeed without it

Survey Delivery Four-year universities Two-year institutions Online student communities Facebook Student.com eLearners.com DegreeInfo Distance Learning Discussion Forums OOEN.net

Four-year universities

Two-year institutions

Online student communities

Facebook

Student.com

eLearners.com

DegreeInfo Distance Learning Discussion Forums

OOEN.net

Demographics 202 responses 68% Female / 32% Male Age

202 responses

68% Female / 32% Male

Age

Demographics Courses taken

Courses taken

Demographics Cohort-based courses Comfortable or very comfortable with technology 1/2 are full time students 1/3 spend more than 16 hours per week on courses 2/3 employed full time

Cohort-based courses

Comfortable or very comfortable with technology

1/2 are full time students

1/3 spend more than 16 hours per week on courses

2/3 employed full time

Demographics

Do Students Agree? YES! All QM items scored >1 39 items scored >1.5 QM 2 just as important as QM 3 QM 1 related to instructions, usability just as important as QM 2

YES!

All QM items scored >1

39 items scored >1.5

QM 2 just as important as QM 3

QM 1 related to instructions, usability just as important as QM 2

Top 10 Student Countdown Assessments are appropriately timed, varied and appropriate Instructions on how to access resources are sufficient and easy to understand Course components are web-based or easily downloaded for use offline Requirements for interaction are clearly explained Opportunity to speak with instructor, classmates through email, bulletin boards, any time / any place

Assessments are appropriately timed, varied and appropriate

Instructions on how to access resources are sufficient and easy to understand

Course components are web-based or easily downloaded for use offline

Requirements for interaction are clearly explained

Opportunity to speak with instructor, classmates through email, bulletin boards, any time / any place

Top 10 Student Countdown Technologies required are readily available – provided or easily downloaded Clear instructions tell me how to get started and to find course components Criteria for how I will be evaluated is descriptive and specific Navigation is logical, consistent, and efficient The grading policy is clearly stated

Technologies required are readily available – provided or easily downloaded

Clear instructions tell me how to get started and to find course components

Criteria for how I will be evaluated is descriptive and specific

Navigation is logical, consistent, and efficient

The grading policy is clearly stated

Themes Technology function / support Clear expectations Navigation Assessments appropriate / varied More traditional communication

Technology function / support

Clear expectations

Navigation

Assessments appropriate / varied

More traditional communication

Do Students Agree? But… Interaction scores lower than expected Build a community Synchronous communication Students as contributors of content Multimedia Games

But…

Interaction scores lower than expected

Build a community

Synchronous communication

Students as contributors of content

Multimedia

Games

Bottom 10 I find course-related content and share it with the instructor and classmates I use wikis, shared documents, or other online collaborative tools to complete group work I am asked to introduce myself to the class I coach other students and help answer their questions Opportunity to speak with my instructor and classmates through… same time, same place technology

I find course-related content and share it with the instructor and classmates

I use wikis, shared documents, or other online collaborative tools to complete group work

I am asked to introduce myself to the class

I coach other students and help answer their questions

Opportunity to speak with my instructor and classmates through… same time, same place technology

Bottom 10 Instructor does not participate in class discussions Course contains interactive games or simulations I author / create course-related content and share it with instructor and classmates I work in groups with other students Course contains audio and video content

Instructor does not participate in class discussions

Course contains interactive games or simulations

I author / create course-related content and share it with instructor and classmates

I work in groups with other students

Course contains audio and video content

Types of Interaction Students’ Rank Learning activities encourage me to interact with content in the course. 1.90 Learning activities encourage me to interact with my instructor. 1.58 Learning activities encourage me to interact with other students. 1.33

Content Interaction Students’ Rank The course provides practice exercises that I may complete on my own to master difficult content. 1.82 Course contains instructional graphics that help explain complex concepts. 1.67 Ungraded practice quizzes are provided so that I may check my own knowledge and understanding of a topic. 1.63 I find course-related content and share it with the instructor and classmates. 1.02 Course contains audio and video content. 0.96 I author / create course-related content and share it with instructor and classmates. 0.90 Course contains interactive games or simulations. 0.86

Instructor Interaction Students’ Rank Clear standards are set for instructor response (turn-around time for email, grade posting, etc.). 2.32 Clear standards are set for instructor availability (office hours, etc.) 2.17 Instructor actively participates in the class discussions. 2.01 Instructor summarizes and guides the class discussions. 1.96 Instructor shares personal insights and experiences above and beyond the reading assignments. 1.74 Instructor does not participate in class discussions. 0.42

Student-Student Interaction Students’ Rank Requirements for my interaction with the instructor, content, and other students are clearly explained. 2.40 Opportunity to ask questions of other students and get help from sources other than the instructor. 1.64 I share my ideas, opinions, and findings with my classmates in a public forum. 1.40 I coach other students and help answer their questions. 1.09 I am asked to introduce myself to the class. 1.06 I use wikis, shared documents, or other online collaborative tools to complete group work. 1.04 I work in groups with other students. 0.93

Themes Faculty presence Same time, same place Collaborative content creation Media / games Student as source of content / mentor Essential vs. supportive

Faculty presence

Same time, same place

Collaborative content creation

Media / games

Student as source of content / mentor

Essential vs. supportive

Design Implications Is <name your technology> essential? Maybe not Is <name your technology> useful? Maybe / maybe not Focus Expectations / outcomes Teaching and learning Less on specific technology / media

Is <name your technology> essential?

Maybe not

Is <name your technology> useful?

Maybe / maybe not

Focus

Expectations / outcomes

Teaching and learning

Less on specific technology / media

Design Implications Clear expectations Relevance Usability Availability Functionality Tied to learning / assessment

Clear expectations

Relevance

Usability

Availability

Functionality

Tied to learning / assessment

What’s missing? Get info prior to course start All-inclusive syllabus Flexible schedule / deadlines Reminders about due dates / keeping on task Affordability Fairness Learning from colleagues / professionals Computer / Internet are all I need

Get info prior to course start

All-inclusive syllabus

Flexible schedule / deadlines

Reminders about due dates / keeping on task

Affordability

Fairness

Learning from colleagues / professionals

Computer / Internet are all I need

What’s missing? Leadership of the instructor Feedback on every graded item / fast turnaround Use of outside resources to increase comprehension Labs (for science courses) Context (how lessons relate to one another) Cohesive course materials (materials from different sources flow together) Quizzes that auto-save answers

Leadership of the instructor

Feedback on every graded item / fast turnaround

Use of outside resources to increase comprehension

Labs (for science courses)

Context (how lessons relate to one another)

Cohesive course materials (materials from different sources flow together)

Quizzes that auto-save answers

Design Implications Technology is “make or break” (functionality) Content and instructor / student interaction are core Media, Web 2.0 supplemental Layered design Facilitation skills / availability Flexibility Convenience Students see the total package

Technology is “make or break” (functionality)

Content and instructor / student interaction are core

Media, Web 2.0 supplemental

Layered design

Facilitation skills / availability

Flexibility

Convenience

Students see the total package

Invitation Student perspectives in quality ongoing study Replicate study with your institution / school? Students have completed at least one online course Students complete the survey online

Student perspectives in quality ongoing study

Replicate study with your institution / school?

Students have completed at least one online course

Students complete the survey online

Contact Info Penny Ralston-Berg Instructional Designer Penn State World Campus [email_address] (papa – lima – romeo – one – five at psu – dot – edu) Slideshare

Penny Ralston-Berg Instructional Designer Penn State World Campus [email_address] (papa – lima – romeo – one – five at psu – dot – edu) Slideshare

Course Overview and Introduction Quality Matters Rubric Standards 2008-2010 edition with Assigned Point Values © 2006 MarylandOnline, Inc. Students’ Rank Clear instructions tell me how to get started and how to find various course components. 2.56 Minimum preparation or prerequisite knowledge I need to succeed in the course is clearly stated. 2.06 Minimum technical skills expected of me are clearly stated. 1.92 The instructor introduces her- or himself. 1.84 A statement introduces me to the purpose of the course and its components. 1.82 Etiquette (or “netiquette”) guidelines for how to behave online are clearly stated. 1.46 I am asked to introduce myself to the class. 1.06

Learning Objectives Quality Matters Rubric Standards 2008-2010 edition with Assigned Point Values © 2006 MarylandOnline, Inc. Students’ Rank Instructions on how to meet the learning objectives are adequate and stated clearly. 2.28 The learning objectives are appropriate for the level of the course. 2.12 The course learning objectives describe outcomes that I am able to achieve. 1.81 The course contains learning objectives listing what I am expected to learn. 1.80 The course contains learning objectives listing what I am expected to learn. 1.80 The module/unit learning objectives describe outcomes that I am able to achieve and are consistent with the course-level objectives. 1.76 All learning objectives are clearly stated and written from my perspective. 1.74

Assessment and Measurement Quality Matters Rubric Standards 2008-2010 edition with Assigned Point Values © 2006 MarylandOnline, Inc. Students’ Rank Assessments measure the stated learning objectives and are consistent with course activities and resources. 2.38 The grading policy is stated clearly. 2.48 Criteria for how my work and participation will be evaluated are descriptive and specific. 2.55 Assessments are appropriately timed within the length of the course, varied, and appropriate to the content being assessed. 2.44 “ Self-check” or practice assignments are provided, and I am provided with timely feedback. 2.08 The course provides practice exercises that I may complete on my own to master difficult content. 1.82 Ungraded practice quizzes are provided so that I may check my own knowledge and understanding of a topic. 1.63

Resources and Materials Quality Matters Rubric Standards 2008-2010 edition with Assigned Point Values © 2006 MarylandOnline, Inc. Students’ Rank Instructional materials have sufficient breadth, depth, and currency for me to learn the subject. 2.34 Instructional materials contribute to the achievement of the course and module/unit learning objectives. 2.28 Instructional materials contribute to the achievement of the course and module/unit learning objectives. 2.28 The relationship between the instructional materials and the learning activities in clearly explained to me. 2.09 All resources and materials used in the course are appropriately cited. 1.72 Activities are included with each lesson. 1.58

Learner Engagement Quality Matters Rubric Standards 2008-2010 edition with Assigned Point Values © 2006 MarylandOnline, Inc. Students’ Rank Requirements for course interaction clearly articulated 5.28 Clear standards set for instructor response and availability 5.20 Activities promote achievement of stated outcomes 5.12 Activities foster content-student interaction 5.00 Course design prompts instructor to be present, active, engaged 4.94 Activities foster instructor-student interaction 4.64 Activities foster student-student interaction 4.24

Course Technology Quality Matters Rubric Standards 2008-2010 edition with Assigned Point Values © 2006 MarylandOnline, Inc. Students’ Rank Required technologies are provided or easily downloaded 5.40 Instructions to access online resources are sufficient and easy to understand 5.24 Tools and media support outcomes and are integrated with text and assignments 4.96

Learner Support Quality Matters Rubric Standards 2008-2010 edition with Assigned Point Values © 2006 MarylandOnline, Inc. Students’ Rank Clear description of technical support offered 4.78 Tutorials and resources to answer basic questions 4.60 Clear description of academic support offered 4.44

Accessibility Quality Matters Rubric Standards 2008-2010 edition with Assigned Point Values © 2006 MarylandOnline, Inc. Students’ Rank QM Rank The course incorporates ADA standards and reflect conformance with institutional policy regarding accessibility in online and hybrid courses. 5.02 3 Course pages and course materials provide equivalent alternatives to auditory and visual content. 4.86 2 Course pages have links that are self-describing and meaningful. 4.56 2 The course ensures screen readability. 4.26 1