Peter Shea Jason Vickers Suzanne Hayes University at Albany Revisiting the Community of Inquiry Framework of Online Learning

Outline Introduction Status of CoI Research Overview of project phases and design Findings and Implications

Introduction

Status of CoI Research

Overview of project phases and design

Findings and Implications

Changes in Higher Education Shift underway in undergraduate education from providing classroom instruction to producing learning With growth in online learning raises parallel question “What matters in online learning?” CoI is the widely accepted framework for explaining and predicting how people learn in online environments in the absence of face to face instruction

Shift underway in undergraduate education from providing classroom instruction to producing learning

With growth in online learning raises parallel question “What matters in online learning?”

CoI is the widely accepted framework for explaining and predicting how people learn in online environments in the absence of face to face instruction

Community of Inquiry Framework Social Presence The ability of participants to identify with the community (e.g., course of study), communicate purposefully in a trusting environment, and develop inter-personal relationships by way of projecting their individual personalities. Cognitive Presence The extent to which learners are able to construct and confirm meaning through sustained reflection and discourse in a critical community of inquiry. Teaching Presence The design, facilitation and direction of cognitive and social processes for the purpose of realizing personally meaningful and educationally worthwhile learning outcomes

Current CoI Research Focused in two areas: Large scale surveys of student perceptions of SP, TP and CP Quantitative content analysis of online discussions

Focused in two areas:

Large scale surveys of student perceptions of SP, TP and CP

Quantitative content analysis of online discussions

What is QCA? Research methodology used to examine transcripts of online interactions to see what participants say to each other Use SP, TP and CP codes to classify these statements Results used to look for patterns within course

Research methodology used to examine transcripts of online interactions to see what participants say to each other

Use SP, TP and CP codes to classify these statements

Results used to look for patterns within course

How has QCA been used? Have focused primarily on threaded discussions Has constrained us from seeing the broader picture of activity within a course TP in threaded discussions only accounts for 20-30% of TP activity

Have focused primarily on threaded discussions

Has constrained us from seeing the broader picture of activity within a course

TP in threaded discussions only accounts for 20-30% of TP activity

Some limitations of past research using QCA in CoI Early studies relied on samples of threaded discussions No studies examining an entire online course Most studies focus on one form of presence at a time Many studies based on a single course Early studies did not fully document IRR measures

Early studies relied on samples of threaded discussions

No studies examining an entire online course

Most studies focus on one form of presence at a time

Many studies based on a single course

Early studies did not fully document IRR measures

Our purpose Address prior methodological shortcomings Use a holistic approach Explore expansion of CoI framework to make it more encompassing and descriptive of all online course activities Determine if previous research results can be verified, refined or extended Improve and enhance CoI model and its constructs

Address prior methodological shortcomings

Use a holistic approach

Explore expansion of CoI framework to make it more encompassing and descriptive of all online course activities

Determine if previous research results can be verified, refined or extended

Improve and enhance CoI model and its constructs

Research Design Two identical courses Based on template developed by content experts and instructional designers Instructors were not the developers Five modules with discussions, mini lectures, case studies, scaffolded research project submitted in stages Upper level business management course from a state college in northeast

Two identical courses

Based on template developed by content experts and instructional designers

Instructors were not the developers

Five modules with discussions, mini lectures, case studies, scaffolded research project submitted in stages

Upper level business management course from a state college in northeast

Discussion: 25% of the final grade Debates: 10% Other Course Activities: 65%

Purposive Sample Four sections offered in Fall 2007 Two selected based initial assessment to identify instructors with different approaches Classes comparable in size Instructor A Instructor B Students enrolled at start of the term 19 20 Students completing course 17 16

Four sections offered in Fall 2007

Two selected based initial assessment to identify instructors with different approaches

Classes comparable in size

Data Analyzed in Multiple Phases Phase One: May 08 – March 09 Online discussions in each of the 5 main course modules in each course (SP, TP, CP) Phase Two: Jan – June 09 Examined student case studies for CP & SOLO taxonomy (Biggs and Collis, 1982) to measure high order thinking Phase Three: Aug 09 + Coding for SP and TP* that take place in course communications (outside of online discussions) Coding of all course documents for SP and TP* * CP coding not meaningful in this context P1 P2 P3

Phase One: May 08 – March 09

Online discussions in each of the 5 main course modules in each course (SP, TP, CP)

Phase Two: Jan – June 09

Examined student case studies for CP & SOLO taxonomy (Biggs and Collis, 1982) to measure high order thinking

Phase Three: Aug 09 +

Coding for SP and TP* that take place in course communications (outside of online discussions)

Coding of all course documents for SP and TP*

* CP coding not meaningful in this context

Course Instructions & Documents Course Material Coded Course Assignments Discussions CP SOLO Case Study Mod 1 X X Case Study Mod 2 X Case Study 3 X X Case Study 4 X X Description of Term Project Research Paper Outline Final Research Paper Discussions SP TP CP Module 1 X X X Module 2 X X X Module 2 Debate** Coding Underway Module 3 X X X Module 4 X X X Module 5 X X X SP TP CP Orientation & Syllabus X X X Introduction Module (w/ Icebreaker) X X Module 1 X X Module 2 X X Module 3 X X Module 4 X Module 5 P1 Communicative Documents P3 P3 P2

Coding Process Paired coders Coded random module for practice Established and documented guidelines to assure consistency Coded target modules Met to negotiate, identify and resolve disagreements, when possible Recorded IRR prior to and following negotiation

Paired coders

Coded random module for practice

Established and documented guidelines to assure consistency

Coded target modules

Met to negotiate, identify and resolve disagreements, when possible

Recorded IRR prior to and following negotiation

Coding Process* Message as Unit of Analysis Recorded presence/absence of category e.g. for SP we marked AF, OC, CH or NC Did not count occurrences of each indicator within each message Discussions & Course Communications Whole paper as Unit of Analysis in SOLO Paragraph as Unit of Analysis in CoI course documents coding (presence/absence of category e.g. for SP we marked AF, OC, CH or NC) Did not count occurrences of each indicator within each message Assignments & Course Documents *Coding for only CoI and SOLO in discussions and case studies required more than 6300 individual decisions.

Message as Unit of Analysis

Recorded presence/absence of category e.g. for SP we marked AF, OC, CH or NC

Did not count occurrences of each indicator within each message

Whole paper as Unit of Analysis in SOLO

Paragraph as Unit of Analysis in CoI course documents coding (presence/absence of category e.g. for SP we marked AF, OC, CH or NC)

Did not count occurrences of each indicator within each message

Cognitive Presence Coding Sheet Phase Code Indicator Socio-Cognitive Process Triggering Event CP-TE-1 Recognize problem Presenting background information that may culminate in a question or presents a problem/issues CP-TE-2 Sense of puzzlement Asking questions or Messages that take discussion in a new direction Exploration CP-EX-1 Exploration within the online community Unsubstantiated agreement or disagreement/contradiction of previous ideas CP-EX-2 Exploration within a single message Many different ideas/themes presented in one message CP-EX-3 Information exchange Personal narratives or description CP-EX-4 Suggestions for consideration Author explicitly characterizes message as exploration CP-EX-5 Leaps to conclusions Offers unsupported opinions Integration CP-IN-1 Integration among groups members; Building on, adding to others' ideas Reference to previous message followed by substantiated agreement or disagreement (I agree/disagree because…) CP-IN-2 Integration within a single message (response to prompt) Justified, developed, defensible, yet tentative hypotheses CP-IN-3 Connecting ideas, synthesis Integrating information from one or more sources - textbook, articles, personal experience, other posts or peer contributions. CP-IN-4 Creating solutions Explicit characterization of message as a solution by participant Resolution/Application CP-RE-1 Vicarious application to real world testing solutions; Providing examples of how problems were solved CP-RE-2 Defending solutions Defending why a problem was solved in a specific manner

Solo Taxonomy Coding 0 = no response 1= prestructural: misses the point 2 = unistructural: minimal response 3 = multistructural: source facts only 4 = relational: substantiated positions 5 = extended abstract: higher level of abstraction Used low and high indicators for levels 2-4.

0 = no response

1= prestructural: misses the point

2 = unistructural: minimal response

3 = multistructural: source facts only

4 = relational: substantiated positions

5 = extended abstract: higher level of abstraction

Used low and high indicators for levels 2-4.

Inter-Rater Reliability Reported Two Ways Cohen’s kappa Corrects for chance or random agreement between 2 coders Conservative May be artificially low (even with high level of agreement) 1.0 perfect agreement; .75+ excellent; .40 poor Measures the extent to which two or more coders agree Holsti’s Coefficient Expressed as a percent Less Conservative Minimum of 80% required 90% is excellent

Cohen’s kappa

Corrects for chance or random agreement between 2 coders

Conservative

May be artificially low (even with high level of agreement)

1.0 perfect agreement; .75+ excellent; .40 poor

Holsti’s Coefficient

Expressed as a percent

Less Conservative

Minimum of 80% required

90% is excellent

Findings: Locus of TP Not just limited to threaded discussion Majority of TP found outside of discussion in communicative processes such as: Course e-mail Private folders Comments on written assignments

Not just limited to threaded discussion

Majority of TP found outside of discussion in communicative processes such as:

Course e-mail

Private folders

Comments on written assignments

Distribution of Teaching Presence within Each Course

Findings: CP Measures in Discussions Consistent with prior research Majority of student postings are at exploration stage Low levels of integration and resolution

Consistent with prior research

Majority of student postings are at exploration stage

Low levels of integration and resolution

Findings: SOLO Measures in Case Studies SOLO: used because it measures outcomes instead of process Students did not "excel" on these measures (i.e. reach 4s and 5s) Student did "average" or slightly below (2s-3s) Mismatch between culture of grading and the ideals of SOLO framework

SOLO: used because it measures outcomes instead of process

Students did not "excel" on these measures (i.e. reach 4s and 5s)

Student did "average" or slightly below (2s-3s)

Mismatch between culture of grading and the ideals of SOLO framework

Findings: SOLO Measures in Case Studies cont. However, SOLO scores combined with CoI metrics do explain a majority of variance in teacher assigned grades The two models combined therefore show promise in describing and explaining learning in online environments - prime goals of a theoretical framework More research using this model is needed Alternate models for measuring outcomes should be considered

However, SOLO scores combined with CoI metrics do explain a majority of variance in teacher assigned grades

The two models combined therefore show promise in describing and explaining learning in online environments - prime goals of a theoretical framework

More research using this model is needed

Alternate models for measuring outcomes should be considered

Implications Design of course learning activities needs to be well integrated to encourage: integration of public discourse in discussion AND documentation of student’s learning in their private written artifacts When this does not occur there is limited opportunities for students to draw associations or reinforce course concepts to learning activities

Design of course learning activities needs to be well integrated to encourage:

integration of public discourse in discussion AND

documentation of student’s learning in their private written artifacts

When this does not occur there is limited opportunities for students to draw associations or reinforce course concepts to learning activities

Findings Disconnect between instructor’s grades and student learning outcomes based on SOLO and CP Failure of meaningful learning or difference Who’s wrong: The instructor or the framework?

Disconnect between instructor’s grades and student learning outcomes based on SOLO and CP

Failure of meaningful learning or difference

Who’s wrong: The instructor or the framework?

Implications Failure of meaningful learning? Difference in cultures of measurement between researchers and instructors?

Failure of meaningful learning?

Difference in cultures of measurement between researchers and instructors?

Thank you! Peter Shea Jason Vickers Suzanne Hayes University at Albany