Are We There Yet

Slide 1: Assessing the Impact of Technology on Learning Karen Swan Research Center for Educational Technology Kent State University

Slide 2: The Great Media Debate Clark V S Kozma

Slide 3: What is the question?

Slide 4: inputs processes outcomes

Slide 5: inputs processes outcomes satisfaction retention success achievement proficiencies performance

Slide 6: achievement worth being familiar with important to know & do enduring understandings Understanding By Design, Wiggins & McTighe

Slide 7: inputs processes outcomes satisfaction retention success achievement proficiencies performance

Slide 8: inputs processes outcomes learner characteristics design learning resources professional development

Slide 9: learner characteristics factors influencing success (Dziuban & Moskal, 2006) Overall 85.9% Arts & Sciences, n=11,286 Health Business Admin., & Pub. Hospitality Mgmt. Education Engineering Affairs 85.8% 91.5% 72.7% 86.7% n=6,460 n=2,079 n=378 n=2,369 O F2F E, M, O F2F E, M F2F, E,86.5% M 74.8% 94.1% 89.1% 64.7% 79.6% n=5,639 n=821 n=1,036 n=1,043 n=148 n=230 females males A&S BA & Hosp. mgmt 88.4% 84.1% 78.5% 68.9% n=3,263 n=2,376 n=526 n=298

Slide 10: learner characteristics reactive behavior patterns (Long-Dziuban frame) aggressive passive high energy, action-oriented, not low energy, not concerned with independent concerned with approval, speaks approval, prefers to work alone, out freely, gets into resists pressure from authority, confrontational situations non-communicative high energy, action-oriented, Low energy, concerned with dependent concerned with approval, rarely approval, highly sensitive to the expresses negative feelings, feelings of others, compliant and performs at or above ability pleasing

Slide 11: inputs processes outcomes learner characteristics design learning resources professional development

Slide 12: design http://www.esac.org/fdi/rubric/finalsurvey/demorubric.asp http://www.tltgroup.org/

Slide 13: inputs processes outcomes learner characteristics course design learning resources professional development

Slide 14: inputs processes outcomes ub iq uitous te c h nolog ie s us e le a rnin g re pre s e nta tio n le a rning s c o nc e ptua liz a tion s UbiComp Effects (RCET, 2006)

Slide 15: m ultiple re pre s e nta tions a c ros s a wide ra ng e of m e dia UbiComp Effects (RCET, 2006)

Slide 16: GROUP S IZE p e rc e nta g e of re g u la r tim e s p e n t in: c la s s ro om S B C AC in divid ua l 1 2.96 8 .52 s m a ll g rou p s 3 1.31 5 8.7 3 la rg e g rou ps 7 .7 8 0 .00 wh o le c la s s 4 7.95 3 2.7 5 UbiComp Effects (RCET, 2006)

Slide 17: CONCEPTUAL UNDERS TANDING a ve ra g e ra ting a c ro s s c la s s e s h ig h a bility 10 me dium a bility 9.4 low a bility 8.5 s pe c ia l ne e ds 9.3 UbiComp Effects (RCET, 2006)

Slide 18: inputs processes outcomes S OC IAL C OG NITIVE P R E S E NC P R E S E NC E E E DUC ATIONAL E XP E R IE NC E TE AC HING P R E S E NC E Community of Inquiry Model (Garrison, 2006)

Slide 19: Teaching Presence Design & Organization 1. The instructor clearly communicated important course topics. 2. The instructor clearly communicated important course goals. 3. The instructor provided clear instructions on how to participate in course learning activities. 4. The instructor clearly communicated important due dates/time frames for learning activities. Facilitation 5. The instructor was helpful in identifying areas of agreement and disagreement on course topics that helped me to learn. 6. The instructor was helpful in guiding the class towards understanding course topics in a way that helped me clarify my thinking. 7. The instructor helped to keep course participants engaged and participating in productive dialogue. 8. The instructor helped keep the course participants on task in a way that helped me to learn. 9. The instructor encouraged course participants to explore new concepts in this course. 10. Instructor actions reinforced the development of a sense of community among course participants.

Slide 20: Direct Instruction 11. The instructor helped to focus discussion on relevant issues in a way that helped me to learn. 12. The instructor provided feedback helped me understand my strengths and weaknesses. 13. The instructor provided feedback in a timely fashion. Social Presence Affective Expression 14. Getting to know other course participants gave me a sense of belonging in the course. 15. I was able to form distinct impressions of some course participants. 16. Online or web-based communication is an excellent medium for social interaction. Open Communication 17. I felt comfortable conversing through the online medium. 18. I felt comfortable participating in the course discussions. 19. I felt comfortable interacting with other course participants.

Slide 21: Group Cohesion 20. I felt comfortable disagreeing with other course participants while still maintaining a sense of trust. 21. I felt that my point of view was acknowledged by other course participants. 4. Online discussions help me to develop a sense of collaboration. Cognitive Presence Triggering Event 23. Problems posed increased my interest in course issues. 24. Course activities piqued my curiosity. 10. I felt motivated to explore content related questions. Exploration 26. I utilized a variety of information sources to explore problems posed in this course. 27. Brainstorming and finding relevant information helped me resolve content related questions. 28. Online discussions were valuable in helping me appreciate different perspectives.

Slide 22: Integration 29. Combining new information helped me answer questions raised in course activities. 30. Learning activities helped me construct explanations/solutions. 31. Reflection on course content and discussions helped me understand fundamental concepts in this class. Resolution 32. I can describe ways to test and apply the knowledge created in this course. 33. I have developed solutions to course problems that can be applied in practice. 34. I can apply the knowledge created in this course to my work or other non-class related activities.

Slide 23: inputs processes outcomes pedagogy interactions assessment

Slide 24: pedagogy content analysis– “additional comments:” (Swan, Schenker, Lin, Shea & Aviv, 2006) course assignments affect course design instructor instructor instructor /discussion feedback learning

Slide 25: pedagogy teaching presence (Shea, et al., 2003, 2004) SUMMER 2002 SPRING 2003 (n=1140) (n=6088) pe r. le a rn. satisfaction per. learn. satisfaction per. learn. r p r p r p r p design & < < < < organization .64 .01 .59 .01 .64 .01 .60 .01 facilitating < < < < discourse .64 .01 .58 .01 .61 .01 .58 .01 direct < < < < instruction .64 .01 .61 .01 .63 .01 .61 .01

Slide 27: interactions interactivity & generations (Hartman,Moskal & Dziuban, 2005)

Slide 28: interactions social software commenting tools (Track Changes, audio/ video feedback) threaded discussion, chat, email whiteboards / application sharing (Elluminate, Wimba, Learnlinc) blogs wikis distributed classification systems (Digg, Plum, Flickr YouTube)

Slide 30: assessment assessment of online discussion (Schenker, Swan, Arnold & Kuo, 2006) ~criteria criteria ~criteria criteria total posts 40.55 58.30 initial posts/stud 0.83 0.82 total threads 15.50 18.80 initial post length 147.19 145.40 posts/thread 1.65 2.04 responses/stud** 1.52 1.96 thread depth 0.71 0.98 response length* 68.73 70.12 greatest depth 1.90 2.70 messages read** 11.83 23.05 *significant at p>.05; **significant at p>.01

Slide 31: assessment worth being familiar Assessment Types with informal checks observation/dialogue important to know & do quizzes/tests academic prompts enduring understandings performance task/project Understanding By Design, Wiggins & McTighe

Slide 32: assessment: rubrics discussion postings rubrics (Pelz, 2004)

Slide 33: assessment: rubrics article critique rubrics points 0 points 1/2 point 1 point article summary no summary of article weak summary of article, clear summary of article, misses some major points addresses all major points covered in article addresses questions posed doesn't address any of the addresses only some addresses all questions questions questions or some well &/or adds original questions superficially critique links to module content, other doesn't link to either weak links to module &/or strong links to module lit. &/or personal experiences module or own experience own experience topics &/or own experience lesson plan rubrics points 0 points 1/2 point 1 point objectives objectives neither state weak objectives (don't state objectives clearly state what students what students will learn what students will learn or will learn from the lesson and are nor match evaluation don't match rubrics) linked to evaluation rubrics assessment no rubrics rubrics don't match rubrics give specific descriptions of objectives &/or don't give skill levels w/in categories and skill levels w/in categories categories match objectives scope and sequence poorly structured lesson weak lesson that doesn't well structured lesson that fits inappropriate for grade clearly address objectives. curriculum and grade level and clearly level, curriculum & addresses objectives objectives activities no activities weak links between excellent activities encouraging active activities, lesson and learning, & transfer of learning to curriculum other areas of curriculum

Slide 34: assessment: e-portfolios collection of student work providing evidence of learning •linked to learning goals •demonstrating progress over time •multiple media •include reflections on works •programmatic, across courses AAHE ePortfolio clearinghouse http://ctl.du.edu/portfolioclearinghouse/

Slide 35: assessment: student response systems self-assessment as well as data collection, engagement, interactivity

Slide 36: inputs processes outcomes

Slide 37: Karen Swan Research Center for Educational Technology Kent State University kswan@kent.edu www.rcet.org

Slideshare.net (beta)