Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Are We There Yet

1,464 views

Published on

by Dr. Karen Swan
Dr. Swan will discuss tools and techniques of assessing the impact of technology on learning, beginning with asking the right questions. Good questions, she argues, specify not just outcomes, but also inputs and, most importantly, learning processes. Each of these will be discussed in terms of categories and measures for guiding assessment.

Published in: Technology, Education
  • Be the first to comment

Are We There Yet

  1. 1. Karen Swan Kent State University Research Center for Educational Technology Assessing the Impact of Technology on Learning
  2. 2. The Great Media Debate Kozma VS Clark
  3. 3. ? What is the question?
  4. 4. outcomes processes inputs
  5. 5. satisfaction retention success achievement proficiencies performance outcomes processes inputs
  6. 6. Understanding By Design , Wiggins & McTighe achievement enduring understandings important to know & do worth being familiar with
  7. 7. satisfaction retention success achievement proficiencies performance outcomes processes inputs
  8. 8. learner characteristics design learning resources professional development outcomes processes inputs
  9. 9. learner characteristics F2F, E, M factors influencing success (Dziuban & Moskal, 2006) 85.9% n=11,286 85.8% n=6,460 91.5% n=2,079 72.7% n=378 86.7% n=2,369 86.5% n=5,639 74.8% n=821 94.1% n=1,036 89.1% n=1,043 64.7% n=148 79.6% n=230 88.4% n=3,263 84.1% n=2,376 68.9% n=298 78.5% n=526 Arts & Sciences, Business Admin., Hospitality Mgmt. Education Engineering Health & Pub. Affairs O females males A&S BA & Hosp. mgmt F2F E, M, O E, M F2F Overall
  10. 10. learner characteristics reactive behavior patterns (Long-Dziuban frame) Low energy, concerned with approval, highly sensitive to the feelings of others, compliant and pleasing high energy, action-oriented, concerned with approval, rarely expresses negative feelings, performs at or above ability dependent low energy, not concerned with approval, prefers to work alone, resists pressure from authority, non-communicative high energy, action-oriented, not concerned with approval, speaks out freely, gets into confrontational situations independent passive aggressive
  11. 11. learner characteristics design learning resources professional development outcomes processes inputs
  12. 12. design http://www.esac.org/fdi/rubric/finalsurvey/demorubric.asp http://www.tltgroup.org/
  13. 13. learner characteristics course design learning resources professional development outcomes processes inputs
  14. 14. learning ubiquitous technologies learning UbiComp Effects (RCET, 2006) outcomes processes inputs use representations conceptualizations
  15. 15. multiple representations across a wide range of media UbiComp Effects (RCET, 2006)
  16. 16. GROUP SIZE UbiComp Effects (RCET, 2006) 32.75 47.95 whole class 0.00 7.78 large groups 58.73 31.31 small groups 8.52 12.96 individual SBCAC regular classroom percentage of time spent in:
  17. 17. CONCEPTUAL UNDERSTANDING UbiComp Effects (RCET, 2006) 9.3 special needs 8.5 low ability 9.4 medium ability 10 high ability average rating across classes
  18. 18. EDUCATIONAL EXPERIENCE Community of Inquiry Model (Garrison, 2006) TEACHING PRESENCE PRESENCE SOCIAL COGNITIVE PRESENCE outcomes processes inputs
  19. 19. Teaching Presence <ul><li>Design & Organization </li></ul><ul><li>1. The instructor clearly communicated important course topics. </li></ul><ul><li>2. The instructor clearly communicated important course goals. </li></ul><ul><li>3. The instructor provided clear instructions on how to participate in course learning activities. </li></ul><ul><li>4. The instructor clearly communicated important due dates/time frames for learning activities. </li></ul>Facilitation 5. The instructor was helpful in identifying areas of agreement and disagreement on course topics that helped me to learn. 6. The instructor was helpful in guiding the class towards understanding course topics in a way that helped me clarify my thinking. 7. The instructor helped to keep course participants engaged and participating in productive dialogue. 8. The instructor helped keep the course participants on task in a way that helped me to learn. 9. The instructor encouraged course participants to explore new concepts in this course. 10. Instructor actions reinforced the development of a sense of community among course participants.
  20. 20. <ul><li>Direct Instruction </li></ul><ul><li>11. The instructor helped to focus discussion on relevant issues in a way that helped me to learn. </li></ul><ul><li>12. The instructor provided feedback helped me understand my strengths and weaknesses. </li></ul><ul><li>13. The instructor provided feedback in a timely fashion. </li></ul>Social Presence Affective Expression 14. Getting to know other course participants gave me a sense of belonging in the course. 15. I was able to form distinct impressions of some course participants. 16. Online or web-based communication is an excellent medium for social interaction. Open Communication 17. I felt comfortable conversing through the online medium. 18. I felt comfortable participating in the course discussions. 19. I felt comfortable interacting with other course participants.
  21. 21. <ul><li>Group Cohesion </li></ul><ul><li>20. I felt comfortable disagreeing with other course participants while still maintaining a sense of trust. </li></ul><ul><li>21. I felt that my point of view was acknowledged by other course participants. </li></ul><ul><li>Online discussions help me to develop a sense of collaboration. </li></ul><ul><li>Cognitive Presence </li></ul><ul><li>Triggering Event </li></ul><ul><li>23. Problems posed increased my interest in course issues. </li></ul><ul><li>24. Course activities piqued my curiosity. </li></ul><ul><li>I felt motivated to explore content related questions. </li></ul><ul><li>Exploration </li></ul><ul><li>26. I utilized a variety of information sources to explore problems posed in this course. </li></ul><ul><li>27. Brainstorming and finding relevant information helped me resolve content related questions. </li></ul><ul><li>28. Online discussions were valuable in helping me appreciate different perspectives. </li></ul>
  22. 22. <ul><li>Integration </li></ul><ul><li>29 . Combining new information helped me answer questions raised in course activities. </li></ul><ul><li>30. Learning activities helped me construct explanations/solutions. </li></ul><ul><li>31. Reflection on course content and discussions helped me understand fundamental concepts in this class. </li></ul>Resolution 32. I can describe ways to test and apply the knowledge created in this course. 33. I have developed solutions to course problems that can be applied in practice. 34. I can apply the knowledge created in this course to my work or other non-class related activities.
  23. 23. pedagogy interactions assessment outcomes processes inputs
  24. 24. content analysis– “additional comments:” (Swan, Schenker, Lin, Shea & Aviv, 2006) pedagogy course design course assignments instructor feedback instructor instructor /discussion affect learning
  25. 25. SUMMER 2002 (n=1140) SPRING 2003 (n=6088) satisfaction per. learn. per. learn. teaching presence (Shea, et al., 2003, 2004) satisfaction per. learn. pedagogy < .01 .61 < .01 .63 < .01 .61 < .01 .64 direct instruction < .01 .58 < .01 .61 < .01 .58 < .01 .64 facilitating discourse < .01 .60 < .01 .64 < .01 .59 < .01 .64 design & organization p r p r p r p r
  26. 26. pedagogy interactions assessment outcomes processes inputs
  27. 27. interactions interactivity & generations (Hartman,Moskal & Dziuban, 2005)
  28. 28. social software commenting tools (Track Changes, audio/ video feedback) threaded discussion, chat, email whiteboards / application sharing (Elluminate, Wimba, Learnlinc) blogs wikis distributed classification systems (Digg, Plum, Flickr YouTube) interactions
  29. 29. pedagogy interactions assessment outcomes processes inputs
  30. 30. assessment *significant at p>.05; **significant at p>.01 assessment of online discussion (Schenker, Swan, Arnold & Kuo, 2006) 2.70 1.90 greatest depth 0.98 0.71 thread depth 2.04 1.65 posts/thread 18.80 15.50 total threads 58.30 40.55 total posts criteria ~criteria 23.05 11.83 messages read** 70.12 68.73 response length* 1.96 1.52 responses/stud** 145.40 147.19 initial post length 0.82 0.83 initial posts/stud criteria ~criteria
  31. 31. enduring understandings important to know & do worth being familiar with assessment Understanding By Design , Wiggins & McTighe Assessment Types informal checks observation/dialogue quizzes/tests academic prompts performance task/project
  32. 32. assessment: rubrics discussion postings rubrics (Pelz, 2004)
  33. 33. lesson plan rubrics article critique rubrics assessment: rubrics strong links to module topics &/or own experience weak links to module &/or own experience doesn't link to either module or own experience links to module content, other lit. &/or personal experiences addresses all questions well &/or adds original critique addresses only some questions or some questions superficially doesn't address any of the questions addresses questions posed clear summary of article, addresses all major points covered in article weak summary of article, misses some major points no summary of article article summary 1 point 1/2 point 0 points points excellent activities encouraging active learning, & transfer of learning to other areas of curriculum weak links between activities, lesson and curriculum no activities activities well structured lesson that fits curriculum and grade level and clearly addresses objectives weak lesson that doesn't clearly address objectives. poorly structured lesson inappropriate for grade level, curriculum & objectives scope and sequence rubrics give specific descriptions of skill levels w/in categories and categories match objectives rubrics don't match objectives &/or don't give skill levels w/in categories no rubrics assessment objectives clearly state what students will learn from the lesson and are linked to evaluation weak objectives (don't state what students will learn or don't match rubrics) objectives neither state what students will learn nor match evaluation rubrics objectives 1 point 1/2 point 0 points points
  34. 34. AAHE ePortfolio clearinghouse http://ctl.du.edu/portfolioclearinghouse/ <ul><li>collection of student work providing evidence of learning </li></ul><ul><ul><li>linked to learning goals </li></ul></ul><ul><ul><li>demonstrating progress over time </li></ul></ul><ul><ul><li>multiple media </li></ul></ul><ul><ul><li>include reflections on works </li></ul></ul><ul><ul><li>programmatic, across courses </li></ul></ul>assessment: e-portfolios
  35. 35. assessment: student response systems self-assessment as well as data collection, engagement, interactivity
  36. 36. outcomes processes inputs
  37. 37. Karen Swan Kent State University [email_address] www.rcet.org Research Center for Educational Technology

×