Uconn Coiro Assessment 2008


Published on

Having the skills and strategies to read, learn from, and communicate with the Internet will play a central role in our students’ success in an information age. But how can we best measure these new literacies? This session explores some of the challenges associated with developing valid and reliable measures of the complex literacy strategies and dispositions required to search for, comprehend, and respond to information on the Internet. The presenter will first share task examples and student responses from several assessments developed to measure online reading comprehension and communication skills. Then, conversation will turn to a number of important issues to consider when developing online literacy assessments that are not only psychometrically sound, but also useful to both researchers and classroom teachers. Participants will have an opportunity to share their own thoughts about how we might rethink the ways in which we evaluate the skills, strategies, and dispositions associated with reading and learning online.

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Uconn Coiro Assessment 2008

  1. 1. How Does The Internet Impact Our Thinking About Effective Literacy Assessment? Julie Coiro, Ph.D. University of Rhode Island jcoiro@snet.net
  2. 2. Goals for today’s conversation Understand the issues that cause us to think differently about literacy assessment View examples of assessments developed to measure the new literacies of online reading comprehension Explore some of the concerns and challenges related to measuring online literacy and learning Dialogue & reflect on connections to issues in your own teaching / learning context
  3. 3. The New Literacies Research Team QuickTimeª and a TIFF (Uncompressed) decompressor are needed to see this picture. Supported by: • Ray and Carole Neag • The Carnegie Corporation of New York • U.S. Department of Education • The National Science Foundation • North Central Educational Research Lab • PBS • The Annenberg Foundation QuickTimeª and a • William and Flora Hewlett Foundation TIFF (Uncompressed) decompressor are needed to see this picture. • Bill and Melinda Gates Foundation • Australian Council of Educational Research
  4. 4. A New Literacies Perspective considers the Internet as this generation’s defining technology for information, communication, and especially for learning. As a result, several important differences require our attention:
  5. 5. New skills and strategies are required to comprehend and learn from information on the Internet You begin by identifying an important question New ways of locating information New reasons for critically evaluating the information New contexts for synthesizing information to answer your questions New ways of communicating the answers to others Leu, Kinzer, Coiro, and Cammack (2004)
  6. 6. New dispositions (attitudes, beliefs, mindsets) are also required Initiative, confidence, self-direction, critical stance, adaptability, emotional resilience, persistence, flexibility, divergent and convergent thinking, personal productivity, leadership, social responsibility, teamwork, curiosity, motivation, openness, … (e.g., American Library Association; Partnership for 21st Century; New Literacies Research Team; Johnston, 2005; Tsai & Tsai, 2003; Whipp & Chiarelli, 2004)
  7. 7. Processes may be even more important than the products (especially for informing instruction) Product: I can’t find it. Process…
  8. 8. Developmental differences appear across each phase of the online inquiry process (and are particular to tasks) Lower Average Higher Locating .com strategy Whole phrase Uses keywords Communicating Copies the long Toggles back and Copy/pastes address by hand forth to type address address with mouse shortcut Evaluating Struggles to locate Judges reliability Examines author’s reliability “About Author” based on length of level of page coverage integrity/expertise Coiro, 2007 Do our assessments adequately capture ability and growth across each of these stages and across a range of tasks in ways that can inform instruction?
  9. 9. Other issues to consider when assessing online literacies Students often work collaboratively in groups or seek help from others online, yet we continue to measure reading performance individually and without online assistance New assessments are needed to evaluate group collaboration and productivity as well as how readers seek assistance from a globally networked community
  10. 10. Other issues to consider when assessing online literacies Today’s authentic problems often require interdisciplinary connections, yet we most often measure reading performance outside of these authentic problem-solving contexts Valid new assessments of online reading comprehension should situate reading tasks within a range of relevant content area/interdisciplinary information challenges Valid new assessments should also integrate the information and communication tools used in the workforce and in students’ daily lives (“real” blogs, wikis, instant message, Flickr sites, etc.) QuickTimeª and a TIFF (Uncompressed) decompressor are needed to see this picture.
  11. 11. The ultimate challenge to consider when assessing online literacies… Online texts, tools, and learning environments will continue to rapidly change… We need to be prepared to continually reconsider, change, and expand what it means to be a skilled online reader. Thus… our measures need to continually change and expand… Which makes it difficult to replicate the use of the same measure over time (establishing reliability) or the same tool/environment over time (ecological validity)
  12. 12. To summarize, at least eight issues impact our thinking about literacy assessment in a digital age New literacy skills and strategies New dispositions New processes to add to products New developmental differences to capture New group measures to add to individual measures New authentic problems to solve across disciplines New tools and technologies for school, work, and daily life Rapid and continual change
  13. 13. Your Turn.. Connecting with you Are you noticing challenges to assessment with and of technology use in your discipline? If so, can you share some examples? Possible solutions?
  14. 14. So, what have we learned in the past three years about designing assessments to measure online reading comprehension?
  15. 15. Assessments of Online Reading Comprehension ORCA-Instant Message ORCA-Blog ORCA Scenarios I and II ORCA Iditarod ORCA Iditarod-Revised Survey of Internet Use and Online Reading ORCA Growth-Curve Measures Formative Assessments of Online Strategy Use ISSUES: Environments; Items; Scoring Systems; Group or Individual; Protocols for Administering & Conducting; “Plan B or C”; Managing Student Data; Analysis; Making Teacher Friendly
  16. 16. ORCA Instant Message: (2004) Homelessness and The Solar System
  17. 17. Psychometric Properties ORCA-Instant Message (Solar System) N= 36 (Three ability levels; N=12 at each level) Admin. individually (1 student, 2-3 researchers - very “real-time” and time consuming!) Content Validity Post-Test (4 tasks, 12 items) Principal components analysis supported one composite explaining 66% of the variance Internal Consistency: Cronbach’s Alpha=.91
  18. 18. ORCA-Blog: Human Body Systems (2005)
  19. 19. Psychometric Properties ORCA-Blog (Human Body Systems) N=89 Admin to group - post to blog simultaneously Content Validity Post-Test (3 tasks, 10 items) Principal components analysis supported one composite explaining 59.2% of the variance Internal Consistency: Cronbach’s Alpha=.84
  20. 20. ORCA-Scenarios I and II (Coiro, 2006)Carbon Monoxide Poisoning, Asthma, and the Respiratory System
  21. 21. Related Variables to Consider… Internet Use Prior Knowledge
  22. 22. Related Variables… Dispositions Open-ended questions: (a) How approach; (b) How respond; (c): Self-efficacy • What is easiest for you about using the Internet for research? • What is hardest for you about using the Internet for research? • Can you think of a time when you had trouble finding something using the Internet? How do you feel when this happens? How long do you keep trying before you give up? • What do you know about using the Internet effectively that some kids your age might not know?
  23. 23. Psychometric Properties ORCA-Scenarios I and II N=120; Group administered (record with Camtasia) ORCA-Scenario 1 Content Validity (3 tasks, 20 items) • Principal components analysis supported one composite explaining 51.7% of the variance • Factor Loadings: Task 1(.617); Task 2(.755); Task 3(.775) Internal Consistency: Cronbach’s Alpha=.92 Inter-Rater Reliablity: k=.96 ORCA-Scenario II Content Validity (3 tasks, 20 items) • Principal components analysis supported one composite explaining 44.1% of the variance • Factor Loadings: Task 1(.683); Task 2(.751); Task 3(.541) Internal Consistency: Cronbach’s Alpha=.91 Inter-Rater Reliablity: k=.96
  24. 24. ORCA-Iditarod (2007)
  25. 25. ORCA-Iditarod Task 1 Formulate Question/Locate Task (and Evaluate Accuracy in Task 2)
  26. 26. ORCA-Iditarod Task 4 Evaluate Bias/Perspective and Reliability Big idea Smaller idea www.newliteracies.uconn.edu/iesproject
  27. 27. ORCA-Iditarod Task 5 Synthesize from Multiple Sources and… www.newliteracies.uconn.edu/iesproject
  28. 28. ORCA-Iditarod Task 5 and… Communicate Your Opinion
  29. 29. Psychometric Properties ORCA-Iditarod Content Validity Post-Test (5 Tasks, 44 items) Principal components analysis supported one composite explaining 53% of the variance with separate task loadings ranging from .613-.795 Task 1 (.659); Task 2 (.786); Task 3 (.795); Task 4 (.613); Task 5 (.766) Internal Consistency: Cronbach’s Alpha=.88 Inter-rater reliability: Cohen’s Kappa = .94
  30. 30. ORCA-Iditarod Revised (2007-08)
  31. 31. Survey of Internet Use and Online Reading (2006-2007)
  32. 32. Psychometric Properties Survey of Internet Use and Online Reading Forced Response Items: (5) P values from .31 to .86 indicating item difficulty was sufficient 3 items “very good” and two items “good” for test discrimination Open-Ended Items: (4) Internal Consistency: Cronbach’s Alpha=.69 Inter-rater reliability: Cohen’s Kappa = .87
  33. 33. ORCA Growth Curve Assessment Develop a measure that reflects growth of online reading comprehension skills and strategy use over time Decrease scoring and analysis time Help directly inform instruction 8 minutes online searching (quick scoring scheme) 12 minutes multiple choice (quick scoring scheme) Administered 5 times over the course of the year and follows the same structure each time
  34. 34. ORCA Growth Curve Assessment
  35. 35. Formative Assessment of Students’ Emerging Knowledge of Internet Strategies (2006-2008) • Create many opportunities for students to practice all three dimensions of strategic knowledge. • Be sure to measure all three dimensions when determining how well your students read and learn on the Internet.
  36. 36. Summary of Assessments of Online Reading Comprehension ORCA-Instant Message ORCA-Blog ORCA Scenarios I and II ORCA Iditarod ORCA Iditarod-Revised Survey of Internet Use and Online Reading ORCA Growth-Curve Measures Formative Assessments of Online Strategy Use
  37. 37. Overall, what are we learning from assessments of online reading? Patterns of effective & ineffective online strategies Generating questions/keywords: location and quality of keywords Searching: skimming for relevancy and flexible strategy use Critical Evaluation: surface level understanding versus application; differences between prompted and unprompted applications Synthesis: incorporating multiple aspects of what’s “best” Communication: efficiency factor; varying complexities depending on the tool/interface used Important variables for online reading comprehension & learning Standardized reading scores: correlated with online reading? Prior Knowledge: challenging our assumptions Dispositions: possibly having an influence on online reading patterns New literacies: critical new online skills & strategies not measured in current assessments of reading comprehension
  38. 38. Real-time videos of Online Reading Comprehension http://www.newliteracies.uconn.edu/coirodissertation/ http://www.newliteracies.uconn.edu/reading.html http://www.newliteracies.uconn.edu/iesproject/videos/
  39. 39. Some challenges with online assessments… Developing tasks when websites come and go Multiple paths to the correct (or incorrect) answer Implications of “electronic scaffolds” Differences in search engines and results Issues of capturing and watching process data Using real-time video for each student Time-consuming scoring The nature of online reading scenarios prompts new ways of thinking about missing data
  40. 40. Your Turn.. Connecting with you What successes have you had with learning how to capture, measure, analyze, and apply information about your students’ proficiency with learning with technology? Other questions…comments?
  41. 41. Thank you for the conversation! For more information: Julie Coiro University of Rhode Island Jcoiro@snet.net http://www.uri.edu/hss/education/faculty/coiro.html New Literacies Research Team http://www.newliteracies.uconn.edu