“Just That They’d Followed The Directions”:Teachers, Wiki Quality, and Wiki Assessment                  Justin Reich      ...
Distributed Collaborative Learning         Communities Project:       Web 2.0 in K-12 Settings• Excellence: How do we make...
Agenda• Motivate the study of wikis• Map out a broad research agenda for  studying wiki usage at scale• Delve into two spe...
Why Study Wikis?• Web 2.0 is Transforming Society• Widespread Adoption in K-12 Settings  – 40% of teachers report using bl...
Distributed Collaborative Learning         Communities Project:       Web 2.0 in K-12 Settings• Excellence: How do we make...
What is good?Quality as 21st Century Skill Development                             Complex       Expert                   ...
Path Diagram of Wiki Research   Classroom                               School Observations                             Le...
How do teachers define wiki           quality?• Why do teachers use wikis?• How do teachers assess quality in wiki  learni...
Survey:  Why do teachers use wikis?• What do you anticipate will be the  benefits for students from using a wiki?  – 193 p...
What do you anticipate will be the benefits for students             from using a wiki? (n=193)                           ...
How do you plan to use your wiki? (n=667)Omitted words: use, wiki, student
Why do teachers use wiki?• Develop technology skills• Develop communication and collaboration  skills• Developing/demonstr...
Wiki quality as opportunities for21st Century Skill Development                           Complex     Expert              ...
How do teachers assess wiki quality?                             68 Interview subjects                                  (n...
How do teachers assess wiki        quality?      Common   Uncommon
Common assessment      categories              Content             Factual information“Did the student discuss the economi...
Common assessment      categories           ParticipationRequired number of contributions during a given time period“We ha...
Common assessment        categories                   Structure    Organization and relation of elements of the wiki pages...
Uncommon assessment     categories         CollaborationThe ways in which students work together effectively              ...
Uncommon assessment         categoriesCommunication of understanding    Use new media design elements to communicate an un...
Overarching Theme  Following Directions  In most classrooms in our study, students are evaluated on           their abilit...
Wiki quality as opportunities         Wiki assessment as               for                       demonstrating21st Century...
Path Diagram of Wiki Research   Classroom                               School Observations                             Le...
Wiki Quality as Opportunities for Students to        Develop 21st Century Skills• Participation   – Do students use wikis ...
The State of Wiki Usage in U.S.           K-12 Schools• What is the distribution of wiki quality?  – Do wikis provide oppo...
Which wikis are in our sample?• Dataset  – All179,851 publicly-viewable education-related wikis    started on the PBworks ...
How did we measure wiki quality?• Sample wiki quality at 7, 14, 30, 60, 100, and  400 days• Two raters independently apply...
Data Analytic Strategy:Multilevel Model for Change
Demographics
What subjects are wikis used for? (n=411)     English / Language Arts                                                     ...
What Grade Levels are K-12 wikis used in?                 (n=411)     K-5                 109     6--8                 118...
How long do K-12 wikis persist?                                                                            (n=411)        ...
Estimated survivor functions for wikis hosted by Title I eligible (n=110) and   33non-Title I eligible schools (n=146).
What is the distribution of wiki           quality?Are great wikis born or made?
Prototypical wiki quality trajectory, controlling for %          FRPL and subject area (n=259)                     25     ...
Prototypical Quality Trajectories for Domain Scale Scores (0-1) of                       Participation, Expert Thinking, C...
Prototypical wiki quality trajectories created in       High-SES (10% FRPL) and Low-SES (90% FRPL)           schools, cont...
Prototypical wiki quality trajectories in subject                          areas, controlling for SES (n=259)             ...
Takeaways• Teachers want to use wikis to develop 21st  century skills  – BUT most teachers assess procedural    compliance...
# of                Time/Scale Web 2.0 ResearchCases                             State Space                           Mod...
Questions for discussion• How can we support teachers in assessing 21st century  skill development in online learning envi...
Back Deck
MrBoyersClass.Pbworks.com                   Page Saves by Day100 90 80 70 60 50                                           ...
Moving average of wiki development                       measured in page savesALLPS   30  20  10   0        0   100   200...
State of Wiki Usage in U.S. K-12 Schools and Just That They Followed the Directions
State of Wiki Usage in U.S. K-12 Schools and Just That They Followed the Directions
State of Wiki Usage in U.S. K-12 Schools and Just That They Followed the Directions
Upcoming SlideShare
Loading in …5
×

State of Wiki Usage in U.S. K-12 Schools and Just That They Followed the Directions

494 views

Published on

Slides

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
494
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
52
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • . In most instances in our data, these common assessment criteria adhere poorly to the domains of 21st century skills while the uncommon criteria had greater alignment.
  • State of Wiki Usage in U.S. K-12 Schools and Just That They Followed the Directions

    1. 1. “Just That They’d Followed The Directions”:Teachers, Wiki Quality, and Wiki Assessment Justin Reich M. Shane Tutwiler Richard Murnane John Willett The State of Wiki Usage in U.S. K-12 Schools Justin Reich Richard Murnane John Willett
    2. 2. Distributed Collaborative Learning Communities Project: Web 2.0 in K-12 Settings• Excellence: How do we make them good?• Equity: Do only certain kids get the good ones?• Analytics: What can we learn about learning from real-time usage data from online learning environments 2
    3. 3. Agenda• Motivate the study of wikis• Map out a broad research agenda for studying wiki usage at scale• Delve into two specific studies – Describe findings about how teachers assess quality in wiki learning environments – Characterize the state of wiki usage in US, K-12 settings
    4. 4. Why Study Wikis?• Web 2.0 is Transforming Society• Widespread Adoption in K-12 Settings – 40% of teachers report using blogs or wikis in instruction (FRSS) – 20% of teachers report having students contribute to blogs or wikis (FRSS)• Democratic, student-centered architecture• New Sources of Data (A Watershed?) – SCalable, Real-time, Individual Behavior and Learning (SCRIBL) data
    5. 5. Distributed Collaborative Learning Communities Project: Web 2.0 in K-12 Settings• Excellence: How do we make them good?• Equity: Do only certain kids get the good ones?• Analytics: What can we learn about learning from SCalable, Real-time, Individual Behavior and Learning (SCRIBL) data maintained by Web 2.0 learning 5
    6. 6. What is good?Quality as 21st Century Skill Development Complex Expert Communi- Thinking cation New Media Literacy
    7. 7. Path Diagram of Wiki Research Classroom School Observations Level SES and Teacher Interviews toUnderstand Wiki Practices Assess How Wiki Develop Wiki Quality Trajectories Measure Wiki Quality Differ by SES and Literature Quality Trajectories TeacherReview of CSCL Attitudes/Practicesand 21st C. Skill Scholarship Initial Wiki User Quantitative Surveys Analysis to Develop Sampling Strategy 7
    8. 8. How do teachers define wiki quality?• Why do teachers use wikis?• How do teachers assess quality in wiki learning environments?
    9. 9. Survey: Why do teachers use wikis?• What do you anticipate will be the benefits for students from using a wiki? – 193 participants in a 2010 online wiki summer camp (out of ~1250)• How do you plan to use your wiki? – 667 wiki creators in summer 2010 (response rate <10%)
    10. 10. What do you anticipate will be the benefits for students from using a wiki? (n=193) 10Omitted words: work, learning, wiki, student
    11. 11. How do you plan to use your wiki? (n=667)Omitted words: use, wiki, student
    12. 12. Why do teachers use wiki?• Develop technology skills• Develop communication and collaboration skills• Developing/demonstrating understanding• Information delivery and course logistics
    13. 13. Wiki quality as opportunities for21st Century Skill Development Complex Expert Communi- Thinking cation New Media Literacy
    14. 14. How do teachers assess wiki quality? 68 Interview subjects (nationwide) 19 Classroom Observations (MA, CA, VA, GA, NH, CT) 14 Randomly- 22 Randomly- 36 Randomly sampled 14 Purposively- sampled sampled 32 Purposively sampled sampled effective teachers teachers Broad cross-section of users wiki users ~25,000 recently edited 411 U.S. publicly- 7 Purposively- K-12 wikis viewable, education sampled urban wiki 1,799 wikis (1% related wikis as of users random sample) September 2009 11 Purposively- 178,851 publicly- sampled viewable, education participants in anrelated wikis hosted on online wiki summer Pbworks.com 2005-8 camp
    15. 15. How do teachers assess wiki quality? Common Uncommon
    16. 16. Common assessment categories Content Factual information“Did the student discuss the economic platform of the country? Did the student include religious data about that area? ”
    17. 17. Common assessment categories ParticipationRequired number of contributions during a given time period“We have a specific number of responses that we expectedfrom student. So even students who weren’t necessarily the most verbose, had to do like five answers in a one week period.”
    18. 18. Common assessment categories Structure Organization and relation of elements of the wiki pages Many teachers required that student included a requisite number of design elements, such as a certain number of pages, paragraphs, or images. Some teachers also made aholistic evaluation of the organization and readability of the site.
    19. 19. Uncommon assessment categories CollaborationThe ways in which students work together effectively on their wiki project“I grade them on their comments to other people too. And I want their comments to other people to be thoughtful and I want them to provoke response.”
    20. 20. Uncommon assessment categoriesCommunication of understanding Use new media design elements to communicate an understanding of academic material that requires building relationships among facts and ideas “When I look at the wiki, I want to see images that in some way bring meaning to what the wiki is about. I want to see links to other websites that bring meaning to what the wiki is about. I like to see students reflecting upon their content in thoughtful ways either in comments or around their own pages.”
    21. 21. Overarching Theme Following Directions In most classrooms in our study, students are evaluated on their ability to complete structured tasks. “Just that theyd followed the directions. It was kind of like, itwasnt high level thinking. But just that they follow the directions and included the information that Id asked them to include.”
    22. 22. Wiki quality as opportunities Wiki assessment as for demonstrating21st Century Skill Development compliance Following 21st C Directions Skills Complex Expert Communi- Thinking cation New Media Literacy
    23. 23. Path Diagram of Wiki Research Classroom School Observations Level SES and Teacher Interviews toUnderstand Wiki Practices Assess How Wiki Develop Wiki Quality Trajectories Measure Wiki Quality Differ by SES and Literature Quality Trajectories TeacherReview of CSCL Attitudes/Practicesand 21st C. Skill Scholarship Initial Wiki User Quantitative Surveys Analysis to Develop Sampling Strategy 23
    24. 24. Wiki Quality as Opportunities for Students to Develop 21st Century Skills• Participation – Do students use wikis to get information? links? do they contribute?• Expert thinking: – Do students use academic content knowledge in wiki activities? – Do students reflect on the process/product?• Complex Communication/Collaboration: – Do students concatenate text on pages? – Do they substantively edit each others work and co-create pages?• New Media Literacy: – Do students use formatting? Wiki Quality Instrument – Do they hyperlink? 25 Questions – Do they embed multimedia? Scale of 1-25
    25. 25. The State of Wiki Usage in U.S. K-12 Schools• What is the distribution of wiki quality? – Do wikis provide opportunities for expert thinking, complex communication, and new media literacy? – Are great wikis born or made?• Do wikis created in affluent schools provide more opportunities for 21c skill development than wikis created in low- income schools?
    26. 26. Which wikis are in our sample?• Dataset – All179,851 publicly-viewable education-related wikis started on the PBworks platform between June 2005 and August of 2008. – Does not include “private” wikis (~70,000)• Sample – Randomly sampled 1,799 wikis (1%) – Coded to identify 411 U.S. based, K-12 wikis – 259 from specific, identifiable public schools • Detailed usage statistics provided by PBworks.com • Demographic school level data from the Common Core of Data (National Center for Education Statistics, 2007-2008) 26
    27. 27. How did we measure wiki quality?• Sample wiki quality at 7, 14, 30, 60, 100, and 400 days• Two raters independently apply wiki quality instrument – All raters must code “training set” of wikis within 1.5 points of master coders – Weekly meetings while coding to discuss categories, difficult cases, etc.• Third rater reconciles disagreements 27
    28. 28. Data Analytic Strategy:Multilevel Model for Change
    29. 29. Demographics
    30. 30. What subjects are wikis used for? (n=411) English / Language Arts 120 Social Studies 70 Science 61Computer Science/ Technology 60 Math 45 Library 26 Art 22 Contained Elementary 20 Modern FL 10 Health/PE 8 Business 6 ESL 5 Classics 4 Education 2 0 20 40 60 80 100 120 140
    31. 31. What Grade Levels are K-12 wikis used in? (n=411) K-5 109 6--8 118 9--12 180Higher Ed 8Unknown 83 0 50 100 150 200
    32. 32. How long do K-12 wikis persist? (n=411) Estimated Seconds Days 1 Lifetimes All PBworks 0.9 25% 250 <1Estimated Survival Probability 0.8 50% 123,613 1.4 0.7 75% 5,282,874 61.1 0.6 K-12 Wikis 25% 2,721 <1 0.5 50% 763,195 8.8 0.4 75% 12,590,074 145.7 0.3 All PBWorks Wikis 0.2 K-12 Wikis 0.1 0 0 20000000 40000000 60000000 80000000 10000000 (231) (463) (694) (926) (1157) Time in seconds (days)
    33. 33. Estimated survivor functions for wikis hosted by Title I eligible (n=110) and 33non-Title I eligible schools (n=146).
    34. 34. What is the distribution of wiki quality?Are great wikis born or made?
    35. 35. Prototypical wiki quality trajectory, controlling for % FRPL and subject area (n=259) 25 20Wiki Quality Score 15 10 5 0 0 100 200 300 400 500 Days
    36. 36. Prototypical Quality Trajectories for Domain Scale Scores (0-1) of Participation, Expert Thinking, Complex Communication, and New Media Literacy, controlling for SES (n=259). 1 0.9 0.8Scaled Domain Score 0.7 0.6 0.5 0.4 Participation 0.3 Expert Thinking 0.2 NML 0.1 Collaboration 0 0 100 200 300 400 500 Days
    37. 37. Prototypical wiki quality trajectories created in High-SES (10% FRPL) and Low-SES (90% FRPL) schools, controlling subject area (n=259) 25 20Wiki Quality Score 15 10 High-SES 5 Low SES 0 0 100 200 300 400 500 Days
    38. 38. Prototypical wiki quality trajectories in subject areas, controlling for SES (n=259) 25 20Wiki Quality Score 15 10 Social Studies English 5 Computer Science Science Math 0 0 100 200 300 400 500 Days
    39. 39. Takeaways• Teachers want to use wikis to develop 21st century skills – BUT most teachers assess procedural compliance in wikis• Wikis are widely adopted in K-12 setttings – BUT most wikis are teacher- centered, content-delivery devices – AND are more persistently and efficaciously used in wealthier schools• Great wikis are born; initial norms matter
    40. 40. # of Time/Scale Web 2.0 ResearchCases State Space Modeling1,000K Usage Simulations Statistics100K Semantic Analysis 10K Surveys1,000 Content Interviews Analysis 100 Discursive 10 Analysis Biometric Design Observational 1 Analysis Research Research Seconds Days Weeks Months Years 40 Duration of data collection and capture
    41. 41. Questions for discussion• How can we support teachers in assessing 21st century skill development in online learning environment?• What kinds of targeted interventions would support teachers in using wikis to develop 21st century skills?• What kinds of actionable advice can we give teachers about wikis design, knowing that high quality wikis start at high levels of quality?• What kinds of targeted interventions in schools serving low-income students would close the “second digital divide” of usage?• How can a national perspective on wiki usage help situate and contextualize local studies?• How can we leverage other forms of SCRIBL data to characterize Web 2.0 usage at scale?
    42. 42. Back Deck
    43. 43. MrBoyersClass.Pbworks.com Page Saves by Day100 90 80 70 60 50 Page Saves 40 30 20 10 0 0 50 100 150 200 250 300 Days
    44. 44. Moving average of wiki development measured in page savesALLPS 30 20 10 0 0 100 200 300 400 500 600 700 800 900 day

    ×