Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Are Great Wikis Born or Made? Are Students Just Posting in the Same Place?

3,440 views

Published on

A pair of papers for AERA #2012 derived from research from the DCLC project (edtechresearcher.org).

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

Are Great Wikis Born or Made? Are Students Just Posting in the Same Place?

  1. 1. Are Great Wikis Born or Made?Evaluating Quality Trajectories in Wiki Learning EnvironmentsJust Posting in the Same Place?Evaluating Student Collaboration in Wiki Learning Environments Justin Reich @bjfr Richard Murnane edtechresearcher.org John Willett
  2. 2. Distributed Collaborative Learning Communities Project: Web 2.0 in K-12 Settings• Excellence: How do we make them good?• Equity: Do only certain kids get the good ones?• Analytics: What can we learn about learning from real-time usage data from online learning environments 2
  3. 3. Path Diagram of Wiki Research Classroom Observations and Teacher Interviews toUnderstand Wiki Practices Correlate wiki Develop Wiki Measure Wiki quality with Quality Literature Quality school level TrajectoriesReview of CSCL SESand 21st C. Skill Scholarship Initial Quantitative Analysis to Develop Sampling Strategy 3
  4. 4. Path Diagram of Wiki Research Classroom Observations and Teacher Interviews toUnderstand Wiki Practices Correlate wiki Develop Wiki Measure Wiki quality with Quality Literature Quality school level TrajectoriesReview of CSCL SESand 21st C. Skill Scholarship Initial Quantitative Analysis to Develop Sampling Strategy 4
  5. 5. Low Income Mid to High Schools Income Schools (n=117) (n=133)Failed or Trial Wiki 50% 30%Teacher-Content Wiki 34% 35%Individual Student-Owned 15% 35%WikiCollaborative Student-Owned 2% 1%WikiMedian Lifetime 6 days 33 days
  6. 6. Path Diagram of Wiki Research Classroom Observations and Teacher Interviews toUnderstand Wiki Practices Correlate wiki Develop Wiki Measure Wiki quality with Quality Literature Quality school level TrajectoriesReview of CSCL SESand 21st C. Skill Scholarship Initial Quantitative Analysis to Develop Are great wikis Sampling Strategy born or made? 6
  7. 7. “Time is Precious”• Reimann (2009) – Collaborative learning unfolds over time • Substantive need – Online learning environments track data in real time • Methodological opportunity
  8. 8. Path Diagram of Wiki Research Classroom Observations and Teacher Interviews toUnderstand Wiki Practices Correlate wiki Develop Wiki Measure Wiki quality with Quality Literature Quality school level TrajectoriesReview of CSCL SESand 21st C. Skill Scholarship Initial Quantitative How did we measure Analysis to Develop collaborative behaviors on Sampling Strategy wikis, and what did we find? 8
  9. 9. Theoretical debates in wikis and collaboration (IJCSCL) Are students meaningfully Or just posting in the collaborating? same place? Glassman and Kang (2011): Dohn (2009): Web 2.0 tools wikis not only enable are theoretically well-suited students to participate in for collaborative knowledge knowledge building, but production practices, but the allow the enactment of educational logics of Dewey’s vision individual assessment prevent these practices from being realized.
  10. 10. The Plan• Explain our research design• Explain how we model wiki quality over time – Answer the question: Are Great Wikis Born or Made?• Explain in detail how we measure collaboration – Answer the question: Who’s right: Dohn or Glassman & Kang?
  11. 11. RESEARCH DESIGN
  12. 12. Which wikis are in my dataset?• All 179,851 publicly-viewable education-related wikis started on the PBworks platform between June 2005 and August of 2008.• Does not include “private” wikis (~70,000) 179,851 PBWorks Wikis 12
  13. 13. Which wikis are in my sample?• Randomly sampled 1,799 wikis (1%)• Coded to identify 406 U.S. based, K-12 wikis• 255 from specific, identifiable public schools – Detailed usage statistics provided by PBworks.com – School level SES data from the Common Core of Data (National Center for Education Statistics, 2007-2008) – 255 wikis are from 41 U.S. states 179,851 PBWorks Wikis 1,799 406 255 1% U.S. K-12 Public Random Wikis School Sample 13
  14. 14. How should we measure wiki quality?
  15. 15. Wiki quality as opportunities for21st Century Skill Development Complex Expert Communi- Thinking cation New Media Literacy
  16. 16. Wiki Quality Instrument• Information Consumption (2 items): – Do students use wikis to get information? links?• Participation (4 items): 24 Questions – Do they contribute? Do they own pages? Wiki Quality Scale• Expert thinking (5 items): of 0-24 – Do students use academic content knowledge in wiki activities? – Do students reflect on the process/product?• Complex Communication/Collaboration (7 items): – Do students concatenate text on pages? – Do they substantively edit each others work and co-create pages?• New Media Literacy (6 items): – Do students use formatting? – Do they hyperlink? Embed media?
  17. 17. How did we measure wiki quality?• Sample wiki quality at 7, 14, 30, 60, 100, and 400 days• Two raters independently apply wiki quality instrument – All raters must code “training set” of wikis within 1.5 points of master coders – Weekly meetings while coding to discuss categories, difficult cases, etc.• Third rater reconciles disagreements 17
  18. 18. Data Analytic Strategy: Multilevel Model for Change• We should have time nested within wikis and use the MMFC to model quality trajectories• What functional form should we use to represent time?• The decision that we make in setting a functional form will answer the research question about time!
  19. 19. Data Analytic Strategy:Multilevel Model for Change
  20. 20. Selected Empirical Growth Plots
  21. 21. Empirical Velocity Plot of Wiki Quality Mean Difference in Score/Days 0.40.35 0.30.25 0.20.15 0.10.05 0 0 100 200 300 400 500
  22. 22. Data Analytic Strategy:Multilevel Model for Change
  23. 23. Comparing Fit Statistics Linear Quadratic LogIntercept 2.8371*** 2.5647*** 1.2073***Time 0.005629*** 0.02078***Time2 -0.00004***Ln(Time) 0.6614***-2LL 5542.2 5468.7 5263.0AIC 5554.2 5482.7 5275.0BIC 5578.3 5510.9 5299.1
  24. 24. Data Analytic Strategy: Multi-level Poisson RegressionSinger and Willett (2008); Rabe-Hesketh and Skrondal (2008)
  25. 25. Prototypical wiki quality trajectory (n=406)Composite Wiki Quality Score 20 15 10 5 0 0 100 200 300 400 Days
  26. 26. Implications• Early wiki quality is a strong predictor of later wiki quality- Great Wikis are Born – Hypothesis: Norms developed early on in wiki learning communities are powerful in shaping later use – FOR TEACHERS: Therefore, wiki design deserves critical attention – FOR RESEARCHERS: Further qualitative studies of wiki using communities should ensure that observers can study wikis from the start of the process
  27. 27. BACK TO COLLABORATION
  28. 28. Theoretical debates in wikis and collaboration (IJCSCL) Are students meaningfully Or just posting in the collaborating? same place? Glassman and Kang (2011): Dohn (2009): Web 2.0 tools wikis not only enable are theoretically well-suited students to participate in for collaborative knowledge knowledge building, but production practices, but the allow the enactment of educational logics of Dewey’s vision individual assessment prevent these practices from being realized.
  29. 29. Path Diagram of Wiki Research Classroom Observations and Teacher Interviews toUnderstand Wiki Practices Measure Wiki Literature QualityReview of CSCLand 21st C. Skill Scholarship Initial Quantitative Analysis to Develop Sampling Strategy 29
  30. 30. Complex CommunicationConcatenation Do multiple students add discrete sections of text to the same page?Copyediting Does at least one student copyedit text created by another student?Co-Construction Does at least one student substantively edit text created by another student?Commenting Does at least one student comment upon another student’s work on the wiki?Discussion Do students respond to each others’ comments for at least four conversational turns?Scheduling Do students schedule meetings or tasks?Planning Do students plan for future work?
  31. 31. ConcatenationConcatenation Do multiple students add discrete sections of text to the same page?Copyediting Does at least one student copyedit text created by another student?Co-Construction Does at least one student substantively edit text created by another student?Commenting Does at least one student comment upon another student’s work on the wiki?Discussion Do students respond to each others’ comments for at least four conversational turns?Scheduling Do students schedule meetings or tasks?Planning Do students plan for future work?
  32. 32. Complex CommunicationConcatenation Do multiple students add discrete sections of text to the same page?Copyediting Does at least one student copyedit text created by another student?Co-Construction Does at least one student substantively edit text created by another student?Commenting Does at least one student comment upon another student’s work on the wiki?Discussion Do students respond to each others’ comments for at least four conversational turns?Scheduling Do students schedule meetings or tasks?Planning Do students plan for future work?
  33. 33. Co-ConstructionConcatenation Do multiple students add discrete sections of text to the same page?Copyediting Does at least one student copyedit text created by another student?Co-Construction Does at least one student substantively edit text created by another student?Commenting Does at least one student comment upon another student’s work on the wiki?Discussion Do students respond to each others’ comments for at least four conversational turns?Scheduling Do students schedule meetings or tasks?Planning Do students plan for future work?
  34. 34. Complex CommunicationConcatenation Do multiple students add discrete sections of text to the same page?Copyediting Does at least one student copyedit text created by another student?Co-Construction Does at least one student substantively edit text created by another student?Commenting Does at least one student comment upon another student’s work on the wiki?Discussion Do students respond to each others’ comments for at least four conversational turns?Scheduling Do students schedule meetings or tasks?Planning Do students plan for future work?
  35. 35. Prototypical wiki trajectory for Complex 20 Communication WQI subdomain (n=406)Complex Communication Score 15 10 5 0 0 50 100 150 200 250 300 350 400 Days
  36. 36. Selected Empirical Growth PlotsComposite Wiki Quality Score Rate Day
  37. 37. Distribution of Complex Communication subdomain score in 406 U.S., K-12 wikis on final occasion of measurementComplex Percentage Frequency CountCommunication Score0 88.92 3611 6.40 262 1.48 63 .25 14 1.48 65 .74 36 0 07 .74 3
  38. 38. Distribution of collaborative behaviors in 406 U.S., K-12 wikis on final occasion of measurement. Percentage Frequency CountConcatenation 5.91 24Copyedit 1.72 7Co-construction 4.43 18Comment 6.4 26Discussion 2.22 9Planning 2.46 10Scheduling 1.72 7
  39. 39. Wiki Quality Subdomain Scores7.0 (Averages for each WQI Level) (n=255)6.0 5.0 4.0 3.0 2.0 Information Consumption Participation 1.0 Expert Thinking 0.0 NewMedia Literacy 0 Complex Communication 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
  40. 40. SourceForge• 70% of projects have only 1 developer,• 14% of projects have 2 developers• 14% of projects have 3-9 developers• 2% of projects have more than 10 developers
  41. 41. Theoretical debates in wikis and collaboration (IJCSCL) Are students meaningfully Or just posting in the collaborating? same place? Glassman and Kang (2011): Dohn (2009): Web 2.0 tools wikis not only enable are theoretically well-suited students to participate in for collaborative knowledge knowledge building, but production practices, but the allow the enactment of educational logics of Dewey’s vision individual assessment prevent these practices from being realized.
  42. 42. Implications• Empirical: Few wikis provide opportunities for student collaborative writing – Very little copyediting or rich discussion – “I DON’T CARE DO UR OWN PAGE!,” (Grant, 2009).
  43. 43. Implications• Hopeful: Research to develop strategies for scaffolding online student collaboration is really important• Despairing: What does it mean if Dohn is right? What if the institutional structures of schooling limit collaborative learning?
  44. 44. blogs.edweek.org/edweek/edtechresearcher
  45. 45. Back Deck
  46. 46. What subjects are wikis used for? (n=411) English / Language Arts 120 Social Studies 70 Science 61Computer Science/ Technology 60 Math 45 Library 26 Art 22 Contained Elementary 20 Modern FL 10 Health/PE 8 Business 6 ESL 5 Classics 4 Education 2 0 20 40 60 80 100 120 140
  47. 47. # of Time/Scale Web 2.0 ResearchCases State Space Modeling1,000K Usage Simulations Statistics100K Semantic Analysis 10K Surveys1,000 Content Interviews Analysis 100 Discursive 10 Analysis Biometric Design Observational 1 Analysis Research Research Seconds Days Weeks Months Years 49 Duration of data collection and capture
  48. 48. Prototypical wiki quality trajectories in subject areas, controlling for SES (n=259) 25 20Wiki Quality Score 15 10 Social Studies English 5 Computer Science Science Math 0 0 100 200 300 400 500 Days
  49. 49. MrBoyersClass.Pbworks.com Page Saves by Day100 90 80 70 60 50 Page Saves 40 30 20 10 0 0 50 100 150 200 250 300 Days
  50. 50. Moving average of wiki development measured in page savesALLPS 30 20 10 0 0 100 200 300 400 500 600 700 800 900 day

×