SCRIBL Data- Scalable, Real-Time, Individual Behavior and Learning Data

Cyberlearning Tools for STEM Education Conference 2011 Join the online discussionbackchannel for this session at:www.cyberlearningSTEM.org/sessions

SCableReal-time SCRIBL Data Justin ReichIndividual Harvard Graduate School of EducationBehavior and Cyberlearning ToolsLearning for STEM conference 3/8/11Data bjr795@mail.harvard.edu

MrBoyersClass.Pbworks.com Page Saves by Day100 90 80 70 60 50 Page Saves 40 30 20 10 0 0 50 100 150 200 250 300 Days

Scatter Plot of Wiki Page Saves by Day (n=1,799)

Moving average of wiki development measured in page savesALLPS 30 20 10 0 0 100 200 300 400 500 600 700 800 900 day

SCableReal-timeIndividualBehavior and SCRIBL DataLearningData

# of Time/Scale Web 2.0 ResearchCases State Space Modeling1,000K Usage Simulations Statistics100K Policy Semantic Makers Analysis 10K Surveys1,000 School Content Interviews Leaders Analysis 100 Discursive 10 Analysis Teachers Biometric Design Observational 1 Analysis Research Research Seconds Days Weeks Months Years 11 Duration of data collection and capture

Research with SCRIBL Data• We can study SCRIBL data with microscopes and telescopes – Plan for interdisciplinarity – New methods are needed• Operationalize time• Don’t invent new platforms, meter widely adopted platforms

Learner Analytic Approaches Netflix vs.Pandora

PBworks wiki lifetimes in seconds/days 1 0.9 (n=179,851)Estimated Survival Probability 0.8 Estimated Seconds Days 0.7 Lifetimes 25% 250 <1 0.6 50% 123,613 1.4 0.5 75% 5,282,874 61.1 0.4 0.3 0.2 All PBWorks Wikis 0.1 0 0 20000000 40000000 60000000 80000000 10000000 (231) (463) (694) (926) (1157) Time in seconds (days)

Learner Analytic Approaches Netflix vs.Pandora

Does wiki persistence differ in Title I and non-Title I schools? (n=259) Non-Title I Eligible Title I Eligible

Wiki Opportunities for Students to Develop 21st Century Skills• Expert thinking: – Do students use academic content knowledge in wiki activities? – Do students reflect on the process/product?• Collaboration: – Do students concatenate text on pages? – Do they substantively edit each others work and co-create pages?• New Media Literacy: Wiki Quality Instrument – Do students use formatting? 25 Questions – Do they hyperlink? – Do they embed multimedia? Scale of 1-25

Do wikis provide opportunities for students to develop 21st century skills? 25 Are great wikis born or made? 20Wiki Quality Score 15 10 5 0 0 50 100 150 200 250 300 350 400 450 Days

Do wikis created in high SES schools provide more opportunities for 21st century skill development?25201510 5 High SES Low SES 0 0 50 100 150 200 Days 250 300 350 400 450

Do wikis created in different subject areas provide different levels of opportunity to develop 21st C skills 25 20Wiki Quality Score Social Studies English 15 Science Computer Science 10 Math 5 0 0 50 100 150 200 250 300 350 400 450 Days

SCableReal-timeIndividualBehavior and SCRIBL DataLearningData

Educators routinely try to gather informationabout their students’ learning on the basis ofwhat students do in class. But for any questionposed in the classroom, only a few studentsrespond. Educators’ insight into what theremaining students do and do not understand isinformed only by selected students’ facialexpressions of interest, boredom, or puzzlement.To solve this problem, a number of groups areexploring the use of various technologies to―instrument‖ the classroom in an attempt to findout what students are thinking.

Back Deck

Classroom Wiki Research Questions• How do we make them good? – What best practices, attitudes and resources produce wiki learning environments that promote and nurture 21st century skills?• Do only certain kids get the good ones? – What is the distribution of high quality wikis across schools serving different student populations? 25

Open Education Resources• Twin Hopes for OER – (Excellence) Teachers will use free, online tools and resources to create student- centered learning environments that prepare students for 21st century life – (Equity) Since these materials are free, poor students will disproportionately benefit. Brown, J. S., & Adler, R. P. (2008). Minds on fire: Open education, the long tail, and learning 2.0. Educause Review, 43(1), 16-32. Bonk, C. J. (2009). The world is open : How web technology is revolutionizing education (1st ed.). San Francisco, Calif.: Jossey-Bass.

Narrowing gaps or a rising tide? Non-poor students Non-poor students21st Century Skills Poor students Poor students

Remapping the Digital Divide• How can we design a study to examine whether or not poor students disproportionately benefit from the availability of free online tools? – Theoretical Framework – Operationalizing the theoretical framework – Research Design

Dimensions of the Digital Divide1st Digital Divide: Access 2nd Digital Divide: Usage• Schools • Attewell (2003): ―[There exists a] – 3.8 Students/Computer in real possibility that computing for schools with <35% already-disadvantaged children students eligible for FRPL may be dominated by games at home and unsupervised drill- – 4.0 students/computer in and-practice or games at schools with >75% student school, while affluent children eligible for FRPL enjoy educationally richer fare• Anywhere with more adult involvement‖ – 86 % of students living in • Jenkins (2007) Participation Gap households making <$30K Attewell, P. (2003). Beyond the digital divide. In P. Attewell, & N. use the internet Seel (Eds.), Disadvantaged teens and computer technologies (pp. 15-34). Munster, Germany: Waxmann. – 97% of students living in Jenkins, H.; Clinton K., Purushotma R., Robison A. and Weigel M.(2007), Confronting the challenges of participatory culture: households making >$70K Media education for the 21st century. Chicago, Il.: MacArthur Foundation. use the internet

How can we operationalize usage?Usage as persistence• Wiki lifetime: number of days of activity of a wiki community – Birth: Creation of wiki subdomain (e.g. ReichWorldHistory.pbworks.com) – Death: Final wiki edit • After a 90 day observational period 30

How can we operationalize usage?Usage as opportunities to develop 21st century skills• Participation• Expert Thinking• Complex Communication• New Media Literacy 31

Research Design

What are our research questions?RQ #1) Persistence: Are wikis created in schools serving affluent students used for greater lengths of time than wikis created in poor schools?RQ#2) Participation: Do wikis created in schools serving affluent students provide more opportunities for students to develop 21st century skills 33

Which wikis are in my sample?• Dataset – All179,853 publicly-viewable education-related wikis started on the PBworks platform between June 2005 and August of 2008. – Does not include ―private‖ wikis (~70,000)• Sample – Randomly sampled 1,799 wikis (1%) – Coded to identify 259 U.S. based, K-12 wikis from specific public schools • Detailed usage statistics provided by PBworks.com • Demographic school level data from the Common Core of Data (National Center for Education Statistics, 2007-2008) 34

What are our data analytic strategies?

RQ#1) Are wikis created in non-poor schools used forgreater lengths of time than wikis created in poor schools? • Estimate survival functions of wiki groups by Title I status using Kaplan-Meier estimation; use Wilcoxon’s test to test for differences where S(ti) is the estimated survival probability in any of t time periods, which are delineated by instances of wiki death ni is the number of wikis still active at the beginning of time period ti; di is the number of wikis that become inert during time period ti.

RQ#2) Do wikis created in non-poor schools exhibit more evidence of collaboration and student involvement than wikis created in poor schools? Estimate wiki quality trajectories using the multilevel model for change.

Findings

How long do K-12 wikis persist? (n=411) Estimated Seconds Days 1 Lifetimes All PBworks 0.9 25% 250 <1 0.8 50% 123,613 1.4Survival Probability 0.7 75% 5,282,874 61.1 0.6 K-12 Wikis 25% 2,721 <1 0.5 50% 763,195 8.8 0.4 75% 12,590,074 145.7 0.3 All PBWorks Wikis 0.2 K-12 Wikis 0.1 0 0 20000000 40000000 60000000 80000000 10000000 (231) (463) (694) (926) (1157) Time in seconds (days)

What subjects are wikis used for? (n=411) English / Language Arts 120 Social Studies 70 Science 61Computer Science/ Technology 60 Math 45 Library 26 Art 22 Contained Elementary 20 Modern FL 10 Health/PE 8 Business 6 ESL 5 Classics 4 Education 2 0 20 40 60 80 100 120 140

Does subject area predict persistence? (n=411) 50% 75% Lifetime LifetimeSubjectComputer Science 28 198English/Language Arts 27 198Science 18 152Social Studies 6 56Math 4 33No Subject 1 8

What Grade Levels are K-12 wikis used in? (n=411) K-5 109 6--8 118 9--12 180Higher Ed 8Unknown 83 0 50 100 150 200

Does grade level predict persistence? (n=411)• No

Do wikis from non-poor schools persist longer than wikis from poor schools? (n=259) Kaplan Meier Survival Estimates of Wikis from Poor and Non-Poor Schools Non-poor schools (n=146) Poor schools (n=110) Days 44

Summary statistics of wiki lifetimes in poor and non-poor schools (n=259)• Day 1 Mortality: – Wikis from poor schools: 40% – Wikis from non-poor schools: ~20%• Median Lifetime: – Wikis from poor schools: 7 days – Wikis from non-poor schools: 58 days• 25% Lifetime: – Wikis from poor schools: 73 days – Wikis from non-poor schools: 259 days 45

RQ#2) Do wikis created in non-poor schools exhibit more evidence of collaboration and student involvement than wikis created in poor schools?

Title I eligible Non-Title I eligible (n=110) (n=146)Concatenation 12 24Copyediting 10 15Co-construction 7 7Commenting 16 22Collaborative Sum= 0 82 (75%) 110 (75%)Collaborative Sum= 1 17 (15%) 13 (9%)Collaborative Sum= 2 7 (6%) 17 (12%)Collaborative Sum= 3 2 (2%) 3 (2%)Collaborative Sum= 4 2 (2%) 3 (2%) 2 Goodness of fit test ( 2=4.2, df=4, p=.38)Student Involvement 28 (25%) 50 (34%) 2 Goodness of fit test ( 2=2.28, df=1, p=.13)Student Involvement and at least 1 16 (15%) 20 (14%)Collaborative Behavior 2 Goodness of fit test ( 2=0.04, df=1, p=.85)

Discussion• Participatory behavior is rare across all wikis, but both student involvement and collaboration can be found in wikis from both poor and non-poor schools• Wikis from non-poor schools persist longer than wikis from poor schools.• The Open Education Resources strategy of promoting free online tools and resources may, counter-intuitively, expand the second digital divide—in the absence of targeted interventions. 48

How can we begin to explain these patterns?• What might explain our empirical findings? – What obstacles do poor schools have in using wikis?• Qualitative Research – Interviews with 50+ wiki-using teachers, many drawn at random from our PBworks samples. – 35+ focus groups with students – Classroom observations in 12 schools in MA, CT, ME, NH, CA, GA, VA

What obstacles exist for wiki use in poor schools?• Differences in resources?• Differences in school culture?

Differences in networked technology Low Poverty Medium Poverty High Povertyresources among public school districts Districts Districts Districts (>20%) (<10%) (11-20%)Provide teachers with their own server space forposting their own Web pages or class materials (Elementary 90% 81% 74% Secondary) 92% 84% 74%Provide students with electronic storage space 76% 60% 50%on a server 92% 85% 72%Provide students with online access to the 82% 69% 66%library catalogue 92% 82% 72%Provide students online access to databases (for 71% 58% 53%library resources) 79% 67% 57%Employ an individual responsible for educationtechnology leadership (Full Time 60% 48% 47% Part Time 26% 35% 33% None) 13% 17% 20%SOURCE: U.S. Department of Education, National Center for Education Statistics, Fast Response Survey System (FRSS),“Educational Technology in Public School Districts, Fall 2008,” FRSS 93, 2008.

Fear, anxiety and worry about student exposure to the InternetBarbara: There was - the superintendent said, "Do want you want but I never want us on the frontpage with some bad headline" [chuckle]. I thought those are kind of interesting. Thats why he told our technology committee. Our main goal is to not make the frontpage with the bad headline attached.Jim: And as far as I know but I dont want to be the person to be in the newspaper for... Look at what this kid was doing on the internet in a school. I dont want that

Framework for Intervention:Cycle of Experiment and Experience Experiment Fear - Growth+ Review Plan (Experience) Institutional Capacity+

The atmosphere at our school in general is to cautiously open up to thepossibilities. So, we’re not being pushed to use technology. If we find our wayto it, and if we find obstacles that are there and we need things to change to getaccess to certain things, the administration generally will make that happen. Butthis, I think in terms of the environment, security is more important thanopenness in general and sometimes that leads us into a few obstacles….We havent gotten to the point where were making a lot of the stuff public yet.Is that possible down the road? I think with a little bit of experience, with alittle bit of, we improve the comfort level then, we can start to say, “Lets makea blog that invites other people throughout the world. Whoever wants to comevisit and check it out and be part of the conversation; lets figure how to dothat.” I dont think I should... Im not really in that place that right now. Im notsure what the response would be if I did. I think we all sort of need to build thatcomfort level piece first. But I also see that it could be pretty awesome to movein that direction.

Dear Justin,I am in the library right now and sat down to do some work onthe Wikispace to get it ready for next year. However, uponsitting down, I discovered that over the summer this websitehas been blocked by the City. I spoke to the librarian aboutappealing blocked content, but he says that they are notunblocking any sites at this time.Consequently, it does not look like we are going to be doingthis project this year. I will be trying to put togethersomething else for this unit, but at this point, I dont knowwhat we will be doing and doubt it will be appropriate foryour project because it will not be on wikispaces.com.I am very sorry; it was quite a surprise to me.

Framework for Inequities in wiki usage Wikis created in non- Intervention: suggest that Web 2.0 poor schools persist Promoting a Cycle of tools may than and longer exacerbate Experiment wikis nd digital schools created in poor divide the 2Experience Experiment Fear - Growth+ Non-poor students21st Century Skills Review Poor students Plan (Experience) Institutional Capacity+

What Next?1. Develop an instrument to measure Wiki Quality2. Correlate wiki quality profiles with teacher attitudes, practices, and resources3. Develop computational tools to automate those analyses at scale4. ?5. Profit

Acknowledgements• Hewlett Foundation Open Education Resources for grant support• PBworks for data support• Hunter Gehlbach, Stone Wiske, Laura Schifter, Anna Savaadra, and other readers of this paper.• Benjamin Mako Hill for coming up the river to offer his thoughts and critique! 58

SCRIBL Data- Scalable, Real-Time, Individual Behavior and Learning Data

by EdTechTeacher.org on Nov 18, 2011

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 2

Statistics

SCRIBL Data- Scalable, Real-Time, Individual Behavior and Learning Data Presentation Transcript