Using Learning Analytics Data to Understand Project-Based Learning

306 views

Published on

The Case of the Carrot Wizards in Globaloria-West Virginia

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
306
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • How do we plan to grow exponentially in a way that is sustainable? Strong and innovative Professional Development programs for educators, principals and students ensure the community can self-manage, grow and develop.Hands-on training: Mentor program – cascading and taken to scaleVirtual support – sustainable, scalable – walking our talkRewards & Recognition – we pay
  • In addition to the amazing impact we have seen over the past few years researching and evaluating the Globaloria program we…We are in unique position of having everything we do be rooted in years of academic research and real-world practice. In the 1980s and 1990s, while at Harvard and the MIT Media Lab, Idit really founded the concept that children learn best by designing – when they are programming computers instead of computers programming them. Ground breaking research with children in Bronx, showing how software designing could change their relationship to education and their engage them in learning in a way that had not been seen before
  • In addition to the amazing impact we have seen over the past few years researching and evaluating the Globaloria program we…We are in unique position of having everything we do be rooted in years of academic research and real-world practice. In the 1980s and 1990s, while at Harvard and the MIT Media Lab, Idit really founded the concept that children learn best by designing – when they are programming computers instead of computers programming them. Ground breaking research with children in Bronx, showing how software designing could change their relationship to education and their engage them in learning in a way that had not been seen before
  • In addition to the amazing impact we have seen over the past few years researching and evaluating the Globaloria program we…We are in unique position of having everything we do be rooted in years of academic research and real-world practice. In the 1980s and 1990s, while at Harvard and the MIT Media Lab, Idit really founded the concept that children learn best by designing – when they are programming computers instead of computers programming them. Ground breaking research with children in Bronx, showing how software designing could change their relationship to education and their engage them in learning in a way that had not been seen before
  • We have taken all this learning and experience and established the cornerstone of the Globaloria Program – It’s Learning Formula. I want to take a minute on this slide: Self-Led Learning- students determine the needs of their learning as they progress through the process of designing their own games. It is purposeful, self-directed and hands-on learingPeer-to-peer learning – it is collaborative learning. Students take on the role of both educators and students. They have a critical role in their own learning and in the advancement of their learning community. They are invested. Co-Learning – we are breaking down the traditional hierarchy in the classroom and having students and educators learn together. Students don’t learn for the grade, but as part of a community of creators. Expert-guided learning – we do not divorce learning from the real-world…it is the real world. Students engage with real professional and see the relevance of their learning in today’s world.
  • Using Learning Analytics Data to Understand Project-Based Learning

    1. 1. Using Learning AnalyticsData to UnderstandProject-Based LearningThe Case of the CarrotWizards in Globaloria-West VirginiaICLS 2012 Pre-Conference Data Workshop***Rebecca Reynolds, Ph.D.Assistant Professor, Library & Information ScienceRutgers University School of Communication andInformation
    2. 2. Background Info• The following slides provide background onthe Constructionist / design-based researchinnovation, “Globaloria,” being implementedby a NYC non-profit called the World WideWokrshop, in several US states with grant andstate education department support (WV, TX,CA, FL, NY)
    3. 3. Globaloria Game Design Program Learning Objectives: Cultivate the SixContemporary Learning Abilities (6CLAs)Developing games in a social learning system offers studentsConstructionist, participatory design and programming experiencesleading to dispositions, practices and expertise necessary for successfulengagement in today’s digital, knowledge-based economy
    4. 4. The Six Contemporary Learning Abilities (6CLAs):Developing games in a social learning system offers participatoryexperiences leading to practices necessary for engagement in today’sknowledge-based economy
    5. 5. Globaloria is currently activein 4 U.S. states: CA, TX, FL, WV
    6. 6. Globaloria-CaliforniaCalifornia, 2011/2012:• Supported by $1M KnightFoundation grant• “Formal” implementationat 2 middle schools (67students)• “Informal” implementationat 2 Boys and Girls Clubcommunity centers (38students)• First year of Californiaparticipation• Potential to compareimplementation contextvariable differences (timeon task; provision of gradesvs. none; linkage to corecurriculum vs. open choicegame topics, etc.)
    7. 7. Globaloria-TXEast Austin CollegePrep, 2011/2012:• 1 charter middleschool• 3 grades (6th, 7th,8th)• ~250 students• 3rd year ofschool’sparticipation• 2 educators• Uniquelongitudinalimplementation(all EACPAstudents takeGlobaloria daily,for credit and agrade for 3 years)
    8. 8. Globaloria-West Virginia2011/2012:• 5th year of Globaloriaimplementation in West Virginia• Individual schools and teachersrange in N of participation years• Majority of students, 1 year only• 17 middle schools• 50 high schools• All in-school implementation,daily, for credit, for grade• Mix of elective game designclasses, and required classeslinked to the core curriculum• Multi-level analysis andimplementation context variabledifferences increasingly beingmeasured (e.g., teacher N ofparticipation years; extent ofteacher training; student gamesubject focus;
    9. 9. Domains of Learning and Expertise• Constructionist digital literacy (skills needed in knowledgeeconomy => 6-CLAs)• Computational thinking through game design in Flash andprogramming in Actionscript• Core curricular subject matter:– When game subjects are linked to core curriculum and studentsdeepen knowledge about topic through online research anddesign• STEM career interests: Technology & Engineering;Computer Science• Motivation, Affect, Attitudes, Life Choices, New Possibilitiesand Horizons, esp. for disadvantaged students
    10. 10. Flash software, Wiki Environment, Curriculum,Tutorials“Hands On” Training Sessions (virtual, local)• Globaloria Academy – In-person, intensive trainings (3)• Online Mini Webinars - Web-based workshops (7)Globaloria Mentors ProgramExperienced educators take on a leadership role bysupporting other educators“24/7” Virtual Support• Expert Support via wikis, blogs, email, WebEx• Educator Community Development – privateeducators community wiki, peer-to-peer mentoring, weeklyeducators newsletter, sharing teaching & learning reportsRewards and Recognition•Teachers: Stipends and Graduate credits are earned•Students: Nationally-Recognized Game DesignCompetitionsLearning Supports for Students and Educators:
    11. 11. Syllabus integrates learning about thegame subject topic andgame programming and mechanics
    12. 12. Syllabus integrates learning about thegame subject topic andgame programming and mechanicsSyllabusTopicsAddressingGame SubjectSyllabus TopicsAddressing GameProgramming andMechanicswww.myglife.org/usa/wv
    13. 13. Syllabus integrates learning about thegame subject topic andgame programming and mechanics
    14. 14. Data Collection forResearch Occurs via 6Main Technology andCurriculum PlatformsDeveloped by theWorld Wide Workshop
    15. 15. Data Collection forResearch Occurs via 6Main Technology andCurriculum PlatformsDeveloped by theWorld Wide Workshop
    16. 16. Data Collection forResearch Occurs via 6Main Technology andCurriculum PlatformsDeveloped by theWorld Wide Workshop
    17. 17. Data Collection forResearch Occurs via 6Main Technology andCurriculum PlatformsDeveloped by theWorld Wide Workshop
    18. 18. The GlobaloriaLearningFormula:Discovery-basedproject-based,student-centered, sociallearning
    19. 19. Some Globaloria project researchgoals at a high level• Understanding the influences and forces at play in this social learningsystem; mapping the terrain from a social science perspective• Design-based research: Iterative improvement of the social learningsystem; generalization towards learning theory, and the instructionaldesign of effective learning innovationsConnecting multiple levels of analysis in our research will be necessary to accomplishthe above . . . And ideally, the below:• Evaluative Diagnostics:– Use predictive modeling to identify trajectories of student success / failureacross time (what factors at multiple levels of analysis influence studentlearning?)– Apply predictive findings towards development of diagnostic tools for earlyintervention that teachers can use for formative assessment• Summative Evaluation: Developing tools for program evaluation andstudent assessment
    20. 20. 2 Main Strands of ResearchQuantitative multi-level analysis modeling (w. Ming Ming Chiu)• Purpose: Model the complex relationships among variables of importance. Currently,DV is evaluation scores for games based on content analysis of their quality• Present data sources:– Teacher and student pre / post surveys– Teacher progress reports– Content analysis evaluation of team games– Wiki log files and behavioral site metrics data (edits, uploads)– Other data TBA (e.g., game subjects; teacher grades; student role-taking; resource use)Qualitative analysis, comparison of cases (w. Cindy Hmelo-Silver)• Purpose: investigate and explain dynamics / inter-relationships among variables;identify key phenomena driving the system; add to quantitative model as relevant• Present data sources:– Semi-structured teacher and student team interviews at two timepoints this past semester(18 middle school teams, TX & WV, and their educators)• Main themes: Teacher/peer/e-learning/non-profit scaffolding; student problem self-structuring;student/teacher resource use; areas of challenge/enjoyment; teamwork processes– Future: Participant observation in situ in the classroom
    21. 21. Challenging Assumptions• To what extent do complex multi-level phenomena anddynamics in this system break down into discretevariables that can be investigated for their inter-relationships in quantitative modeling? Is thephenomena in such a social learning system betterdescribed quantitatively? Qualitatively? Mixedmethods? How can varying analyses inform eachother?– Perhaps something we can discuss today
    22. 22. Analytical difficulty Statistical Discourse Analysis strategyOutcome variableBinaryInfrequentTeams within classrooms(Nested data)Multiple outcomesIndependent variablesIndirect, mediation effectsFalse positivesLogit (Kennedy, 2008)Logit bias estimator(King & Zeng, 2001)Multilevel analysis(Goldstein, 1995)Multi-outcome analysis(Goldstein, 1995)Multilevel M-Tests(Krull & MacKinnon, 2001)Two-stage linear step-up procedure(Benjamini et al., 2006)
    23. 23. Mean pre-intrinsicmotivation+0.176 *Mean increase inintrinsic motivation+0.163 *Mean SWF flashuploads per month+0.166 *Mean age –0.276 ***Team StructureDesign mixedgenre game+0.833 **Team ProcessesDesignGame playConceptMean self-learning hoursProgramming–0.452 ***Teacher Outcome
    24. 24. Analysis highlights3 levelsClassroom (teacher)Team (mean of group members)Team process (mean of team processes)Test hypotheses about outcomesw/ explanatory variables at lower levels--characteristics of samplesMean, SD, max, min, …Model multiple outcomes simultaneously
    25. 25. Qualitative analysis1. What are the theoretical instructional design assumptions implicit in thedesign and development of Globaloria?– Guided discovery-based learning, building on Constructionism– Scaffolding provided via a range of structures (explored in research)2. How are students at different schools interacting with the range ofscaffolds provided? Autonomous/controlled/task-driven– A focus on student self-structuring of problems and resource uses to support theirlearning, at various stages of the process3. Linking Scaffolds, Problem-Solving, and Resource Use:– To what extent does it appear that the structuring elements / scaffolds above helpstudents meet learning goals: conceptualize the idea for a game, develop a prototype,implement a game in Flash, learn programming in Actionscript, complete the final game,do this in a team context (combining files), and upload the final game to the wiki?4. Does this process appear to bring about other outcomes besides discreteCLAs/Constructionist digital literacy, and/or other domains of expertise(computational thinking; STEM career interest)– Resourcefulness?
    26. 26. Approach to the ICLS 2012Data WorkshopWhat can we learn about multi-level phenomena froman in-depth focus on 1 class of 2010/2011 gamedesigners, using an eclectic array of data sources?• Investigate the experience of a single case study team ofgame designers from TVHS, consisting of 3 HS students whowon the annual game design competition in the 2010/2011school year with their Civics content game, “Henry theHedgehog”– Understanding team processes can expose some of thevalues/norms built into the system (what makes a good game?)• Investigate relationships among variables that appearinfluential to learning.• Consider this case in context, given that similar data areavailable for over 2000 students across 3+ years• Apply findings to ongoing research being conducted
    27. 27. Data Sources Provided to Participants, and Corresponding Levels of AnalysisLevel ofAnalysisLocal context Data source FilenamesInnovation Curriculum Description of Globaloria syllabus,programoverview2.Curriculum.pdfICLS2012_dataworkshop.pptxE-learningplatformDescription of features of e-learning platform;screenshots; wiki log files (trace data of studentactivity on the wiki)2.Curriculum.pdfICLS2012_dataworkshop.pptx13. Aggregate wiki log _ contentanalysis.xlsxSystem WVDepartmentof EducationGame competition press release; information onjudges and judging criteria.14.game_comps_press_2011-1.pdfSampleWikiCurricLesson_Drag and Drop.docxWikiScreenshots folderOrganizational School Description of the school 3.Case Study Data File, Carrot Wizards.docxICLS2012_dataworkshop.pptxWorld WideWorkshopnon-profitDescription of non-profit, goals 3.Case Study Data File, Carrot Wizards.docxICLS2012_dataworkshop.pptxClass Teacher Progress report from teacher offering reflection onPD experiences, learning and teaching; andobservations of students -- individuals and teams4.teacherprogress report.docTeam Game designteamsTeam key (who is in what team); Carrot Wizardscase study report; evaluation of team games; wikilog files5.teammembers_fullclass_tvhs_y4.csv7.1.CarrotWizardsFINALGAME.docx8.yr4_globaloria_eval_researchfile_TVHS.xlsx13. Aggregate wiki log _ contentanalysis.xlsxIndividual StudentparticipantsPre- and Post-Surveys; student blog URLs; casestudy report; wiki log files9.yr4_globaloria_postsurvey_qualitative_TVHS.xls10.yr4_globaloria_presurvey_qualitative_TVHS.xls11.blogURLsforCarrotWizards.docx13. Aggregate wiki log _ contentanalysis.xlsxArtifact Studentgames;student wikiworkActual team games; wiki screenshots; gameevaluation6.File_Cwfinal2atyler.fla7.1.CarrotWizardsFINALGAME.docx7.File_Cwfinal2atyler.swf3.Case Study Data File, Carrot Wizards.docxWikiScreenshots folder8.yr4_globaloria_eval_researchfile_TVHS.xlsx
    28. 28. Further context about TVHS• WV high school class at TVHS took Globaloria as a game designelective, meeting daily for 90 minutes per day, for credit and agrade across an entire school year (2010/2011)• This was TVHS and educator A. Leef’s first year in the programimplementing Globaloria; she participated in the standardGlobaloria Academy PD training and ongoing offerings throughoutthe year.• Students in 8 teams of 1, 2 or 3, created a total of 8 games about arange of topics relating to either Civics or STEM• 2 games focused on Civics: Carrot Wizards’ Henry the Hedgehoggame, and Skull Crusher’s game Blast.• Henry the Hedgehog game won the Globaloria Annual Civics GameDesign competition in the 2010/2011 school year
    29. 29. Analysis: Narrowing in on Team Process step ofquantitative multi-level analysis, using Wiki log files• Behavioral trace log file data available, measuringindividual-level edits to wiki pages, and uploads ofgame artifacts, graphics, .FLAs and .SWFs.• Research Questions:– How do edits / uploads of teams at TVHS vary across time?• Totals, team means, indivs-within-teams– General observations about wiki use patterns– What assumptions/limitations associated with usingfrequency data in multi-level analysis?– What stories do the data tell, that the aggregate frequencynumbers miss?
    30. 30. Preliminary Researcher Analysis ofData Sources: Wiki log files• Edits to team pages (main locus on the wiki for team-level activity):• Game project file uploads
    31. 31. Team Rankings,Total Sum Evaluation ScoreTeam Name:TOTALSUMCarrot_Wizards 57Shock-a-holics 53LEET_HAX 51Bull_Loany_Productions 50Gummy worms.(: 49Thomas the TrainsPancakes of Pain 49TEAM_SKULL_CRUSHER 46TOMORROW 19
    32. 32. Aggregate Sum Results, EditsTeam NameN of TeamMembers Total Edits Mean EditsCarrot_Wizards 3 161.00 53.67Gummy_worms.(: 2 144.00 72.00LEET_HAX 1 66 66Shock-a-holics 3 91 30.33TEAM_SKULL_CRUSHER 2 60 30Thomas_the_Trains_Pancakes_of_Pain 3 60 20TOMORROW 1 12 12
    33. 33. Descriptive ResultsTotal Edits to Wiki Team PageWeek of 9/20/10 – week of 5/9/110.0020.0040.0060.0080.00100.00120.00140.00160.00180.00Total editsMean edits
    34. 34. Total N of Edits to Team Wiki Pages,Week of 9/20/10 – week of 5/9/11How does team activity vary across time?• Carrot Wizards appears to have invested greater effort in editing their team wiki page atthe beginning than other teams. Evidence of possible impact of early planning.• Carrot Wizards and Gummy Worms appear most consistent in their wiki editing acrosstime• Carrot Wizards appears to have exerted more effort towards the end of the school yearthan other teamsNote: 1 outlier data point from a single day by a single student in 2010w12, Leethax team omitted
    35. 35. Mean Edits to Team Wiki Pages,Week of 9/20/10 – week of 5/9/11(Mean = Total N of page edits / N of team members)How does standardizing for N of team members influence patterns?• Gummy worms had fewer team members, thus logged more average activity with consistency as ateamNote: 1 outlier data point froma single day by a single student in 2010w12, Leethax team omitted
    36. 36. Explaining Gummy Worms greater N ofedits (individuals within teams)• Consistency in editing among both team members
    37. 37. Explaining Gummy Worms greater N ofedits (individuals within teams)• Tyler was primary wiki team page editor. The other 2 team membersonly did so in S2 at the end.• What is the difference in team dynamic, between having 2 vs. 3 teammembers?
    38. 38. Total N of File Uploads,Week of 9/20/10 – week of 5/9/11How does team activity vary across time?• All teams have greater frequency/consistency of uploads in S1; uploads become moreconcentrated during S2• File types vary to a greater extent in S1 (includes more static graphics files)• Carrot Wizards and Gummy Worms files are mainly SWFs and FLAs in S2
    39. 39. Explaining Gummy Worms greater N ofuploads (individuals within teams)• Brianna uploads most frequently and consistently• Nicole outlier in week 30
    40. 40. Carrot Wizards individuals-within-teamUploads• Relative consistency among team members• What was occurring at week 29-34 in the program curriculum to spark high uploadsamong these 2 teams?
    41. 41. Connecting team and individual levelsof analysis• How is wiki work distributed across individualteam members?– Procedure:• For Carrot Wizards only, reviewed log files acrosstime, by team member• Review actual wiki content edits made by teammembers across time• Note categorical types of wiki content edits• Make observations regarding individual activity withinthe team– Link inferences to team-level findings above
    42. 42. Carrot Wizards distribution of individualcontributions to wiki
    43. 43. Carrot Wizards distribution of individualcontributions to wiki
    44. 44. Carrot Wizards distribution of individualcontributions to wiki
    45. 45. Carrot Wizards distribution of individualcontributions to wiki
    46. 46. Carrot Wizards distribution of individualcontributions to wiki
    47. 47. Connecting Wiki activityto Evaluation Scores• To what extent do patterns of wiki activityrelate to evaluation outcomes?– This is a next step in our analysis.– We welcome the input of other researchers
    48. 48. Next Steps• Summarize findings on individualcontributions within the team context• Consider results in relation to game evaluationfindings
    49. 49. Opportunities for WorkshopParticipants• Conduct social network analysis / sequentialinteraction analysis on individual and team levelwiki activity results for TVHS teams– Raw aggregated wiki log file and evaluation outcomedata are available in the Excel file, “13. Aggregate wikilog _ contentanalysis.xls”• Relate wiki raw frequency and social networkanalysis findings to final game evaluationoutcomes• Explore relationships between qualitative findingsin Dropbox, and results reported herein

    ×