1. PLAY Minecraft!
Assessing secondary engineering education using game
challenges within a participatory learning environment
(Research to Practice)
Strand: Addressing the NGSS: Supporting K-12 Teachers in Engineering
Pedagogy and Engineering-Science Connections
Erin Shaw, Computer Scientist, University of Southern California,
Information Sciences Institute
Minh La & Richard Phillips, Undergraduates, USC Computer Science
Erin Reilly, Director Innovation Lab, USC Annenberg School of Communications &
Journalism
Thank you to Aninoy Mahapatra, Sam Shuster and Dr. Anthony Maddox

2. Warning
This paper is not about playing Minecraft!
Sorry :(
We studied
how children talk about Minecraft!
This is good stuff, too :)

3. Project overview
Goals
• Apply participatory learning to STEM
• Assess learning using Next Generation Science Standards
• Explore computational methods for assessing learning
Key ideas
• Social media/multimedia based science learning
• Discourse/context based instructional assessment

4. Project overview
What
• USC Undergraduate Research Associates Program (URAP)
• Ten weeks during summer of 2013
Who
• Two undergraduates (2B junior and sophomore)
• Graduate student advised on NLP/ML tools
• Programmer advised on platform and database
• “Committee” consisted of faculty and staff from schools
of engineering, communication and education

5. Outline
1. Learning theory – Participatory learning
2. Platform – PLAYground
3. Subject – Minecraft
4. Standards – NGSS Science & Engineering Practices
5. Methods – Topic modeling
6. Data – Focus session
7. Processing – Data mining
8. Analysis – Results display

6. 1. Learning theory – Participatory learning
Participatory cultures (Jenkins)
• Have low barriers to engagement
• Support artistic expression, sharing creations, mentorship
• Foster New Media Literacies (NML)
- e.g., performance, appropriation, distributed cognition
• Focus on social skills developed through NMLs
Four C’s of participatory learning
• Creativity, Connection, Collaboration, Circulation

7. 2. Platform - PLAYground
PLAY! (Participatory Learning and You!)
• Project consists of technology and pedagogy
PLAYground
• Multimedia authoring environment
- Users create “Challenges”
• Designed around 4 Cs of PL
• Piloted by high school teachers
• Never used for STEM subjects

8. PLAYground canvas

9. 3. Subject – Minecraft
What is Minecraft?
• Popular, free-to-play computer game
• Has dynamic virtual worlds built of LEGO-like blocks
• Players can combine blocks to “craft” items and to build
fantastical structures
Minecraft is a virtual engineering environment

10. Minecraft creations

11. Minecraft wiki site

12. 4. Standards – NGSS SEPs
NGSS emphasize 3 dimensions
• Science & Engineering Practices (SEPs)
- Behaviors that scientists engage in as they investigate and build models
and theories about the natural world
- And that engineers use as they design and build models and systems
• Crosscutting Concepts
- Provide an organizational schema for interrelating knowledge from
various science fields
• Disciplinary Core Ideas
- Focus curriculum, instruction and assessments on the most important
aspects of science

13. NGSS dimensions

14. 5. Methods – Topic modeling
Build corpora for subject and standards
• Minecraft – Scrape wiki
• SEPs – Copy all SEP text from NGSS document
• Both - Clean text (NLTK)
Process
• Pre-determine categories
• Fit topic model (MALLET)
- Resulted in each SEP and Minecraft category being associated with a
topic distribution (each topic consists of weighted vector of words)
-

15. Final topic categories
SEP categories
- Analyzing and interpreting data
- Asking questions and defining problems
- Constructing explanations and designing solutions
- Developing and using models
- Engaging in argument from evidence
- Obtaining, evaluating, and communicating information
- Planning and carrying out investigations
- Using mathematics and computational thinking
Minecraft categories
- Blocks, Items, Enchanting, Mobs, Trading, Redstone Circuits, and
Modifications.

16. 6. Data – Focus session
Small user study conducted
• Five boys
- grade 4(2), grade 8(2) and grade 11(1)
• Two hours
- 30 minutes discussing seed challenges (undergraduate created)
- 60 minutes creating own challenges
- 30 minutes discussing peer challenges
• Simple questionnaire

17. Seed challenge discussion

18. Student “challenges”

19. 7. Processing – Data mining
Export database tables
Organize user interactions
• Collect, link and order data
• Attach context, prompts, questions, etc.
Output to table for validation

20. 8. Analysis – Results display
Analyze user discourse
• Performed topic similarity matching of SEP and Minecraft
topics using topic distributions from pre-processed
categories

21. Engineering standards application
G4
G8a
G11
G8b
G14
G15
G8b mathematically gifted
G4 in only one category
G11 in seven categories

22. Minecraft topics application
G4
G8a
G11
G8b
G14
G15

23. Standards application per student
G11 in seven categories

24. Discussion
Results
• Application of SEPs increases as grade level increases
• Mathematically gifted student identified through SEP assess
• Good subject category identification
Methods
• Validate, acquire more text, data, further experimentation
• New research to improve results using wordnet
Future
• Propose to apply techniques to middle school sciences

25. THE END
Thank you for listening.
Questions?
For more information please contact me at
shaw@isi.edu