Development Post-mortem and Research Results for Mecanika
Upcoming SlideShare
Loading in...5
×
 

Development Post-mortem and Research Results for Mecanika

on

  • 376 views

G4LI Games for Learning Day at G4C 2011

G4LI Games for Learning Day at G4C 2011

Statistics

Views

Total Views
376
Views on SlideShare
376
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Had an interest in educational games, but wasn’t sure which subject to pickTurns out students are loosing interest in science – you probably all know that. But it’s even worst in physics. Organisation for Economic Co-operation and Development recommends…A particular misconceptions: 2 balls with different weight fall – the heavier one is going to hit the floor first. There has been a lot of research in the past 30 years to understand them, and 20 years ago a standard test was developed. It can be used to assess whether or not a new way of teaching physics changes students misconceptions into Newtonian conceptions.
  • Direct instruction doesn’t work. Teaching from formulas doesn’t work. It doesn’t depend on the teacher. We think we’re teaching mechanics, because the students are passing, but we’re mostly teaching formulas. What does work is to involve the students more actively, and to make them aware of their own misconceptions. Students’ conceptions in mechanics are resilient to change (Brown & Hammer, 2008; diSessa, 1993)
  • And so games are a natural fit here. We’ve had these awesome physics puzzle games for years now, and you could build in theory a puzzle game which game mechanics are used to focus on misconceptions. This is what we tried to do with Mecanika. Your goal is to make small boxes go over stars. You do this by placing robots in the boxes paths. You can place impulse robots, which are going to punch the boxes around, robots that create wind zones, cause circular movement, and then you have to pass through zones that restrict your speed. The idea was to produce something that was going to be used in classrooms. It took 2 years to produce 50 levels. It is not something we intend to throw away after the experience is done, and in that sense it is a “real” game, not a “research” game. Levels are linked with misconceptions that were identified originally by the authors of the FCI test. We’re trying to lure students into making mistakes, to bring their intuitions into focus (cognitive conflict).
  • Goal was to study games where they would be used: in the classroom. If you see an effect in controlled labs conditions, but no effect when teachers use the game, that makes the game useless. Did this for 2 teachers, and a total of 8 groupsWe didn’t assign the game to randomly selected students, because we wanted teachers to do debriefings in the entire classroom
  • Significant difference between the two groupsSeems like a small number, but it is really hard to change student’s conceptions. A month of teaching from the same teacher didn’t change much. This is why having a control group is interesting, and should be done more often in game studies. You could say that teachers changed the way they teach with the game. But if that were true, you would see an increase with the control groups as well.
  • You could say that teachers changed the way they teach with the game. But if that were true, you would see an increase with the control groups as well. See additional slide after questions on the National modeling project, and why this result can be compared to other experiments.
  • This shows not only the gain, but the difference in gain from experimental to control. So if Exp got a 15% increase on a question, and the control group had a 5% increase, what you would see here is the difference between the two: 10%. No sign. diff. Not expecting people to increase here
  • Significant difference for 1st Law of Newton questions
  • Levels 20 to 30 didn’t produce much of a difference, as measured by the FCI (see “detailed FCI items” in slides”).
  • Significant difference between exp. and control groups on Kinematics and on 1st Law of Newton. Would be interesting to know what kind of impact levels 40 to 50 have.
  • Reason for the focused impact is simple: the game design didn’t focus on all misconceptions: Would have been really hard to create a coherent and fun game design that would allow us to bring all misconceptions into focus. Not pretending like the game could replace teachers, or even be used for the entire curriculum. It could be that the guides have an impact on how well students learned. Hypothesis: we never explain any concept in the game – they only play situations that are likely to trigger some misconceptions.
  • It can’t be something that the teacher doesHypothesis: we never explain any concept in the game – they only play situations that are likely to trigger some misconceptions. The game is also played for only 1.5 hours, somewhere in a full month period. Therefore we thought no gain would be observed.
  • Turns out the game had almost as big an impact when it’s just loosely integrated in the classroom. No sign. diff between the two gains. Could it be because the teachers are changing their ways of teaching? No->that would have shown in one of the control groups. Could mean that actual classroom debriefing was not done properly (9.2 gain could be much higher). Or that the game just teaches well by itself, and that there is no need for classroom debriefings.
  • Email me at francoisbg@gmail.com for additional resources (like guidebooks and training videos)We’re open to share mecanika for more research projects. Useful since we know that Mecanika works
  • DiSessa did a lot of work in this area, but Hestenes created a test that was widely popular in the early 90s. The ball question misconception is in that testTeachers expect their passing students to perform well on this tests, and they’re flabbergasted when they see them achieving only around 24%. 20% is what you get if you answer randomly.
  • Surge had some solid results.
  • Nationwide project in 1995-98Modelers – after the first year. Similar difference between control/exp and modelers/traditional. The mecanika experiment was much shorter, and didn’t require any training on our part. Could we have pushed it further, for more than a month? What would happen if we gave our teachers 3 weeks of training on top of that?
  • Would have been really hard to create a coherent and fun game design that would allow us to bring all misconceptions into focus. Q8:0% increase for the control group. To be expected, direct mapping. Surge had a significant increase as well (not difference), although the scale wasn’t revealed. Q20: Understand what acceleration is, or even questions about what forces are active on a standing chair
  • Some items did not improve, since the representation in the game (wind zones) and in the test (rockets) were different.
  • Retention, measured with paired t-test: p=1.00, +0.0%, effect size d=0.00, N=55Girls and boys did not get a significant gain difference, but when asked, boys did think that the game and the guidebooks were more useful, and that the game was more fun (p<0.05, effect size ranges from d=0.43 to d=0.49)

Development Post-mortem and Research Results for Mecanika Development Post-mortem and Research Results for Mecanika Presentation Transcript

  • Mecanika
    Development post-mortem and research results for Mecanika, a game to learn Newtonian concepts
  • The state of science education
    OECD (2008)
    Students in physics and mathematics
    Change how we teach physics
    More attractive
    Focus on conceptions
    Force Concept Inventory
  • Direct instruction
  • G3. Heavier objects fall faster
    Mecanika
    I5. Circular impetus
    CI2. Force compromise determines motion
    CI3. Last force to act determines motion
  • Methodology
    Control group
    Experimental group
    FCI pretest
    FCI posttest
  • Results (Paired samples t-test)
    +1.9%
    Effect size: d = 0.19
    N = 82
    p = 0.08
    +9.2%
    • Effect size: d = 0.95
    • N = 51
    • p < 0.001
  • What this means
    Gain obtained in a short period
    No training required
    The game + debriefing + guides are the only factor
    Is this only due to playing Mecanika?
  • Played 10/50 levels
  • Played 20/50 levels
  • Played 30/50 levels
  • Played 40/50 levels
  • What this means
    Focused impact
    Does not replace teachers
    Does the learning happens when playing, or outside of the game?
  • Methodology
    FCI posttest
    FCI pretest
    FCI post- posttest
  • Classroom integration
    +9.2%
    +7.3%
    What this could mean
    Teachers are changing
    Debriefings done poorly
    Game works by itself
  • The future
    Available now for free (French/English)
    www.gameforscience.ca/physica
    Research projects
    Mecanika 2?
  • ?
    francoisbg@gmail.com
  • Force Concept Inventory
    Multiple choice questionnaire
    No mathematics
    Validated tool
    Allows comparison
  • Previous work
    White (1984)
    SpaceFart: Potvin et al. (2010)
    Surge: Clark et al. (2011)
  • Compared to other experiments
    Compare difference in gain %
    Modeling Instruction Project
    “an intensive 3-week Modeling Workshop that immerses them in modeling pedagogy and acquaints them with curriculum materials designed expressly to support it.”
    Modelers: N = 3394, 66 teachers
    Gain difference: 10% VS 7.4%
  • Detailed FCI items
    Game design didn’t focus on all misconceptions
    Expected items
    From 42% to 96% after playing the game
    Other items we didn’t expect to improve
  • Postmortem
    Assess learning potential earlier
    Manipulate multiple force types
  • Additional findings
    Retention after 1 month: no significant decrease
    Boys thought the game was more fun, and the guides were more useful than girls (p < 0.01)
    No significant difference between genders on gain
  • Limitations
    Number of players
    Play mandatory, but not reinforced
    Technical problems
    Number of teachers
    8 classrooms, but only 2 teachers
    Teachers were recruited by interest