The document describes a study that developed a concept for measuring the learning outcomes of a lean simulation game. The concept involved pre- and post-tests to assess competency development, and interviews to evaluate actions during game play. The simulation game was tested with students to address problems in objectively evaluating competency transfer from such games. Results found an increase in test scores and self-reported competencies, supporting the effectiveness of using simulation games for lean education.

1. European Lean Educator Conference 2017
(ELEC 2017)
Nijmegen, 7th – 10th November 2017
Measuring Lean
Competencies – An
Approach for
Quantifying the
Learning Outcome
of a Simulation
Game

2. Agenda
1. Introduction Technology Centre PULS
2. Problems evaluating Competence Transfer of Lean Simulation
Games
3. Measuring the Effectiveness of Simulation Games
4. Lean Simulation Game “From Push to Pull”
5. Concept of Measuring the Learning Outcome of a Lean Simulation
Game
6. Application and Results of the Measuring Concept
7. Conclusion
15.11.2017 Landshut University of Applied Sciences, Germany 2

3. Agenda
1. Introduction Technology Centre PULS
2. Problems evaluating Competence Transfer of Lean Simulation
Games
3. Measuring the Effectiveness of Simulation Games
4. Lean Simulation Game “From Push to Pull”
5. Concept of Measuring the Learning Outcome of a Lean Simulation
Game
6. Application and Results of the Measuring Concept
7. Conclusion
15.11.2017 Landshut University of Applied Sciences, Germany 3

4. Introduction Technology Centre PULS (Production and
Logistics Systems)
 4 professors with
doctoral students
and administration
personnel
 Intensive
cooperation with
industry and science
 State of the art and
best practice centre
for Lean Production
and Lean Logistics
15.11.2017 Landshut University of Applied Sciences, Germany 4

5. Agenda
1. Introduction Technology Centre PULS
2. Problems evaluating Competence Transfer of Lean Simulation
Games
3. Measuring the Effectiveness of Simulation Games
4. Lean Simulation Game “From Push to Pull”
5. Concept of Measuring the Learning Outcome of a Lean Simulation
Game
6. Application and Results of the Measuring Concept
7. Conclusion
15.11.2017 Landshut University of Applied Sciences, Germany 5

6. Problems evaluating Competence Transfer of Lean
Simulation Games
 Simulation games
in Lean education
 Few empirical
analysis of learning
effectiveness
 Subjective
measurement of
competency
development
15.11.2017 Landshut University of Applied Sciences, Germany 6
Source: own representation based on (Schenk und Wojanowski 2004; Beckmann 2004)
 Research question: Does the participation in a Lean simulation game
increase statistically significant the Lean competency of the participants?

7. Agenda
1. Introduction Technology Centre PULS
2. Problems evaluating Competence Transfer of Lean Simulation
Games
3. Measuring the Effectiveness of Simulation Games
4. Lean Simulation Game “From Push to Pull”
5. Concept of Measuring the Learning Outcome of a Lean Simulation
Game
6. Application and Results of the Measuring Concept
7. Conclusion
15.11.2017 Landshut University of Applied Sciences, Germany 7

8. Measuring the Effectiveness of Simulation Games
15.11.2017 Landshut University of Applied Sciences, Germany 8
Source: own representation based on (North und Reinhardt 2005)

9. Measuring the Effectiveness of Simulation Games
Competency Transfer Table
15.11.2017 Landshut University of Applied Sciences, Germany 9
Source: own representation according to (Tisch et al. 2014)

10. Measuring the Effectiveness of Simulation Games
Approach
1. Identification of competencies and respective activities
2. Creation of a scenario, where it is possible to carry out the activities
repeatedly
3. Communication of the starting scenario and the scenario targets to
the participants
15.11.2017 Landshut University of Applied Sciences, Germany 10

11. Agenda
1. Introduction Technology Centre PULS
2. Problems evaluating Competence Transfer of Lean Simulation
Games
3. Measuring the Effectiveness of Simulation Games
4. Lean Simulation Game “From Push to Pull”
5. Concept of Measuring the Learning Outcome of a Lean Simulation
Game
6. Application and Results of the Measuring Concept
7. Conclusion
15.11.2017 Landshut University of Applied Sciences, Germany 11

12. Lean Simulation Game “From Push to Pull”
Competencies:
 design of a waste-free internal
and external logistics system
with the use of Lean Logistics
methods
 design of a waste-free
workplace with focus on high
quality and maximum
productivity
 design of a customer oriented
and holistic production
system, based on the
principles of Lean Production
15.11.2017 Landshut University of Applied Sciences, Germany 12

13. Agenda
1. Introduction Technology Centre PULS
2. Problems evaluating Competence Transfer of Lean Simulation
Games
3. Measuring the Effectiveness of Simulation Games
4. Lean Simulation Game “From Push to Pull”
5. Concept of Measuring the Learning Outcome of a Lean Simulation
Game
6. Application and Results of the Measuring Concept
7. Conclusion
15.11.2017 Landshut University of Applied Sciences, Germany 13

14. Concept of Measuring the Learning Outcome of a Lean
Simulation Game
15.11.2017 Landshut University of Applied Sciences, Germany 14

15. Agenda
1. Introduction Technology Centre PULS
2. Problems evaluating Competence Transfer of Lean Simulation
Games
3. Measuring the Effectiveness of Simulation Games
4. Lean Simulation Game “From Push to Pull”
5. Concept of Measuring the Learning Outcome of a Lean Simulation
Game
6. Application and Results of the Measuring Concept
7. Conclusion
15.11.2017 Landshut University of Applied Sciences, Germany 15

16. Application and Results of the Measuring Concept
Pre- and Post-Test
15.11.2017 Landshut University of Applied Sciences, Germany 16
Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7
pre-test 52% 47% 49% 56% 57% 46% 44%
post-test 57% 66% 60% 60% 66% 57% 62%
30%
35%
40%
45%
50%
55%
60%
65%
70%
percentageofcorrectanswers
Results of the pre- and post-test
pre-test post-test

17. Application and Results of the Measuring Concept
Push and Pull Systems
Correlation of evaluation results of different game roles
Immersion Avg. results 1 (push) Avg. results 2 (pull)
1 = time measurement 4,21 5,53
2 = quality 3,93 5,80
3 = logistics 4,14 6,14
4 = secondary assembly 4,33 4,74
5 = main assembly 3,77 6,32
r = - 0,33 0,13
15.11.2017 Landshut University of Applied Sciences, Germany 17
 No statistically significant correlation between game role and learning
outcome

18. Application and Results of the Measuring Concept
Interviews
15.11.2017 Landshut University of Applied Sciences, Germany 18
Correlation of evaluation results of different game roles
# Interviewee 1 2 3 4 5 6 7 8 Ø
Ø in
%
Result logistics task (max. 4) 4 1 2 4 3 3 2 4 2,9 72 %
Result case study (max. 5) 5 2 3 3 4 3 1 4 3,1 63 %
Self-evaluation about attained
new competencies
(1 = not the case; 5 = fully applies)
4 3 5 4 5 5 4 4 4,3
Would you prefer a more realistic
gaming environment?
(1 = not the case; 5 = fully applies)
4 2 1 3 2 1 2 1 2,0
Correlation of self-evaluation to
objective results
r = 0,418

19. Agenda
1. Introduction Technology Centre PULS
2. Problems evaluating Competence Transfer of Lean Simulation
Games
3. Measuring the Effectiveness of Simulation Games
4. Lean Simulation Game “From Push to Pull”
5. Concept of Measuring the Learning Outcome of a Lean Simulation
Game
6. Application and Results of the Measuring Concept
7. Conclusion
15.11.2017 Landshut University of Applied Sciences, Germany 19

20. Conclusion
 Competency transfer table successful
 Growth in knowledge easily determinable
 Observation of actions during game rounds difficult
 Interviews necessary for observation of actions
 Recommendation:
parallel design of evaluation concept and simulation game
15.11.2017 Landshut University of Applied Sciences, Germany 20

21. Technology Centre PULS
(Production and Logistics Systems)
Bräuhausgasse 33 ∙ D-84130 Dingolfing
www.tz-puls.de
Stefan Blöchl, M.Eng.
Research Assistant and
doctoral candidate on the
field of Industry 4.0 in
production logistics
Mathias Michalicki, M.Eng.
Research Assistant and
doctoral candidate on the
field of Controlling for Lean