This document outlines a study that aims to identify variations in how people experience dealing with problems in complex systems. It will do so using a phenomenographic research approach, which seeks to understand the limited number of qualitatively different ways a phenomenon can be experienced. 20 participants, including students and professionals, will complete thinking-aloud tasks centered around complex systems problems. The results will be analyzed using Marton's approach to identify hierarchies of increasingly powerful ways the tasks are understood. The study expects to produce one of the first cognitive theories of systems thinking development from an engineering education perspective, with potential impacts for assessment, curriculum design, and supporting claims of current programs.

1. Identifying the variation of
experiences when dealing with
problems in complex systems.
John Mendoza Garcia1, 2, Monica E. Cardella1
William C. Oakes1
1Purdue University, IN
2Pontificia Universidad Javeriana, Bogotá,
Colombia, South America

2. Agenda
• Introduction
• Summary of literature
• Research Approach
• Impact
8/28/2015 Tool to Assess Interest in PP 2

3. Introduction
8/28/2015 Tool to Assess Interest in PP 3
• Introduction
• Summary of
literature
• Research
Approach
• Impact
High order
thinking skills
Technical
knowledge
• Creativity
• Critical
thinking
• Design thinking
• Problem
solving
• Systems
thinking

4. Who is calling?
Eng.
Professional
societies
Eng.
Academics
Management
Scientist
Thought
leaders in Eng
Ed.
Industry ABET
Policy makers
• Introduction
• Summary of
literature
• Research
Approach
• Impact

5. Why?
8/28/2015 Tool to Assess Interest in PP 5
Technological
systems
Environment
Human
systems
designing and
implementing solutions
– Sustainable products
– Processes
– Organizations
– Public policies.
– Etc.
• Introduction
• Summary of
literature
• Research
Approach
• Impact

6. We have been teaching
Systems Thinking
for several years…
8/28/2015 Tool to Assess Interest in PP 6
• Why perception says… engineers lack its development?
• Why research findings show failure in its learning?
But…
• Introduction
• Summary of
literature
• Research
Approach
• Impact

7. Curriculum design
8/28/2015 Tool to Assess Interest in PP 7
Effective curriculum design
Achievement of
learning goals
• Introduction
• Summary of
literature
• Research
Approach
• Impact

8. Curriculum design
8/28/2015 Tool to Assess Interest in PP 8
Effective curriculum design
Achievement of
learning goals
How can we know if my
curriculum is working??
Guilty?
• Introduction
• Summary of
literature
• Research
Approach
• Impact

9. 8/28/2015 Tool to Assess Interest in PP 9
Curriculum design
8/28/2015 TravelGrant 9
Content
Assessment Pedagogy
We need to know effectiveness of teaching of
Systems Thinking• Introduction
• Summary of
literature
• Research
Approach
• Impact

10. Assessment Triangle (Pellegrino, 2002)
8/28/2015 TravelGrant 10
Observation Interpretation
Cognition
• Introduction
• Summary of
literature
• Research
Approach
• Impact

11. Research question
Iteration 1
–How people develop systems
thinking?
• Introduction
• Summary of
literature
• Research
Approach
• Impact

12. What do we know about the development of
systems thinking and its assessment?
- Studies from expert’s point of view
- Lineal path based on Bloom: Hooper and Stave (2008)
- Studies from learner’s point of view (qualitative)
- Frank, 2000
- 30 Laws for Capacity for Engineering systems thinking
- Richmond, 2001
- A subset of 7 critical thinking skills
- Davidz, 2006
- experiential learning, individual characteristics and
supportive environments key to develop systems thinking
• Introduction
• Summary of
literature
• Research
Approach
• Impact

13. Variation theory of learning
learning is about an increase of the amount of
critical aspects and features the learner can
discern and be focused on simultaneously
[from a dimension or conception]. (Marton,
2014, p. 60
Learning can be seen as moving from one
way of seeing and dealing with the
problem to another more powerful
(Marton, 2014)
• Introduction
• Summary of
literature
• Research
Approach
• Impact

14. 8/28/2015 Tool to Assess Interest in PP 14
This study’s research question
“what are the variations in the
ways people experience dealing
with problems in complex
systems?”
• Introduction
• Summary of
literature
• Research
Approach
• Impact

15. 8/28/2015 Tool to Assess Interest in PP 15
• Introduction
• Summary of
literature
• Research
Approach
• Impact
Engineering complex systems
Complexity
Environmental
Technological
Humans

16. 8/28/2015 Tool to Assess Interest in PP 16
• Introduction
• Summary of
literature
• Research
Approach
• Impact
Development of systems thinking
from learner’s point of view
Davidz, H,. (2006)
Graphic from Richmond.
(1990)

17. 8/28/2015 Tool to Assess Interest in PP 17
Systems thinking inventory
and its test
Identify intuitive systems
thinking
Measuring change of
mental models
Assessment of
systems thinking
• Introduction
• Summary of
literature
• Research
Approach
• Impact

18. 8/28/2015 Tool to Assess Interest in PP 18
Variation theory of
learning
Transfer of learning Conceptual change
Cognitive/constructivist
learning theories
• Introduction
• Summary of
literature
• Research
Approach
• Impact

19. 8/28/2015 Tool to Assess Interest in PP 19
• Introduction
• Summary of
literature
• Research
Approach
• Impact
Phenomenography
• Understanding the variation in how “we “see” the
world and how it “shows itself” to us (Marton,
2014, p. 122)
• Main assumption
– limited number of qualitatively different ways

20. 8/28/2015 Tool to Assess Interest in PP 20
• Introduction
• Summary of
literature
• Research
Approach
• Impact
Phenomenography
Figure 1- Points of departure between phenomenography
and other research approaches. Trigwell (2000)

21. 8/28/2015 Tool to Assess Interest in PP 21
Data Collection
How many participants?
– I will recruit 20 participants for this study
• 10 students
• 10 professionals
20 – 30 (Bowden, 2005)
10-15 (Trigwell, 2000)
10 (Harding, 2013)
20 (Sandberg (2000), citing
Alexandersson (1994) who cited
more than 1000 research and 50
doctoral theses)
17 Sandberg (2000)
• Introduction
• Summary of
literature
• Research
Approach
• Impact

22. 8/28/2015 Tool to Assess Interest in PP 22
Data Collection
• Undergraduate students
– Random survey to capture variation
• Graduate/Professionals
– Purposeful sampling
– maximize variation
• Criteria for variation
– Framework: Enablers of systems thinking
(Davidz,2006)
• Introduction
• Summary of
literature
• Research
Approach
• Impact

23. 8/28/2015 Tool to Assess Interest in PP 23
Data Collection
• Think aloud
• 5 tasks tested – 2 chosen.
– Scene it!
– Wolves and Rabbits (L. B. Sweeney & Sterman, 2007)
– The manufacturing case: (L. Sweeney & Sterman, 2001)
– The coffee challenge (Purzer, Ş. & Wertz, 2013).
– Disaster relief scenario (Adams et al., 2010)
• Introduction
• Summary of
literature
• Research
Approach
• Impact

24. 8/28/2015 Tool to Assess Interest in PP 24
Data Analysis
Marton’s approach
- Look for different ways to address the problem
- Have them in one place
- Rip them off their context and the participant
- Define different categories
- Define a hierarchy
- Simple to powerful
• Introduction
• Summary of
literature
• Research
Approach
• Impact

25. 8/28/2015 Tool to Assess Interest in PP 25
Data Analysis
Marton’s approach
- Look for different ways to address the problem
- Have them in one place
- Rip them off their context and the participant
- Define different categories
- Define a hierarchy
- Simple to powerful
• Introduction
• Summary of
literature
• Research
Approach
• Impact

26. 8/28/2015 Tool to Assess Interest in PP 26
Data Analysis
• Introduction
• Summary of
literature
• Research
Approach
• Impact
Expected outcome
Ways to address the task, from less to
more powerful

27. 8/28/2015 Tool to Assess Interest in PP 27
• Introduction
• Summary of
literature
• Research
Approach
• Impact
• Maybe one of firsts theories cognitive
theories about systems thinking
development from engineering education
• Several disciplines and fields in academy
and in industry
– Assessment!
– Curriculum design
Impact

28. 8/28/2015 Tool to Assess Interest in PP 28
• Introduction
• Summary of
literature
• Research
Approach
• Impact
• Current programs could support claims
• Application of variation theory of learning
Impact

29. 8/28/2015 Tool to Assess Interest in PP 29
Thank you …