This presentation examines how the creative process can be measured using a semi-structured interview protocol.

1. Creativity Conference at Southern Oregon University
August 5, 2018
Deliberate Assessment of the
Creative Process
Lisa DaVia Rubenstein
Gregory L. Callan
Lisa M. Ridgley
Supported by the
Academic Excellence Grant from Ball State University

3. 1. Not magical muses
2. Not just art
3. For everyone
4. Can be taught/improved
Modern-ish
Creativity Tenets

4. Pre
Assessment
Strategic Intervention
Post
Assessment
(e.g., Hargrove & Nietfield, 2015; Ma, 2006; Scott, Leritz, & Mumford, 2004)

5. Pre
Assessment
DTM Strategic Intervention
Post
Assessment
Did they use the strategy?

8. Session Agenda
✤ What is creativity? (This should only take a minute.)
✤ How might the Self-Regulated Learning (SRL) Model
support studying the creative process?
✤ How do we measure it?
✤ How did we design an assessment method to study the
creative process? What did we find?
✤ How might our protocol be used in the future?

9. What is creativity?
What is it?
How to
Measure?
SRL Benefits
Our
Protocol
Our
Results
Applications

10. “Creativity is the interaction among aptitude, process, and
environment by which an individual or group produces a
perceptible product that is both novel and useful as
defined within a social context.”
(Plucker, Beghetto, & Dow, 2004, p. 90)

11. Defining the Creative Process
✤ Dual-Process Approach
✤ Stage-Based Approach

12. Anchoring Creative Process in
the Self-Regulated Learning Model
What is it?
How to
Measure?
SRL
Benefits
Our
Protocol
Our
Results
Applications

13. Self-Regulated Learning (Zimmerman)
Forethought
Performance
Self-Reflection

14. Forethought Phase
CPS Task
Performance Phase
Reflection Phase
Goal-Setting: What is the goal?
Planning: What might help you reach the goal?
Motivation: Do you believe you can do it? Are
you interested?
Strategy Use: Use creative process strategies
to support idea generation and selection.
Following stages in the Design Thinking Model.
Monitoring: Track number of ideas generated.
Track use of strategies.
Self-Evaluation: Did I generate enough quality
ideas?
Satisfaction: How do I feel about the outcomes?
Attributions: Why did/didn’t this work?
Adaptive Inferences: Should I select this model
or strategies to use in the future?
Situating the Creative Process within
Zimmerman’s (2000) Self-Regulated Learning Framework
Rubenstein, L. D., Callan, G. L., & Ridgley, L. M. (2018). Anchoring the creative process within a self-regulated learning framework:
Inspiring assessment methods and future research. Educational Psychology Review, 30, 921-945.

15. Benefits of Embedding in SRL
✤ Creative process can be learned.
✤ Creative process strategies may inspire general
learning strategies.
✤ Organize existing creativity models and studies.
✤ New methods for measuring the process.

16. Assessment of Creativity
What is it?
How to
Measure?
SRL Benefits
Our
Protocol
Our
Results
Applications

17. Heuristic Framework
(Batey, 2012)

18. Key Assessment Goals
✤ Goal 1: Represent full definition of creativity within
the creative process.
✤ Generation: Fluency, Flexibility
✤ Selection: Originality, Usefulness within a social
context

19. ✤ Goal 2: Understanding the internal process.
✤ Micro-growth: Strategies are not internalized easily,
so what is the length, the progression? Can you see
the change in process before you see an outcome
change?
✤ Exploratory Tool: Allows for comparisons between
experts and novices
Key Assessment Goals

20. ✤ Goal 3: Explore how the creative process affects (or is
affected by) outcome variables
✤ How does strategy use, strategic planning, and self-
efficacy influence outcome variables?
✤ How does the task influence the process? How does
this occur naturally? Can this be taught?
Key Assessment Goals

21. (Glăveanu, 2015)
(Gilhooly, Fioratou, Anthony, & Wynn, 2007)
Innovative Creative Process
Measurement Techniques

22. Our Protocol
What is it?
How to
Measure?
SRL Benefits
Our
Protocol
Our
Results
Applications

23. Creative Problem Solving-Microanalysis Interview Protocol
Interview
Components
Measure Description and Sample Questions
1. CPS-Intro-Tasks CPS story problems (e.g., “The school nurse accidentally ordered 1,000 boxes
of bandaids rather than 1,000 individual bandaids. Can you help the nurse
come up with different ways to use the extra boxes of bandaids?”…“Which
one idea would the nurse like best? Tell me more about that.”)
2. Strategic-
Planning
Two measures: (1) strategic planning to identify solutions and (2) strategic
planning if the task is challenging. (E.g., “Before I tell you the next problem,
what could you do to help you come up with solutions to the problem?”)
3. Self-Efficacy “How sure are you that you can create many ideas to solve the problem?”
4. Interest “How interested are you in solving these types of problems?”
5. CPS-Final-Task One CPS story problem similar to previous CPS-Intro-Tasks.
(“The concession manager accidently orders 5,000 bags of forks instead of
5,000 individual forks. Can you help the concession stand manager.....”)
6. Strategy-Use “Tell me all the things you did to help you solve this problem.”
7. Self-Evaluation “How likely is it that the stakeholder will select your best solution as the one
to implement or use?”
8. Effort “How much mental effort did you put into this problem?”

24. Creative Problem Solving-Microanalysis Interview Protocol
Interview
Components
Measure Description and Sample Questions
1. CPS-Intro-Tasks CPS story problems (e.g., “The school nurse accidentally ordered 1,000 boxes
of bandaids rather than 1,000 individual bandaids. Can you help the nurse
come up with different ways to use the extra boxes of bandaids?”…“Which
one idea would the nurse like best? Tell me more about that.”)
2. Strategic-
Planning
Two measures: (1) strategic planning to identify solutions and (2) strategic
planning if the task is challenging. (E.g., “Before I tell you the next problem,
what could you do to help you come up with solutions to the problem?”)
3. Self-Efficacy “How sure are you that you can create many ideas to solve the problem?”
4. Interest “How interested are you in solving these types of problems?”
5. CPS-Final-Task One CPS story problem similar to previous CPS-Intro-Tasks.
(“The concession manager accidently orders 5,000 bags of forks instead of
5,000 individual forks. Can you help the concession stand manager.....”)
6. Strategy-Use “Tell me all the things you did to help you solve this problem.”
7. Self-Evaluation “How likely is it that the stakeholder will select your best solution as the one
to implement or use?”
8. Effort “How much mental effort did you put into this problem?”
Tasks

25. Types ofTasks
✤ Abundance Tasks
✤ Endurance Tasks
✤ Future Ideas: Discipline/career specific tasks, scarcity
tasks

26. Creative Problem Solving-Microanalysis Interview Protocol
Interview
Components
Measure Description and Sample Questions
1. CPS-Intro-Tasks CPS story problems (e.g., “The school nurse accidentally ordered 1,000 boxes
of bandaids rather than 1,000 individual bandaids. Can you help the nurse
come up with different ways to use the extra boxes of bandaids?”…“Which
one idea would the nurse like best? Tell me more about that.”)
2. Strategic-
Planning
Two measures: (1) strategic planning to identify solutions and (2) strategic
planning if the task is challenging. (E.g., “Before I tell you the next problem,
what could you do to help you come up with solutions to the problem?”)
3. Self-Efficacy “How sure are you that you can create many ideas to solve the problem?”
4. Interest “How interested are you in solving these types of problems?”
5. CPS-Final-Task One CPS story problem similar to previous CPS-Intro-Tasks.
(“The concession manager accidently orders 5,000 bags of forks instead of
5,000 individual forks. Can you help the concession stand manager.....”)
6. Strategy-Use “Tell me all the things you did to help you solve this problem.”
7. Self-Evaluation “How likely is it that the stakeholder will select your best solution as the one
to implement or use?”
8. Effort “How much mental effort did you put into this problem?”
Forethought Questions

27. Creative Problem Solving-Microanalysis Interview Protocol
Interview
Components
Measure Description and Sample Questions
1. CPS-Intro-Tasks CPS story problems (e.g., “The school nurse accidentally ordered 1,000 boxes
of bandaids rather than 1,000 individual bandaids. Can you help the nurse
come up with different ways to use the extra boxes of bandaids?”…“Which
one idea would the nurse like best? Tell me more about that.”)
2. Strategic-
Planning
Two measures: (1) strategic planning to identify solutions and (2) strategic
planning if the task is challenging. (E.g., “Before I tell you the next problem,
what could you do to help you come up with solutions to the problem?”)
3. Self-Efficacy “How sure are you that you can create many ideas to solve the problem?”
4. Interest “How interested are you in solving these types of problems?”
5. CPS-Final-Task One CPS story problem similar to previous CPS-Intro-Tasks.
(“The concession manager accidently orders 5,000 bags of forks instead of
5,000 individual forks. Can you help the concession stand manager.....”)
6. Strategy-Use “Tell me all the things you did to help you solve this problem.”
7. Self-Evaluation “How likely is it that the stakeholder will select your best solution as the one
to implement or use?”
8. Effort “How much mental effort did you put into this problem?”
Performance Question

28. Creative Problem Solving-Microanalysis Interview Protocol
Interview
Components
Measure Description and Sample Questions
1. CPS-Intro-Tasks CPS story problems (e.g., “The school nurse accidentally ordered 1,000 boxes
of bandaids rather than 1,000 individual bandaids. Can you help the nurse
come up with different ways to use the extra boxes of bandaids?”…“Which
one idea would the nurse like best? Tell me more about that.”)
2. Strategic-
Planning
Two measures: (1) strategic planning to identify solutions and (2) strategic
planning if the task is challenging. (E.g., “Before I tell you the next problem,
what could you do to help you come up with solutions to the problem?”)
3. Self-Efficacy “How sure are you that you can create many ideas to solve the problem?”
4. Interest “How interested are you in solving these types of problems?”
5. CPS-Final-Task One CPS story problem similar to previous CPS-Intro-Tasks.
(“The concession manager accidently orders 5,000 bags of forks instead of
5,000 individual forks. Can you help the concession stand manager.....”)
6. Strategy-Use “Tell me all the things you did to help you solve this problem.”
7. Self-Evaluation “How likely is it that the stakeholder will select your best solution as the one
to implement or use?”
8. Effort “How much mental effort did you put into this problem?”
Reflection Questions

29. Initial Findings
What is it?
How to
Measure?
SRL Benefits
Our
Protocol
Our
Results
Applications

30. Methods
✤ Participants: 58 fifth and sixth graders from a rural
school in midwestern USA
✤ Mean Age = 11.3 years
✤ 81% qualified for National School Lunch program
✤ 33 identified as female
✤ Measures: ISTEP-Language Achievement Data,
Creative Problem Solving-Microanalysis Interview
Protocol (CPS-MIP)

31. Methods (cont.)
✤ Procedures: Researchers were trained in the protocol and met individually with
students. Interviews were recorded, transcribed, and coded by 2 researchers.
✤ Coding Overview (Long Process)
✤ Fluency: Number of ideas stated (Kappa = .71)
✤ Flexibility: Number of different types of ideas (Kappa = .90)
✤ Originality: Uniqueness of self-selected best idea within the sample
✤ Usefulness: Ability of best idea to meet the stakeholders’ needs (Kappa = .90)
✤ Strategies: Types of strategies used in CPS (Kappa = .75)

32. ✤How were strategies
coded?

33. Research Questions
✤ RQ1a: To what extent do all SRL processes measured by our protocol
relate to and predict the generation of ideas (i.e., fluency and
flexibility)?
✤ RQ1b: What specific strategies influence students’ generation of
ideas?
✤ RQ2a: To what extent do all SRL processes measured by our protocol
relate to and predict the selection of ideas (i.e., originality and
usefulness)?
✤ RQ2b: What specific strategies influence students’ selection of ideas?

34. Considering
Generation of Ideas

35. Predicting Fluency

36. ✤To what extent do
specific strategies
discussed during
strategic planning
correlate with
generation
outcomes?

37. Detailed Follow-Up:
Using Specific Strategies to Predict Fluency
✤RQ2

38. ✤To what extent
does reported
specific strategy
use correlate with
fluency?

39. Fluency Summary
✤ Forethought variables: Planning (general and stuck)
and Self-Efficacy Mattered (55%)
✤ Perspective-Taking was most important individual
strategy in the planning stage.
✤ Specific brainstorming and past experiences mattered
most in the performance phase.

40. Predicting Flexibility

41. ✤To what extent do
specific strategies
discussed during
strategic planning
correlate with
generation
outcomes?

42. Detailed Follow-Up:
Using Specific Strategies to Predict Flexibility
✤RQ2

43. ✤To what extent
does reported
specific strategy
use correlate with
specific creativity
outcomes?

44. Flexibility Summary
✤ Self-Efficacy and Planning (general) mattered most
(44%).
✤ Perspective-Taking, Brainstorming, and Help-seeking
were specific planning strategies that mattered.
✤ No individual, specific performance strategies were
significantly correlated to flexibility.

45. Considering the
Selection of Ideas

46. Predicting Originality

47. ✤To what extent do
specific strategies
discussed during
strategic planning
correlate with
originality?

48. ✤To what extent
does reported
specific strategy
use correlate with
specific creativity
outcomes?

49. Originality Summary
✤ Planning-Stuck and Self-Evaluation mattered in
originality performance (27%).
✤ Re-examine the problem, experimentation were
important planning strategies.
✤ Perspective-taking mattered during the performance
stage.

50. Predicting Usefulness

51. ✤To what extent do
specific strategies
discussed during
strategic planning
correlate with
usefulness?

52. ✤To what extent
does reported
specific strategy
use correlate with
usefulness?

53. Usefulness Summary
✤ Self-Evaluation mattered in usefulness performance
(12%).
✤ No significant planning strategies.
✤ Perspective-taking mattered during the performance
stage.

54. Discussion
✤ Application of New Measurement Technique
✤ Processes differentially directly influence outcomes. Generation more easily
predicted than selection.
✤ Strategic Planning and Use Findings
✤ Fragmented, informal strategies
✤ Certain strategies may be differentially adaptive for creative
outcomes.
✤ Popular Strategies: Help-seeking and consulting past experiences
✤ Most Adaptive Strategy: Perspective Taking

55. Limitations
✤ Sample size, diversity
✤ Limited options for strategy use given task constraints
✤ Limited repertoire of strategies (age and lack of
intervention)

56. FutureWork
✤ Additional Samples (e.g., experts, novices)
✤ Different Tasks (e.g., domains, TTCT, longer time)
✤ Deliberate exploration of selection of ideas
✤ Interventions with Perspective Taking with New
Measurement Possibilities (Galinsky, Maddux, Gilin, & White, 2008; Mohrman,
Gibson, Mohrman, 2001; Doron, 2017)

57. Sample Application
What is it?
How to
Measure?
SRL Benefits
Our
Protocol
Our
Results
Application

58. Use in Development of Creative
Thinking Course
Pre
Mid
Strategies e.g.,
DTM, SCAMPER, Analogical Thinking

59. Use in Development of Creative
Thinking Course
Pre
Mid
Wait…WHAT????

60. Use in Development of Creative
Thinking Course
Pre
Mid
Post
Strategies

61. Thank you.
Questions? Thoughts? Ideas?
Contact: Lisa Rubenstein
(lmrubenstein@bsu.edu)Supported by the
Academic Excellence Grant from Ball State University