This document summarizes a presentation on lessons learned from embedding design into a developmental evaluation. The presentation discussed the challenges of developmental evaluation and using a design framework to conceptualize an education program for novice educators. An example was provided of developing a graduate program through developmental evaluation. Key lessons included the importance of balancing power, managing ownership of ideas when ideas don't work out, and ensuring the organizational culture supports flexibility, testing ideas, and allowing failure.

1. L E S S O N S - L E A R N E D F R O M
E M B E D D I N G D E S I G N I N T O A
D E V E L O P M E N T A L E V A L U A T I O N :
C H I Y A N L A M , P H D ( C ) , C E
A E A 2 0 1 5 . F R I D A Y , N O V E M B E R 1 3 , 2 0 1 5 ( 8 : 0 0 A M - 9 : 3 0 A M )
S E S S I O N 1 4 1 4 : D E S I G N T H I N K I N G F O R E X E M P L A R Y E V A L U A T I O N : T H R E E
E X A M P L E S O F D E S I G N F O R E V A L U A T I O N A N D O R G A N I Z A T I O N A L L E A R N I N G
T H E S I G N I F I C A N C E O F
P O W E R , O W N E R S H I P , A N D
O R G A N I Z A T I O N A L C U L T U R E

2. E V A L U A T I O N & D E S I G N

3. Summative
Evaluation

4. Formative
Evaluation

5. Developmental Evaluation
• Program
Development:
ongoing shaping and
re-shaping of a
program in response
to emerging
evidence
• Developmental
Evaluation (Patton, 1994)

6. (Patton, 2011)
• DE is laden with
conceptual and
practical
challenges. (Not a
criticism of the
approach, but an
indication of its
complexity). !
• What questions
do you actually ask
in a developmental
evaluation?!
Developmental Evaluation

7. T H E
D E V E L O P M E N T A L
E V A L U A T I O N C A S E
• Faculty of Education
• Graduate-level Diploma
and Professional Masters in
Education
• To support the
conceptualization, planning,
and ‘decision-making’

8. T A S K F O R C E
a program “in theory” a program “in action”

9. H O W M I G H T W E …
D E V E L O P A P R O G R A M T H A T
S U P P O R T S N O V I C E E D U C A T O R S I N
E M B R A C I N G P R O F E S S I O N A L
I N Q U I R Y T H R O U G H T E C H N O L O G Y ?
a program in theory a program in action

10. I N C O N V E N T I O N A L
P L A N N I N G T E R M S …
I N A D E S I G N F R A M E …
H O W M I G H T W E …
C O U R S E W O R K
W H A T D O W E T E A C H ? W H A T
A R E T H E L E A R N I N G
O U T O C M E S ?
C O N S T R U C T M E A N I N G F U L
L E A R N I N G E X P E R I E N C E S
T H A T E N G A G E O U R
L E A R N E R S I N K N O W L E D G E
C O N S T R U C T I O N ?
L E A R N I N G M A N A G E M E N T
S Y S T E M
W H I C H S Y S T E M D O W E
A D O P T ?
L E V E R A G E T H E
F U N C T I O N A L I T Y O F L M S A N D
E X T E R N A L T O O L S T O E N A B L E
P A R T I C U L A R K I N D S O F
L E A R N I N G ?
M A R K E T I N G
W H A T C H A N N E L S A N D T O O L S
D O W E U S E T O S E L L T H E
P R O G R A M ?
B E S T H E L P P O T E N T I A L
S T U D E N T S L E A R N A B O U T T H E
F E A T U R E S A N D V A L U E O F
O U R P R O G R A M ?
M A R K E T S
W H O D O W E T A R G E T ? W H O
W O U L D ‘ B U Y ’ T H I S ?
I D E N T I F Y P R O G R A M U S E R S
W H O C O U L D B E N E F I T F R O M
T H I S ?

11. U S E R - C E N T E R E D
P E R S P E C T I V E
T H I N K
C R E A T I V E L Y
P R O D U C T I V E
P R O B L E M A T I Z I N G

12. U S E R - C E N T E R E D
P E R S P E C T I V E
T H I N K
C R E A T I V E L Y
P R O D U C T I V E
P R O B L E M A T I Z I N G
R E A L I T Y -
T E S T
H O L D O U R
I D E A S
L I G H T L Y

13. P O W E R • Design-informed DE
requires buy-in from
leaders.
• necessitates
participation from all
levels
• the role of an
evaluator, is
sometimes, to
rebalance and shift
the power

14. O W N E R -
S H I P
• investment in the
design/development
process means that
staff can feels strongly
about the ideas
• when the idea doesn’t
work in reality, it can
be difficult to accept
• speaking truth
requires a delicate
touch

15. O R G A N I -
Z A T I O N A L
C U L T U R E
• Attention to process is not
enough.
• Structure matters
• Traditional structure:
• budgeting
• committee
• cross-functional team
• Consider “intrapreneurship"
• flexibility to pivot and
change ideas
• resource to test and fail.
• permission to fail

16. Q U E S T I O N S ?