My recent keynote to Japan's great new consortium to promote entrepreneurship education, focusing here on how to assess its impact/outcomes. Fit nicely with my project with EU called "EPIC" (http://epic.ecorys.com) Great audience, great questions - when the video is published, i'll link to it!

2. An EPIC Adventure:
How Should We Assess
Entrepreneurship Education?
Norris Krueger, PhD
Nagoya, October 2018
Norris.krueger[at]gmail.com

3. Great programs do great assessment
• Or do they?
• What makes an entrepreneurship education
program great?
• Evidence from primary/secondary
• Assessment is not in a vacuum
• Evidence from research – not encouraging
– https://journals.aom.org/doi/10.5465/amle.2015.0026
• One impressive data point, though…

4. Evidence from primary/secondary
• What do great programs all share?
• Constructivistic approaches (e.g., project-
based learning)
• Deep engagement with external communities
• Student-led, student-designed
• Highly-trained instructors
• And.. productive assessment (not rote)
– Why not assess on these criteria?

5. Evidence from the Netherlands
• http://www.bit.ly/OECDdelta
• Key takeaways of relevance here:
– Some great programs to emulate but…
– Ecosystem matters.. Deeply (mindset<->ecosystem)
– Putting stakeholders first
– Building networks/co-immersion
– Evidence-based (before, during, after)

6. Learning is a Process
• Inputs  Throughputs  Outputs
• Assess all three?
• KEY INPUTS
– Students, Instructors, Pedagogy, Setting (& ?)
• KEY THROUGHPUTS
– Key activities (are we doing the right things the
right way?)
• KEY OUTPUTS (where we usually focus)

7. the neuroscience of learning:
Novice  Expert

8. Cognitive load?
• Great education optimizes learner’s cognitive
load…
https://www.science20.com/carl_wieman/scientific_approach_science_education_reducing_cognitive_load
• Why not measure that , given its importance,
what unique insights?
• What other educator/student pedagogical
behaviors? (use of specific tools )
• AND…
• How could we use neuroscience tools – fMRI,
EEG, etc.?? (London cabbies) Can we assess
SKILLS? https://www.wsj.com/articles/brain-scans-can-detect-
who-has-better-skills-1538589600

9. Other cognitive measures
• Cognitive complexity
• Cognitive flexibility
• Cognitive reflection
• “Hot” vs. “Cold” cognition
– https://www.nature.com/articles/456168a ;
http://www.alanbarrell.com/uploads/Nature%20Commentary.pdf
• Changes in working memory?? (does high-
uncertainty/high-velocity/high-stakes decision making
experiences expand working memory?)
• Metacognition! https://catlintucker.com/2018/09/4-
strategies-metacognitive-thinking/ <-incredibly
important for PBL and entrep ed!

10. Want to be EPIC?
• EEEPHEIC (Evaluating Entrepreneurship Education
Program Impact in Higher Education Consortium)
• EU’s DG-eac
• Diverse core team plus 35 country informal
advisory board plus student advisory board!
• I encourage you to take a look – and fill out their
pilot study at http://epic.ecorys.com
• (see also www.bit.ly/EEEPHEIC1 )
POC for EPIC: Gabi.Kaffka@ecorys.com

11. Pre-EPIC efforts
• Entrepreneurship 360 – OECD’s 360 degree
assessment of primary/secondary programs
http://www.oecd.org/site/entrepreneurship360/
• HE Innovate – higher ed innovation (a very
useful tool there)
https://ec.europa.eu/education/resources/heinnovate_en
Key contact: Andrea Hofer of OECD, andrea-
rosalinde.hofer@oecd.org

12. Methods?
• Quantitative
• Qualitative – reflective journals!
– https://digitalknowledge.babson.edu/fer/vol32/iss6/12/
– https://digitalknowledge.babson.edu/fer/vol34/iss4/15/
• Neurological? (again, London cabbies)
– www.academia.edu/download/43550664/Neuroentrepreneurship_2014.pdf
– (but need not involve fancy tech; e.g., field experiments)
Triangulation!
Method?
By level?? (action vs. thinking vs. biological)
By type of impact???

13. An EPIC Toolkit
10/17/2018 13
Mindset
What
else?
Behavior
External
Stakeholder
impact
Skills
Knowledge
› Does the entrepreneurial
mindset change? In what
ways?
› Do learners acquire
more/different skills? Do
they improve existing skills?
› Do learners know more?
What are critical knowledge
items/categories?
› P.S. Metrics need NOT be
quantitative; we can have
great qualitative KPIs
›What new behaviors are
exhibited? What behaviors
are more (less) frequent?
›What are the external
impacts? On each of the
critical stakeholders?
›What else? What are you
measuring that is not on this
list?
LOREM IPSUM

14. Behaviors
DIRECT?
• Start a business? Other entrepreneurial acts?
INDIRECT?
• Intrapreneurial?
• Mentoring/Volunteering?
PERSONAL ENTREPRENEURIAL ORIENTATION?
Risk-taking, Innovativeness, etc. but behavorial
PSED list of “concrete behaviors” (nascent E’s)

15. PSED/GEM list
• Have marketing or promotional efforts been
started for the product or service this new
business will be selling?
• Has this new business developed any
proprietary technology, processes, or
procedures that no other company can use?
• http://www.psed.isr.umich.edu/psed/downlo
ad_node/166

16. Knowledge
• Weakest measure –highly subjective &
hostage to many biases
• Myths and misconceptions?
• Script cues? (e.g., Abelson)

17. Script cues from Mitchell… (48 items)
•
•
• 1. I am rarely surprised by:
• (a) developments in a new business
• (b) human nature
• 2. Are you more attracted to people who are:
• (a) ready to take action
• (b) thoroughly informed
• 3. I have more highly developed contacts in the:
• (a) new venture area specifically
• (b) community generally
• 4. If asked to give my time to a new business I would decide based on how this venture fits:
• (a) into my past experience
• (b) my values
• 5. There are:
• (a) times when after I finish a job I wish that I had done it better, or worked harder at it
• (b) never any jobs or tasks I complete which need more work
• (items available upon request)

18. Entrepreneurial Mindset
• “noncognitive” = deep cognition ;)
• Evidence/lessons from neuroscience
– NOT intent, much more
– DEEP cognitive beliefs/anchoring assumptions
– That we construct, often not mindfully, and influenced by
context and social influences… we buiild/change knowledge
structures based on context and people
– Multiple facets/dimensions
– Indirect measures -> Markers that reflect beliefs
http://www.bit.ly/ETP2007
APPLIED: http://bit.ly/EntrepEdChapt
https://www.researchgate.net/publication/242175809_Neuroentrepreneurs
hip_Can_Entrepreneurship_Learn_from_Neuroscience_1
https://www.researchgate.net/publication/276892928_In_Search_of_the_En
trepreneurial_MindsetInsights_from_Neuroscience

19. measurable mindset facets*
• Action-orientation/proactiveness
• Innovativeness (presumably discontinuous)
• Resilience to adversity/optimism
• Persistence at goal-directed behaviours
• Domain-specific self-efficacy (possibly general self-efficacy)
• Role identity (mental prototypes)
• Entrepreneurial intensity
• Tolerance for ambiguity and uncertainty /risk-aversion (lower)
• Future orientation (ability to delay gratification?)
• Entrepreneurial intentions
• Value creation (versus opportunism)
• Market orientation
• Cognitive flexibility
*Research instrument available upon request

20. Skills
• EntreComp!
• https://ec.europa.eu/jrc/en/publication/eur-
scientific-and-technical-research-
reports/entrecomp-entrepreneurship-
competence-framework
• POC for EntreComp:
Margherita.Bacigalupo@ec.europa.eu

22. External impacts?
• Who are your critical stakeholders?
– Students?
– Alumni?
– Economic development?
– Civic community?
– Media??
– Funding entities?
• What do they consider success?
• Think Value Proposition for them
• https://www.slideshare.net/norriskrueger/how-to-be-your-
ecosystems-liaison-animateur
• This might be the right place to start?

23. Policymakers WILL recognize great
experiential learning
In part because we documented impact on the community
http://bit.ly/TEAMSovw & http://bit.ly/EntrepEdChapt

24. A little homework!
• Identify one important stakeholder that
needs to be persuaded that world-class
entrepreneurship education is valuable.
• What metric/KPI/marker would persuade
them or make them happy?

25. So how do you assess?
inputs throughputs outputs
behaviors
knowledge
skills
mindset
external impact
other?
Which do you WANT to assess?

26. Additional reading
• http://www.oecd.org/cfe/leed/Entrepreneurial-Education-Practice-
pt1.pdf
• http://www.oecd.org/cfe/leed/Entrepreneurial-Education-Practice-
pt2.pdf [the great Andy Penaluna!]
• https://www.oecd.org/cfe/leed/BGP_Entrepreneurship-in-
Education.pdf [the also great Martin Lackeus, see also his youtube channel!]
• http://www.bit.ly/TEAMSovw [my old PBL program]
My first chapter on this… http://bit.ly/EntrepEdChapt
• https://www.researchgate.net/publication/305876584_New_horizo
ns_in_entrepreneurship_from_teacher-led_to_student-
centered_learning [from Helle Neergaard’s team]
• Other good research names: Colin Jones, Francisco Linan, Alain
Fayolle, Elena Ruskovaara, Chih Hsieh, Helle Neergaard
• Other good educators: Nigel Adams, Doan Winkel, Elin McCallum,
Aard Groen, Boo Edgar, Alex Bruton, Suresh Kumar, Padmakumar
Nair, Mats Lundqvist, Jing Zhang [China], Dafna Kariv

28. How can I help you to be EPIC?
Norris Krueger, PhD
+1 208 440 3747 (also on WhatsApp)
Norris.krueger@gmail.com
@entrep_thinking
LinkedIn, Facebook
Join the adventure!