Making social innovation work
Day 3
Edward Gardiner
Day 3
09.30-10.00 Recap on Day 2
10.00-10.30 The three strands of value
10.30-11.30 Social business models
11.30-12.00 Break
12.00-13.00 Experimenting with social impact
13.00-14.00 Lunch
14.00-14.30 Feedback and discussion
14.30-15.30 Activity: Business model/Measuring
15.30-16.30 Activity: Planning for the future
The three strands of value
Entrepreneurship
The pursuit of opportunity
beyond the tangible
resources that you currently
control.
Emphasis on opportunity
recognition, committing to
the opportunity, gaining
control over resources.
Often focuses on
breakthroughs and new
needs.
Social
entrepreneurship
Innovative, social value
creating activity that can
occur within or across the
non-profit, business, or
government sectors.
The drive is to create social
value, rather than personal
or shareholder wealth.
Often focuses on serving
basic, long-standing needs
more effectively.
Stevenson (1983), Austin et al (2006)
IDEO
Shift Design (2014)
Theoretical differences
• Market failure: emerge when there is a social-
market failure, i.e. commercial market forces
do not meet social need.
• Mission: creating social value for the the
public good vs. creating profitable operations
resulting in private gain.
• Resource mobilisation: ability to access
financial resources and capital markets.
• Performance measurement: measuring social
impact is harder than tangible and objective
financial measures.
Austin et al (2006)
Austin et al (2006)
Social business models
Design Council
Nesta
Strategyzer
Tandemic
Big Society Capital
Acumen
Gov.uk
Different legal structures
UnLtd
Types of business model
• Fee for service: Sell a good or service
directly to beneficiaries.
• Connector: Help facilitate trade
relationships.
• Intermediary: Help beneficiaries distribute
to an expanded market.
• Cross-compensation: Sell a good to one
group in order to subsidise another.
• Hybrid: Sell a good to one market in order
to support beneficiaries in another.
M-KOPA Solar (fee for service)
• Founded by Nick Hughes, Chad Larson and Jesse
More in 2011.
• Customers pay a deposit of $35 and then pay $0.5 per
day for a period of one year to own the solar system
using M-PESA.
• $200 device includes two LED bulbs, a flashlight,
rechargeable battery and adaptors for phone charging,
and is charged by a solar panel on the roof.
• Meant for off-grid customers using kerosene lamps.
• Connected over 500,000 homes to affordable solar.
• Current customers projected to make USD$375m
savings over the next four years, and enjoy 62.5
million hours of kerosene free lighting per month.
M-Kopa / Acumen
Kiva.org
Kiva.org (connector)
• Founded by Matt Flannery and Jessica Jackley in
2005 following a trip to Uganda.
• They met many entrepreneurs who wanted to
share their stories with friends and family.
• Mission to connect people and provide
entrepreneurs with dignity and respect.
• Local microfinance institutions worked with
entrepreneurs to get their business on Kiva’s
website. Investors anywhere could choose
entrepreneurs to loan money to.
• Over $1 billion lent, lenders and entrepreneurs in
150 countries worldwide.
Aaker & Chang (2010)
Social Innovation Lab
Marketing platform
development
Entrepreneurs in
developing
countries
# of entrepreneurs
$ lend
$ repaid
# dependents
Create jobs, reduce
poverty
Borrower
Family of
borrower
Well-off individuals
who are socially
conscious
Service
Kiva.org
Repayment risk
Managing
risk
Managing
platform
Screening MFIs
Optional donations
Collecting loan
capital
Proving loans
Kiva.org platform
Local MFIs
Payapl
Donor network
Brand
Financial ability to
start business
Promote market
based solutions to
povertyLocal microfinance
institutions
Ten Thousand Villages
(intermediary)
• Non-profit fair trade organisation founded in
1946 (one of the largest and oldest).
• Markets crafts made by disadvantaged
artisans from over 30 countries.
• Over USD$140m has been earned by
makers who would otherwise be unemployed.
• Income earned by the producers helps to pay
for food, education, healthcare and housing.
• Price mutually agreed with upfront payment,
builds long term partnerships, ensures good
working conditions, emphasise reuse/recycle.
Ten Thousand Villages
Ziqitza (cross-compensation)
• Mumbai and other cities in India lack reliable
ambulances or emergency services.
• Ziqitza Health Care Ltd founded in 2005 by group of
young professionals, providing 24/7 call centres and
ambulance tracking.
• In Mumbai, operates a sliding price scale that depends
on a customer’s ability to pay and circumstance
(accident victims travel for free).
• ZHL operaties 980 ambulances across five states, has
served over 2 million people, has been awarded $80m
in government contracts.
• Was the first ambulance services to arrive at the Taj
and Trident hotels during the Mumbai attacks in 2008.
Ziqitza / Acumen
Hot Bread Kitchen (hybrid)
• Founded by Jessamyn Rodriguez. Bakers in
training programme launched in 2007, and HBK
Incubates launched in 2011.
• Employees, mostly low-income immigrant women,
bake bread inspired by their countries of origin.
• They learn job skills that can lead them to
management positions in the food industry.
• Bakery is widely acclaimed for its innovative
selection of international breads.
• 65% of operating budget is funded by bread sales
and kitchen rental. Philanthropic support makes it
possible to provide industry-specific training.
Hot Bread Kitchen
Other models
• Add ons (e.g. airlines)
• Advertising (e.g. online publishing)
• Auctions (e.g. eBay)
• Bait and hook (e.g. razors, coffee makers)
• Franchise (e.g. TED talks, restaurants)
• Freemium (e.g. mobile gaming)
• Long tail (e.g. micro-breweries)
• Low cost (e.g. airlines, credit cards)
• Subscription (e.g. health clubs)
Break
Experimenting with social
impact
How experimentation can be used to
understand what works (and improve
what doesn’t)
Three types of question
• Descriptive: describe what is going on or
what exists.
• Relational: look at the relationships
between two or more variables.
• Causal: determine whether one variable
causes or affects one or more outcomes.
“In the 20th century, randomized, controlled
trials have revolutionized medicine by
allowing us to distinguish between drugs that
work and drugs that don't work. And you can
do the same randomized, controlled trial for
social policy. You can put social innovation to
the same rigorous, scientific tests that we use
for drugs.”
Esther Duflo
Abdul Latif Jameel Poverty Action Lab
TED (2010)
First principle of science
To demonstrate whether an activity causes
an outcome to occur, you need to provide
evidence for two statements:
1. If the activity is given, then the outcome
occurs.
2. If the activity is not given, then the
outcome does not occur.
Treatment group
Control group
Drug
Placebo
time
1. RANDOMISE 2. IMPLEMENT 3. MEASURE
BETWEEN GROUPS
WITHIN GROUP
Sample
Strength of conclusion
Intervention
What you do
Observation
What you see
Causes
Alternative
causes
Alternative
causes
LAB ENVIRONMENT
Is random
assignment used?
Randomised or
true experiment
Is there a control
group or multiple
measures?
Quasi-experiment
or matched trial
Non-experiment
Yes No
NoYes
Who are the sample?
• Citizens, staff, customers, volunteers,
managers, people.
• Is it a natural experiment or will you
need to provide an incentive to
participate?
• Do you have enough people or will
there be a cost of recruitment?
• Are they representative of the
population?
How will you randomise?
• Assign individuals, clusters, locations
or areas.
• Is it possible to randomise individuals
or cluster to different groups?
• Is it possible to personalise what
individuals or clusters receive?
• Will different groups know what the
other groups are getting?
What are you testing?
• New intervention versus nothing,
current practice or different variations.
• Is there existing evidence to support
what you’re testing?
• Is the intervention an incremental
change or a radical new approach?
• Are you testing proof of concept or the
final version before scaling up?
What are you measuring?
• Attitudes, experiences, decisions or
actions.
• Psychological scales, surveys and
observation.
• Are they subjective or objective
measures?
• Are they direct or proxy measures?
• When will the measures be taken?
61m person experiment in social
influence
Context
Voter turnout is significantly correlated among
friends, family members and co-workers,
although anecdotal accounts suggest that online
mobilization has made a big difference in recent
elections, a meta-analysis of email experiments
suggests that online appeals to vote are
ineffective.
Method
• 18 yrs who accessed on 2 Nov 2010.
• A: Social (n=61m, message, poll
information, counter and button, friends).
• B: Informational (n=600k, as above, no
faces of friends).
• C: Control (n=600k, no message).
Bond et al (2012)
Incentives for immunisation
Context
Every year an estimated 2 to 3 million people die
from vaccine-preventable diseases. A lack of
understanding of the benefits or even suspicion
are possible reasons rates remain low in India.
Cash incentives are effective but too expensive.
Can small incentives have a big impact on
immunisation rates?
Method
• Clustered randomised controlled trial with
174 villages in Udaipur, India.
• Intervention A (30): mobile immunization
team conducted monthly camps.
• Intervention B (30): same but parents also
offered 1kg of lentils each time.
• Control group (74): nothing.
Banerjee et al. (2010)
Reducing aggression in A&E
Context
Violence and aggression towards frontline staff,
including 59,000 physical assaults, is estimated
to cost the NHS at least £69 million a year in
staff absence, loss of productivity and additional
security. How can better visual communication
reduce frustration, anxiety and abuse towards
staff?
Method
• Pre/post, matched controlled trial across
four NHS Trusts in the UK.
• Pilot sites (2): signage and people project
design solutions implemented.
• Comparative control sites (2): no design
solutions implemented.
Design Council / Frontier Economics (2014)
The questions shape the answers
• How short was
Napoleon?
• Should concerned
parents use infant car
seats?
• Where do you enjoy
drinking beer?
• Do you always eat
breakfast?
• How would you describe
Napoleon’s height?
• Do you think special car
seats should be required
for infant passengers?
• What do you normally like
to drink?
• How many days a week
do you eat breakfast?
Survey Monkey
Correlation ≠ causation
Intervention
What you do
Observation
What you see
Correlation
State of the world
Not what you do
Tyler Vigen
Tyler Vigen
Regression to the mean
A statistical
phenomenon that if a
variable is extreme on
its first measurement,
it will tend to be
closer to the average
on its second
measurement, and
vice versa.
Research methods knowledge base
Extrapolating beyond data
xkcd
BBC News
OK Cupid
Netflix
Debunking some myths
1. We don’t necessarily know ‘what works’
2. RCTs don’t have to cost a lot of money
3. There are ethical advantages to using
RCTs
4. RCTs do not have to be complicated or
difficult to run
Test, Learn Adapt (Cabinet Office)
Lunch
Feedback and discussion
Discussion
Business model canvas
Strategyzer
Activity
Measuring what works
Three types of question
• Descriptive: describe what is going on or
what exists.
• Relational: look at the relationships
between two or more variables.
• Causal: determine whether one variable
causes or affects one or more outcomes.
Activity
Planning for the future
Project premortem
1. Leader informs the room that the project has
failed spectacularly.
2. Everyone independently writes down every
reason they can think of for the failure (especially
things they ordinarily wouldn’t mention).
3. Leader asks each team member to read one
reason from his or her list. Everyone states a
different reason until all are listed.
4. After the session, the project manager reviews
the list, looking for ways to strengthen the plan.
Klein (2007)
Activity

Making Social Innovation Work Day 3

  • 1.
    Making social innovationwork Day 3 Edward Gardiner
  • 2.
    Day 3 09.30-10.00 Recapon Day 2 10.00-10.30 The three strands of value 10.30-11.30 Social business models 11.30-12.00 Break 12.00-13.00 Experimenting with social impact 13.00-14.00 Lunch 14.00-14.30 Feedback and discussion 14.30-15.30 Activity: Business model/Measuring 15.30-16.30 Activity: Planning for the future
  • 3.
  • 4.
    Entrepreneurship The pursuit ofopportunity beyond the tangible resources that you currently control. Emphasis on opportunity recognition, committing to the opportunity, gaining control over resources. Often focuses on breakthroughs and new needs. Social entrepreneurship Innovative, social value creating activity that can occur within or across the non-profit, business, or government sectors. The drive is to create social value, rather than personal or shareholder wealth. Often focuses on serving basic, long-standing needs more effectively. Stevenson (1983), Austin et al (2006)
  • 5.
  • 6.
  • 7.
    Theoretical differences • Marketfailure: emerge when there is a social- market failure, i.e. commercial market forces do not meet social need. • Mission: creating social value for the the public good vs. creating profitable operations resulting in private gain. • Resource mobilisation: ability to access financial resources and capital markets. • Performance measurement: measuring social impact is harder than tangible and objective financial measures. Austin et al (2006)
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
    Types of businessmodel • Fee for service: Sell a good or service directly to beneficiaries. • Connector: Help facilitate trade relationships. • Intermediary: Help beneficiaries distribute to an expanded market. • Cross-compensation: Sell a good to one group in order to subsidise another. • Hybrid: Sell a good to one market in order to support beneficiaries in another.
  • 20.
    M-KOPA Solar (feefor service) • Founded by Nick Hughes, Chad Larson and Jesse More in 2011. • Customers pay a deposit of $35 and then pay $0.5 per day for a period of one year to own the solar system using M-PESA. • $200 device includes two LED bulbs, a flashlight, rechargeable battery and adaptors for phone charging, and is charged by a solar panel on the roof. • Meant for off-grid customers using kerosene lamps. • Connected over 500,000 homes to affordable solar. • Current customers projected to make USD$375m savings over the next four years, and enjoy 62.5 million hours of kerosene free lighting per month. M-Kopa / Acumen
  • 21.
  • 22.
    Kiva.org (connector) • Foundedby Matt Flannery and Jessica Jackley in 2005 following a trip to Uganda. • They met many entrepreneurs who wanted to share their stories with friends and family. • Mission to connect people and provide entrepreneurs with dignity and respect. • Local microfinance institutions worked with entrepreneurs to get their business on Kiva’s website. Investors anywhere could choose entrepreneurs to loan money to. • Over $1 billion lent, lenders and entrepreneurs in 150 countries worldwide. Aaker & Chang (2010)
  • 23.
    Social Innovation Lab Marketingplatform development Entrepreneurs in developing countries # of entrepreneurs $ lend $ repaid # dependents Create jobs, reduce poverty Borrower Family of borrower Well-off individuals who are socially conscious Service Kiva.org Repayment risk Managing risk Managing platform Screening MFIs Optional donations Collecting loan capital Proving loans Kiva.org platform Local MFIs Payapl Donor network Brand Financial ability to start business Promote market based solutions to povertyLocal microfinance institutions
  • 25.
    Ten Thousand Villages (intermediary) •Non-profit fair trade organisation founded in 1946 (one of the largest and oldest). • Markets crafts made by disadvantaged artisans from over 30 countries. • Over USD$140m has been earned by makers who would otherwise be unemployed. • Income earned by the producers helps to pay for food, education, healthcare and housing. • Price mutually agreed with upfront payment, builds long term partnerships, ensures good working conditions, emphasise reuse/recycle. Ten Thousand Villages
  • 27.
    Ziqitza (cross-compensation) • Mumbaiand other cities in India lack reliable ambulances or emergency services. • Ziqitza Health Care Ltd founded in 2005 by group of young professionals, providing 24/7 call centres and ambulance tracking. • In Mumbai, operates a sliding price scale that depends on a customer’s ability to pay and circumstance (accident victims travel for free). • ZHL operaties 980 ambulances across five states, has served over 2 million people, has been awarded $80m in government contracts. • Was the first ambulance services to arrive at the Taj and Trident hotels during the Mumbai attacks in 2008. Ziqitza / Acumen
  • 29.
    Hot Bread Kitchen(hybrid) • Founded by Jessamyn Rodriguez. Bakers in training programme launched in 2007, and HBK Incubates launched in 2011. • Employees, mostly low-income immigrant women, bake bread inspired by their countries of origin. • They learn job skills that can lead them to management positions in the food industry. • Bakery is widely acclaimed for its innovative selection of international breads. • 65% of operating budget is funded by bread sales and kitchen rental. Philanthropic support makes it possible to provide industry-specific training. Hot Bread Kitchen
  • 30.
    Other models • Addons (e.g. airlines) • Advertising (e.g. online publishing) • Auctions (e.g. eBay) • Bait and hook (e.g. razors, coffee makers) • Franchise (e.g. TED talks, restaurants) • Freemium (e.g. mobile gaming) • Long tail (e.g. micro-breweries) • Low cost (e.g. airlines, credit cards) • Subscription (e.g. health clubs)
  • 32.
  • 33.
    Experimenting with social impact Howexperimentation can be used to understand what works (and improve what doesn’t)
  • 34.
    Three types ofquestion • Descriptive: describe what is going on or what exists. • Relational: look at the relationships between two or more variables. • Causal: determine whether one variable causes or affects one or more outcomes.
  • 35.
    “In the 20thcentury, randomized, controlled trials have revolutionized medicine by allowing us to distinguish between drugs that work and drugs that don't work. And you can do the same randomized, controlled trial for social policy. You can put social innovation to the same rigorous, scientific tests that we use for drugs.” Esther Duflo Abdul Latif Jameel Poverty Action Lab TED (2010)
  • 36.
    First principle ofscience To demonstrate whether an activity causes an outcome to occur, you need to provide evidence for two statements: 1. If the activity is given, then the outcome occurs. 2. If the activity is not given, then the outcome does not occur.
  • 37.
    Treatment group Control group Drug Placebo time 1.RANDOMISE 2. IMPLEMENT 3. MEASURE BETWEEN GROUPS WITHIN GROUP Sample
  • 38.
    Strength of conclusion Intervention Whatyou do Observation What you see Causes Alternative causes Alternative causes LAB ENVIRONMENT
  • 39.
    Is random assignment used? Randomisedor true experiment Is there a control group or multiple measures? Quasi-experiment or matched trial Non-experiment Yes No NoYes
  • 40.
    Who are thesample? • Citizens, staff, customers, volunteers, managers, people. • Is it a natural experiment or will you need to provide an incentive to participate? • Do you have enough people or will there be a cost of recruitment? • Are they representative of the population?
  • 41.
    How will yourandomise? • Assign individuals, clusters, locations or areas. • Is it possible to randomise individuals or cluster to different groups? • Is it possible to personalise what individuals or clusters receive? • Will different groups know what the other groups are getting?
  • 42.
    What are youtesting? • New intervention versus nothing, current practice or different variations. • Is there existing evidence to support what you’re testing? • Is the intervention an incremental change or a radical new approach? • Are you testing proof of concept or the final version before scaling up?
  • 43.
    What are youmeasuring? • Attitudes, experiences, decisions or actions. • Psychological scales, surveys and observation. • Are they subjective or objective measures? • Are they direct or proxy measures? • When will the measures be taken?
  • 45.
    61m person experimentin social influence Context Voter turnout is significantly correlated among friends, family members and co-workers, although anecdotal accounts suggest that online mobilization has made a big difference in recent elections, a meta-analysis of email experiments suggests that online appeals to vote are ineffective. Method • 18 yrs who accessed on 2 Nov 2010. • A: Social (n=61m, message, poll information, counter and button, friends). • B: Informational (n=600k, as above, no faces of friends). • C: Control (n=600k, no message). Bond et al (2012)
  • 46.
    Incentives for immunisation Context Everyyear an estimated 2 to 3 million people die from vaccine-preventable diseases. A lack of understanding of the benefits or even suspicion are possible reasons rates remain low in India. Cash incentives are effective but too expensive. Can small incentives have a big impact on immunisation rates? Method • Clustered randomised controlled trial with 174 villages in Udaipur, India. • Intervention A (30): mobile immunization team conducted monthly camps. • Intervention B (30): same but parents also offered 1kg of lentils each time. • Control group (74): nothing. Banerjee et al. (2010)
  • 47.
    Reducing aggression inA&E Context Violence and aggression towards frontline staff, including 59,000 physical assaults, is estimated to cost the NHS at least £69 million a year in staff absence, loss of productivity and additional security. How can better visual communication reduce frustration, anxiety and abuse towards staff? Method • Pre/post, matched controlled trial across four NHS Trusts in the UK. • Pilot sites (2): signage and people project design solutions implemented. • Comparative control sites (2): no design solutions implemented. Design Council / Frontier Economics (2014)
  • 48.
    The questions shapethe answers • How short was Napoleon? • Should concerned parents use infant car seats? • Where do you enjoy drinking beer? • Do you always eat breakfast? • How would you describe Napoleon’s height? • Do you think special car seats should be required for infant passengers? • What do you normally like to drink? • How many days a week do you eat breakfast? Survey Monkey
  • 49.
    Correlation ≠ causation Intervention Whatyou do Observation What you see Correlation State of the world Not what you do
  • 50.
  • 51.
  • 52.
    Regression to themean A statistical phenomenon that if a variable is extreme on its first measurement, it will tend to be closer to the average on its second measurement, and vice versa. Research methods knowledge base
  • 53.
  • 54.
  • 55.
  • 56.
  • 57.
    Debunking some myths 1.We don’t necessarily know ‘what works’ 2. RCTs don’t have to cost a lot of money 3. There are ethical advantages to using RCTs 4. RCTs do not have to be complicated or difficult to run Test, Learn Adapt (Cabinet Office)
  • 58.
  • 59.
  • 60.
  • 61.
  • 62.
  • 63.
  • 64.
  • 65.
    Three types ofquestion • Descriptive: describe what is going on or what exists. • Relational: look at the relationships between two or more variables. • Causal: determine whether one variable causes or affects one or more outcomes.
  • 67.
  • 68.
  • 69.
    Project premortem 1. Leaderinforms the room that the project has failed spectacularly. 2. Everyone independently writes down every reason they can think of for the failure (especially things they ordinarily wouldn’t mention). 3. Leader asks each team member to read one reason from his or her list. Everyone states a different reason until all are listed. 4. After the session, the project manager reviews the list, looking for ways to strengthen the plan. Klein (2007)
  • 70.

Editor's Notes

  • #5 Austin, J., Stevenson, H., & Wei‐Skillern, J. (2006). Social and commercial entrepreneurship: same, different, or both?. Entrepreneurship theory and practice, 30(1), 1-22. Stevenson, H. H. (1983). A perspective on entrepreneurship.
  • #6 https://www.ideou.com/pages/design-thinking
  • #7 http://shiftdesign.org.uk/the-3-strands-of-value-vital-to-social-tech-ventures/
  • #8 Austin, J., Stevenson, H., & Wei‐Skillern, J. (2006). Social and commercial entrepreneurship: same, different, or both?. Entrepreneurship theory and practice, 30(1), 1-22.
  • #9 Austin, J., Stevenson, H., & Wei‐Skillern, J. (2006). Social and commercial entrepreneurship: same, different, or both?. Entrepreneurship theory and practice, 30(1), 1-22.
  • #11 http://www.designcouncil.org.uk/news-opinion/design-process-what-double-diamond
  • #12 http://www.nesta.org.uk/publications/innovation-flowchart
  • #13 https://strategyzer.com/canvas/business-model-canvas
  • #14 http://www.socialbusinessmodelcanvas.com
  • #17 https://www.gov.uk/guidance/social-impact-bonds
  • #18 https://unltd.org.uk/portfolio/3-7-determining-the-right-legal-structure-for-your-social-enterprise/
  • #21 http://acumen.org/investment/m-kopa/
  • #23 https://www.gsb.stanford.edu/faculty-research/case-studies/kivaorg-power-story
  • #24 http://www.socialbusinessmodelcanvas.com/wp-content/uploads/Social-Business-Model-Canvas-Example.pdf
  • #26 http://www.tenthousandvillages.com/about-us/
  • #28 http://acumen.org/investment/zhl/
  • #30 https://hotbreadkitchen.org
  • #32 http://www.plusacumen.org/courses/business-models-social-impact
  • #36 https://www.ted.com/talks/esther_duflo_social_experiments_to_fight_poverty
  • #46 Bond, R. M., Fariss, C. J., Jones, J. J., Kramer, A. D., Marlow, C., Settle, J. E., & Fowler, J. H. (2012). A 61-million-person experiment in social influence and political mobilization. Nature, 489(7415), 295-298.
  • #47 Banerjee, A. V., Duflo, E., Glennerster, R., & Kothari, D. (2010). Improving immunisation coverage in rural India: clustered randomised controlled evaluation of immunisation campaigns with and without incentives. Bmj, 340.
  • #48 http://www.designcouncil.org.uk/projects/reducing-violence-and-aggression-ae
  • #49 https://www.surveymonkey.com/blog/2015/02/11/5-common-survey-mistakes-ruin-your-data/
  • #51 http://www.tylervigen.com/spurious-correlations
  • #53 http://www.socialresearchmethods.net/kb/regrmean.php
  • #54 https://xkcd.com/605/
  • #55 http://www.bbc.co.uk/news/technology-28051930
  • #56 https://theblog.okcupid.com/we-experiment-on-human-beings-5dd9fe280cd5
  • #57 https://medium.com/netflix-techblog/its-all-a-bout-testing-the-netflix-experimentation-platform-4e1ca458c15
  • #58 http://www.behaviouralinsights.co.uk/publications/test-learn-adapt-developing-public-policy-with-randomised-controlled-trials/
  • #70 Klein, G. (2007). Performing a project premortem. Harvard Business Review, 85(9), 18-19.