7 Steps for Data-Driven Decision Making


Published on

Measuring Success introduces nonprofit professionals to proven techniques on how to move from anecdotal to data-driven decision making and steer your organization to success. Gain insights on how to focus your limited organizational time and energies on the issues that are supported by data instead of anecdotes. Learn techniques for using data to track and measure progress over time, report impact to stakeholders, and manage toward success.

Published in: Business

7 Steps for Data-Driven Decision Making

  1. 1. 7 Steps for Data-Driven Decision-Making Sacha Litman, Managing Director, Measuring Success José Fernández, Director of GuideStar Exchange Questions? info@measuring-success.com #7steps
  2. 2. Agenda 5 minutes Opening poll and introduction 5 minutes Benefits to using data 15 minutes Illustrative Case study 5 minutes Q&A and concurrent poll 25 minutes 7 Steps to data-driven decisions 5 minutes Take-away principles & talking points 10 minutes Q&A and concurrent closing poll 1 minute Conclusion Questions? info@measuring-success.com #7steps
  3. 3. A. Benefits Questions? info@measuring-success.com #7steps
  4. 4. Benefits of data-driven decision making to Customers Success matters because your mission is important! If you don’t measure and use data, you are not achieving the full potential of your mission. Questions? info@measuring-success.com #7steps
  5. 5. Benefits to Management & Staff Focus limited time and energies on key activities that are real issues and have impact on our mission Set goals for staff and hold them accountable “Cannot manage what you do not measure” – Peter Drucker “What gets measured gets done” “High performing organizations use data analytics roughly three times as extensively as the lower performers in their field” – Competing on Analytics, HBS Press Questions? info@measuring-success.com #7steps
  6. 6. Benefits to Boards and Donors Boards Donors Focus their energies away Ability to show “return on from emotions & anecdotes investment” Decisions not made by loudest Attract large gifts and grants or wealthiest voice by confidence engendered by data “Analytical projects aimed at improved outcomes had a Data analytics was one of the median ROI of 55% for CRM top three areas of increased and 139% for financial investment by companies management” during the recession. – Competing on Analytics, HBS Press Questions? info@measuring-success.com #7steps
  7. 7. Popular Press Examples Professional sports: Oakland A’s (Moneyball) Maximize team wins while minimizing payroll By hiring players as indicated by data, not scouts’ gut Health Care: Intermountain Health Care (NY Times Magazine Nov 9, 2009) Improve patient outcomes (survival, longevity, quality of life) while minimizing cost By selecting treatment/procedure supported by evidence not Doctor’s gut instinct or unneccessary procedures Technology: Google Increase advertising revenues ($21 billion) by targeting ads to right customers By mining data on all users of free Google consumer software Questions? info@measuring-success.com #7steps
  8. 8. B. Quick Case Study Questions? info@measuring-success.com #7steps
  9. 9. 1. Chapter-based community centers Experiencing some with member drops, financial insolvency. Challenge to business model & brand consistency. Association HQ Local Affiliates or Regions (N=150) Individual Members / Users (N=1000 to 10,000 per affiliate) Questions? info@measuring-success.com #7steps
  10. 10. 2. Association sought “early warning system” & understanding of what led to successful outcomes Questions? info@measuring-success.com #7steps
  11. 11. 3. Build measurement and dashboard system as pilot experiment Engaged 6 willing chapters Built: Customer survey Employee survey Financial analysis tool Member participation tool Questions? info@measuring-success.com #7steps
  12. 12. 4. Focused on rankings and metrics that were statistically valid Regression analysis ties activities to outcomes Comparison Against peer chapters in other geographies Against local competition from other organizations Against own prior measures (longitudinal) Within demographic segments of member base (see following slides) Questions? info@measuring-success.com #7steps
  13. 13. 4a. Identify activities associated with outcomes (multiple regression analysis) Budget Management versus Value for Membership Dollar Correlation Value for Membership Dollar Average Score 1-5 Scale Perceived Budget Management & Transparency (1= strongly disagree, 5 = strongly agree) Questions? info@measuring-success.com #7steps
  14. 14. 5. Like doctor, ran diagnostics on each chapter every 2 years Rolled out pilot to all chapters; readiness factors to participate Mutually agreed on a plan for improvement Some consulting support Select Measures from Customer Rank Score Priority Goals & Strategy Survey (of 15) % Str Agree Membership Value for the Dollar 7 25% Medium Focus on quality, budget perceptions Professionals welcoming 2 35% Low Budget Perceived as well 14 16% High Double scores in managed 2 years. Questions? info@measuring-success.com #7steps
  15. 15. Data Creates Organizational Alignment & Intentionality 2. Consumer Latent Needs (Purpose-Driven) 1. Consumer 3. Institutional Expressed Strategic Vision Wants 6. Contribution to 4. Financial People Pipeline Sustainability (Intake, Attrition) 5. Resource (Surplus/Deficit) Intensiveness (Mgmt Time, Sqft, Foot Traffic) Questions? info@measuring-success.com #7steps
  16. 16. 6. Turnaround: focus lead to improvement from low to average in 2 years; now aiming for top Personal Conversations with Customers Monthly or more often 27% Questions? info@measuring-success.com #7steps
  17. 17. Data encourages prioritization: 80% of board & mgmt team hypotheses about what we anecdotally “believe” is a problem is not supported by data! Questions? info@measuring-success.com #7steps
  18. 18. 7. Chapter ruled by anecdotes: 80% of assumptions were not supported by data Assumed they were strongest with eldest & wealthiest portions of participant base Likelihood to recommend to a friend % Strongly Agree 25-34 35-44 45-54 55-64 65-74 Over 75 50% 43% 50% 48% 45% 40% Less than $100,000- $200,000- $300,000- $400,000- $500,000 and $100,000 $199,999 $299,999 $399,999 $499,999 over 40% 42% 67% 63% 42% 46% Shock, challenged the data, acceptance Management team focused energies on improvement with eldest and wealthiest. 2 years later, scores significantly higher there. Questions? info@measuring-success.com #7steps
  19. 19. 8. Rising Tide Lifts All Boats: top, average, and low performers all improved Surplus Margin for Early Childhood Program Surplus as % of Expenses, after allocating all overhead X X Questions? info@measuring-success.com #7steps
  20. 20. 9. New policy: value, not price Highest Willingness Assumed that key driver of To Pay (center) Demand Function member retention was price C. Perceived Analysis shows not price, but Quality of Services Least Ability to Affect perceived quality and value- for-dollar Result: association stopped encouraging price B. Commitment A. Financial subsidization, encouraged To Issue Ability perceived quality improvement Perceived value and value for dollar are tracked carefully and promoted system wide Questions? info@measuring-success.com #7steps
  21. 21. 10. Outcome improvement: participation Chapters that embraced this approach outperformed others significantly in participant enrollment and financial sustainability, despite the recession Participant Retention Rate: 91% to 96% New Participant Rate: 5% to 10% Net Participation: 96% to 106% Financial sustainability increased: coverage ratio (% expenses from fees & membership) grew from 74% to 80% Questions? info@measuring-success.com #7steps
  22. 22. C. 7 Steps to Data-Driven Decision Making Questions? info@measuring-success.com #7steps
  23. 23. Data driven decision making is parallel to what we learned in middle school science class Identify issue State hypothesis: “I believe…” Perceived mechanism/ cause Design experiment Examine data Confirm or reject hypothesis Questions? info@measuring-success.com #7steps
  24. 24. 7 Stages of Data-Driven Decision Making 1. Framing the 2. Problem Hypothesis Develop- ment 3. Data Collection 4. Data Analysis 5. Inter- pretation 6. Decision Making 7. Commun- ication Questions? info@measuring-success.com #7steps
  25. 25. 2. Hypothesis Development Brainstorm hypotheses. Many will turn out to be incorrect, but that’s ok so long as you can articulate a plausible mechanism. Remember, our consistent experience with organizations is that over 80% of their initial hypotheses as to what is driving a problem are not supported by the data! During brainstorm define: Hypothesis (may need to revise several times) Mechanism (cause & effect) How could measure it Questions? info@measuring-success.com #7steps
  26. 26. 2. Hypothesis Development: Gann Academy Initial Hypothesis: unevenness in students’ experience getting their individual learning needs met by the school Plausible Mechanism: additional academic support (learning center). Major differentiator in students’ learning experience (for those in the same classrooms) Final Hypothesis: students who received additional academic support from learning center felt their individual learning needs were better met by the school. Questions? info@measuring-success.com #7steps
  27. 27. 2. Hypothesis Development: Gann Academy How measure? Parent survey Question on individual learning needs met by school Cut data by whether student used learning center t Scale 10a. Teachers Understand Needs & Employ Effective Learning Learning Techniques Center Yes Center No Strongly Disagree 3% Disagree 9% Neither Agree Nor Disagree 16% Count 84 240 Agree 44% Average 3.5 4.0 Strongly Agree 28% Average 3.8 Questions? info@measuring-success.com #7steps
  28. 28. 3. Data Collection Do… Don’t… Prioritize hypotheses Just go after what’s Work backwards: figure out easiest how will use results of data Overtax your resources – first before collect it data collection can be very Consider several methods of time intensive data collection Go after more data than Save time and money by you can act upon – considering simple, pre- Garbage In Garbage Out built tools allowing some (GIGO) customization (instead of building own or buying highest end product you won’t use) Questions? info@measuring-success.com #7steps
  29. 29. 3. Data Collection Tool Pro Con Survey Data collection phase is To get quality results and high automated and not resource response rates, need to put lots intensive of time into asking the right questions, making it look Obtain perceptions of quality professional, and assuring confidentiality Financial analysis Data your organization already Requires reorganizing and “fine has slicing” the financials, so need a strong CFO Tracking system Integrated into daily efforts Takes a lot of discipline and (CRM) time to build data Self-reported from Already have the database Time consuming if database existing databases does not contain the right information may end up with lots of estimates Questions? info@measuring-success.com #7steps
  30. 30. 3. Data Collection : Gann Academy Good: financial modeling Bad: admission tracking Clarified board’s needs up Purchased high end front: 5 year projections, database system key levers of sustainability Not trained properly Used already-built but Orphan Excel spreadsheets simple model (in Excel) Never clarified what instead of building own management needed for Worked hard to gather decision making, so consistent financial data “GIGO” (counting $ in development and CFO’s office) “What if” scenario building enabled board buy-in Questions? info@measuring-success.com #7steps
  31. 31. 3. Data Collection : Gann Academy Work backwards. Decide how you will use the data first, as this will dictate whether and how to ask for the data. Questions? info@measuring-success.com #7steps
  32. 32. 5. Interpretation: draw correct conclusions from data. Danger in “seeing what want to believe” You don’t need to be a statistician to correctly interpret data However, do need to understand basic concepts Mean Standard deviation Counts (sample size) Benchmarking Frequently, someone will “find” something in the data that is not valid but it supports their personal views Questions? info@measuring-success.com #7steps
  33. 33. 5. (Mis)Interpretation: Gann Academy Relative Strength of Gann Academy’s Board member who was Attention to Individual Needs vs. convinced the most severe Most Attractive Alternative School (higher score is better) threat on enrollment was from charter and public schools, not Gann vs. Gann vs. Gann vs. area private schools Charter Hypothesis: public schools are or General Private better at supporting individual magnet public Schools public schools student needs and thus pose the schools greatest risk of attrition for Gann Academy families. Count 50 4 51 Data source: teacher attention Average 3.8 3.4 3.6 to individual needs cut by most attractive alternative option Questions? info@measuring-success.com #7steps
  34. 34. 5. Interpretation: Benchmarking Relative to What? Contextualization is critical: Peer group (similar organizations nationally/internationally) Local competition* Own longitudinal history Across demographic groups* *Depicted on a prior slide Questions? info@measuring-success.com #7steps
  35. 35. 5. Interpretation: Benchmarking Relative to Peer Group of other similar high schools around the country Note if had used primary schools (K-8) as peer group Questions? info@measuring-success.com #7steps
  36. 36. 7. Communication Sharing with board, participants, donors, employees Fear: sharing data will undermine management’s authority Reality: sharing data engenders trust Diffuse concerns through transparency Even if current scores are not as high as would like, donors value a roadmap toward and metrics for success Questions? info@measuring-success.com #7steps
  37. 37. 7 Stages of Data-Driven Decision Making 1. Framing the 2. Problem Hypothesis Develop- ment 3. Data Collection 4. Data Analysis 5. Inter- pretation 6. Decision Making 7. Commun- ication Questions? info@measuring-success.com #7steps
  38. 38. Common Non-Profit Applications of Data-driven Decision Making Outcome measurement Identifying activities that give greatest “bang for buck” in limited resource environment Financial sustainability Setting program fees or membership dues levels Identifying most susceptible demographic groups Donor prioritization using predictive models Board reporting Program evaluation Questions? info@measuring-success.com #7steps
  39. 39. If your organization makes data-driven decisions you will be able to… Focus limited energies and resources on only the 20% of hypotheses that are supported by the data Motivate staff and volunteers because they know they are investing in what works Align efforts of staff, volunteers, board, donors, and customers Monitor progress toward goals Have a dashboard for your performance Make strategic board decisions more efficiently and gain more buy-in Distinguish yourself against and outperform your competition by developing a core competence few non-profits have Observe improvements in key areas such as: financial sustainability, donations, participation, mission impact Questions? info@measuring-success.com #7steps
  40. 40. 7 Steps for Data-Driven Decision-Making Sacha Litman, Managing Director, Measuring Success José Fernández, Director of GuideStar Exchange Questions? info@measuring-success.com #7steps