Analytics that deliver Value


Published on

Framework for creating Analytics that delivers cost effective Value. From what to output, to how to scale and motivate a team, passing through data acquisition. Analytics has become a critical asset for the most competitive organizations; practitioners must ensure their ability to create and communicate insight, especially the most senior decision-makers is effective and efficient.

Published in: Business, Technology, Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Headquartered in Boston with offices across the US, Europe, and Brazil, we are a cloud-based platform enabling brands to use Big Data for smarter marketingExclusive license to real-time decision system created by aerospace PhD founders at MIT (first used for NASA Mars Mission project). Photo is an Atlas V Launch Vehicle Real-timing bidding on digital media inventory is how DataXu gets the Big Data to power more intelligent marketing for brands…
  • I moved this slide here to illustrate the overarching complexities of leading analytics
  • Analytics that deliver Value

    1. 1. Analytics for Decision Making smart people don’t make blind bets © 2013 Sandro Catanzaro 1
    2. 2. About this deck This deck was prepared for a Meetup, presented in Boston in August 2013 Some friends suggested it could be useful if shared more widely About me • I’m a mech eng, from back in the day when you would draw blueprints by hand • MIT grad - best place to learn about systems – and rockets • Worked as consultant for a while (hence the ‘beautiful’ slides) • Lead the Analytics and Innovation practices at DataXu – of which I’m co-founder Funny picture about me © 2013 Sandro Catanzaro 2
    3. 3. Founded in 2009; Headquartered in Boston, MA; 300+ employees, 14 offices globally, PetaByte scale dataset Real-time decision system based on MIT research by Bill Simmons and Sandro Catanzaro (first used for NASA Mars Mission project) Easy-to-use software platform enabling brands to use Big Marketing Data 5th fastest growth US company Rated #1 algorithmic optimization technology by Forrester Research Powering blue-chip brands: Ford, Lexus, General Mills, Wells Fargo, Discover DataXu is all about applying data science to the art of marketing © 2011-2013 DataXu, Inc. Privileged & Confidential
    4. 4. 4 4© 2011-2013 DataXu, Inc. Privileged & Confidential
    6. 6. PLEASE MAKE MAGIC HAPPEN © 2013 Sandro Catanzaro 6
    7. 7. What about Analytics • Analytics popularity driven by its results • MIT Sloan and IBM survey shows business believes on its value* • Science reliance in massive datasets – DNA data mining Search interest trend for “Analytics” 20132011200920072005 Google Search Trends 0% 50% 100% 2010 2011 MIT Sloan – IBM survey Decision-makers belief analytics is a source of competitive advantage © 2013 Sandro Catanzaro 7 *Source: Create Business Value with Analytics – MIT Sloan Management Review – Fall 2011”
    8. 8. Just a trend? • Declared the new black – Symptom of shark jumping? • Analytics will stay relevant as long as it continues providing the right answer – Or the right answer more often than not © 2013 Sandro Catanzaro 8
    9. 9. So, there is value on Analytics • Yes, but.. • What is it? • And how to make it happen? © 2013 Sandro Catanzaro 9
    10. 10. Analytics is a bridge Raw data Decision-making Cleaning Aggregates Selection of what matters Error quantification Model limits Insight As any other bridge, a feat not a science, but of engineering • Goes to a specific point to another specific point • Good enough foundations (data and question clarity) • Can be used by most people who want to cross it • Has explicit durability assumptions Engineering = art of tradeoffs with just enough information Technology Action oriented question © 2013 Sandro Catanzaro 10
    11. 11. Analytics is a system Cleaning Aggregates Selection of what matters Error quantification Model limits Insight Technology Presentation Data Data stakeholders Insight recipients QA P&L owner Brittle interface Easier to manage interface Input Process Output © 2013 Sandro Catanzaro 11
    12. 12. Types of Analytics 1. Research – Unclear need, unclear answer, high risk 2. Development – Clear answer and method, but some risks on implementation 3. Operations – Good experience with execution. Can be automated – QA for Automation becomes a type 1 problem, again #1 = most fun / most challenging = focus of this slide-set © 2013 Sandro Catanzaro 12
    13. 13. Analytics Leaders rely on four core competencies … • Valuable insights take into consideration business, domain and analytics process… • …with just-good-enough technology • The Analytics Manager main job is to staff a team that can master them all Business & Communication Domain knowledge Analytics & reasoning Technology 13
    14. 14. … to address a diversity of issues Dirty data Too much data Far away data Scaling up the team New toolset dynamics Thinking takes time R&D type of risk Unclear need Answer not simple enough Output contradicts business Unwillingness to pay Politics NIH – fear of change Little intellectual curiosity Market noise – new tools Threat of disruption from output to input Too little data Legal implications Technical stakeholders Business – Domain stakeholders Input Process Output Proving ROI Reputational risk Trust and proprietary data Most challenging issues © 2013 Sandro Catanzaro 14
    15. 15. Reporting vs Analytics Just a “report writer software” will do it • Analytics = curated reports – Curation: identifying what matters, editing out what doesn’t – Joining the dots • Analytics is a Bridge to the right insight – Not a pile of needles to replace a pile of hay – Decisions are made with just THE ONE needle – Careful you all detail oriented nation! © 2013 Sandro Catanzaro 15 Misconceptions
    16. 16. Do some analytics for me • Many problems are not clear – Domain experts and business managers, even you, don’t know boundaries of what can be answered – “catch 22 situation” – Even worse, the problem existence is unclear • A solution is identified before the problem is clear (this is really bad) – The myth of big data – and the heavy lifting in data collection • Analytics are perceived as a “nice to have” by less technically oriented stakeholders – Do it on your free time You are smart, you can come up with something interesting, can’t you? Unclear need Unwillingness to pay © 2013 Sandro Catanzaro 16 Challenges
    17. 17. Make sure the answer is X • Clear questions and enough data may still provide the “wrong” answer – May not make business sense – Or not be of the liking of business • Or, may require complex answers difficult to digest by intended audience • Or, politics may be a stronger driver for the decision process Answer not simple enough Output contradicts business Politics © 2013 Sandro Catanzaro 17 Challenges
    18. 18. Look at the big picture AND focus! • Good analysts who are willing to see the big picture are scarce – Hiring analytics management is harder: Peters Principle • Moving too fast or too slow in tool selection – In the middle between bleeding alpha and cobol. Think “second best” • Analysis “gold polishing” – Some of us may be perfectionists • … and finally, there intrinsic risk that the analysis may not work Scaling up the team New toolset dynamics Thinking takes time R&D type of risk © 2013 Sandro Catanzaro 18 Challenges
    19. 19. The right data, all in one place • Data is often in the wrong format or has rows that need to be ignored – Cleaning is tedious, often manual • More often than not, there is superfluous data that we keep “just in case” – Bogs down processing, blows out costs Dirty data Too much data Far away data • Data is rarely centralized – May require collaboration: “What is in it for me” © 2013 Sandro Catanzaro 19 Challenges
    20. 20. FRAMEWORK (TO MAKE MAGIC WORK) © 2013 Sandro Catanzaro 20
    21. 21. Where to start Clear and present need “you are lucky” Need is not crisp, but challenge is known “we just need Analytics” Lack of awareness about challenge “Beware” Crisp and clear statement of the “question to answer” Can you tell us how sales correlates with factors A, B, C, D and E; using a quarter by quarter dataset, and data from 2010 to date Please do some attribution modeling good bad © 2013 Sandro Catanzaro 21
    22. 22. The pull model Starts from the end Business problem Audience Is this truly important to you? Are you willing to pay for it? Analysis output (analysis tools) Processing (aggregation, p rocessing technology) Dataset (storage technology) Data collection (data sources, pick, p arse cleanup) © 2013 Sandro Catanzaro 22
    23. 23. The pull model Starts from the end… © 2013 Sandro Catanzaro 23 …You would cross the “bridge” 3 times • First time from where you are, to the business stakeholder and problem • Second time from the business problem, all the way back to the data, so that every piece “pulls” the next previous one • With business problem in hand, you will know the minimum analysis output that provides the answer • With the analysis in hand, you will know analysis tools needed to prepare, and also what is the processing and aggregations needed to support it • With the aggregations in hand, you will know the minimum dataset required • .. All the way to data collection • Third time from the data, you will come back as you execute the analysis, step by step to finally arrive to the desired answer
    24. 24. Continuous iteration Crisp question statement Stakeholders Stakeholders Stakeholders Stakeholders “the contract” • Share interim results: continue selling on value, continue selling on model belief • May modify the “crisp question” – ensure you also reframe time, resources, cost • Whether you’ll share the methodology depends on the audience • Creates dialog and sense of shared ownership in the answer Mockup answer (Answer First) Back of the envelope answer Complete answer Incorporate feedback Share output (maybe methodology) © 2013 Sandro Catanzaro 24 Nothing fancy here Just risk mitigation / project management 101…
    25. 25. Be flexible • At each iteration, the question may move… – Nothing is worse than answering the wrong question – What is valuable may evolve • … yet ensure you “get your victory lap” if the question moves – Thinking evolved thanks to analysis – Re-evaluate timing, resources • Practical signs you’re on the right track: – Experienced stakeholders confirm your findings by ‘gut’ – They ask a LOT of clarifying questions & recommend potential next steps to your analysis/message © 2013 Sandro Catanzaro 25
    26. 26. Best models: 70% belief - 30% math • Simplify • Share with all shareholders – get them to believe in the answer and model • Simplify • Acquire buy in – “prewire meetings” – Must have on board the P&L owner – Also recruit influencers, other stakeholders • Simplify again – Review and clean up model – Remove unneeded items, keep just enough to prove your point – Maybe add a single cherry (non-requested-by-so-cool-to-know insight) Be careful on adding accuracy at the cost of complexity © 2013 Sandro Catanzaro 26
    27. 27. Risk mitigation • Understand your project risks and tackle them in order – The scariest first – even if you don’t solve it, you’ll raise awareness, understand it better, adjust timeline if needed – Leave the easiest for last – even if out of sequence • Ensure you can create full end-to-end passes with increase accuracy each time © 2013 Sandro Catanzaro 27 Nothing fancy here Just risk mitigation / project management 101…
    28. 28. Scaling the team • Best bet is for introducing geeks, into big picture thinking – Smarts beat expertise – “Spin Doctors” are useful, to a point • Aim for “T shaped” people – Broad big picture understanding – Deep expertise in one area • Provide context – Make sure team understands where does the analysis fit • Be willing to let go – Understand costs of potential mistakes – Responsibility is never delegated © 2013 Sandro Catanzaro 28
    29. 29. Accessing and processing data • Data accumulates “close” to the data user (formats, quirks) – Data that is not used is rarely correct, updated on time • Help from data users is critical, yet them may feel threatened – Create quid-pro-quo partnerships. Giving credit can be enough • Avoid changing technologies mid-way a project – If new tech advantages are leaps and bounds above, parallel run both until the new replicates results © 2013 Sandro Catanzaro 29
    30. 30. The Truth • The value of Analytics team is guiding decision-making using fact-derived insight • Not easy or in some cases impossible – Not enough data or tools to arrive to conclusion – Business pressure to change the answer – The analysis is not ready in time (pesky bugs?) • Never ever yield to pressure, and present an answer that does not follow from analysis, data, facts and correct mathematical foundation Analysis that can’t be trusted is worse than no analysis. Providing such, damages your personal reputation
    31. 31. SOME ANECDOTES © 2013 Sandro Catanzaro 31
    32. 32. Make digital attribution Large banking institution, looking to show innovation: “We want to know the value of the ad sequencing So, please prepare an attribution model for us • Make it say something interesting • Actionable • … and that we haven’t heard before” © 2013 Sandro Catanzaro 32 Beware of unclear demands Use constructive questioning to build an idea of “what is it” Share with actual decision maker within customer and get feedback Incorporate feedback to make model ownership shared with customer Use a mockup made with fabricated data (for sharing before analysis is complete)
    33. 33. What can you say about their personal values? Ad agency executive for a large CPG account: “Using the data you are seeing from our ad campaign, what can you tell us about our consumers’ values?” © 2013 Sandro Catanzaro 33 Adjust expectations about what is possible, early If possible, avoid saying NO. Better to re-interpret question and provide a possible answer Other cases are “doable” but at great cost; if so, charge enough, and ensure burden is understood In some cases no alternative but saying NO: No intent on paying, no real usage for analysis
    34. 34. Create wizards: One, two, three, go! Executive to analytics manager and recruiting manager all in one: “We need you to build our new Analytics group from the ground up” © 2013 Sandro Catanzaro 34 Create a project plan and secure enough funds for hiring Don’t over-commit yourself: you cannot be hiring manager, star analyst, trainer, and main client contact. Delegate in the order of the previous list (avoid being HR, try to keep client contact) Be prepare to adjust expectations if you find yourself over-committed without wanting
    35. 35. Sandro’s Analytics principles 1. Start with the business/domain problem. Never with data 2. Don’t ever ignore humans – especially the one who pays 3. 70% belief – 30% math 4. Teach smart geeks to think big picture & let go 5. Simplify - weight complexity vs. accuracy 6. Quick and good enough wins the day 7. Model and iterate – rinse and repeat 8. Tackle risks from riskiest to least risky 9. Be flexible in scope, but take victory lap and re-budget 10. Never compromise with “the truth” © 2013 Sandro Catanzaro 35
    36. 36. APPENDIX … Because you always need to move 70% of slides to their appendix… © 2013 Sandro Catanzaro 36
    37. 37. No ROI • In theory, you should' t care • In practice, is critical to understand the value of your work, and the big picture – Ask questions, understand how your answer fits in • Good fitness means – Project stability – Organizational visibility – Less re-work Thanks to Vishal Kumar for the graphics – stolen from his blog “The project has been cancelled – sorry” © 2013 Sandro Catanzaro 37