Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Acceptance, Accessible, Actionable and Auditable


Published on

A model for the digital transformation and excellence in analytics
Presented at Google Academy for Data Festival London 2018. This is an updated version of my AAAA model presentation for WAW Copenhagen in October 2017

Published in: Data & Analytics
  • Be the first to like this

Acceptance, Accessible, Actionable and Auditable

  1. 1. Acceptance, Accessible, Actionable and Auditable A model for the digital transformation and excellence in analytics Alban Gérôme @albangerome Data Festival London 15 June 2018
  2. 2. "Le doute, morne oiseau, nous frappe de son aile... Et l'horizon s'enfuit d'une fuite éternelle !..." "Doubt, dismal bird, beat us down with its wing... - And the horizon rushes away in endless flight!..." Arthur Rimbaud, Sun and Flesh (Credo in Unam)@albangerome
  3. 3. Endemic misestimating • “More than half of senior executives experienced a backlog of at least two years on critical new analytics applications.” • 4 in 5 of Fortune 1000 companies claim that their big data investments are successful • 1 in 4 claim they have started seeing signs of a data driven culture @albangerome
  4. 4. Cheap data storage • “You cannot improve what you cannot manage” has lead to an ever increasing appetite for data, driving the cost of data storage down • Intelligence agencies notorious for collecting data with little discrimination • Historical data is good, useless historical data costs little so let’s collect data without discrimination @albangerome
  5. 5. Losing support • Projects need key supporters from the start to protect resources allocation from demands from other competing projects • Complex, multi-disciplinary projects get split into smaller parts and the projects moves only as fast the lowest prioritised part • A new project comes along with better prospects of success and steals a key supporter and then three, ten @albangerome
  6. 6. IT cannot implement analytics • Analytics tagging is mostly Javascript, IT has Javascript developers but IT is not competent to deliver an analytics implementation. You will need one or several implementation consultants • The vendor is only too happy to sell you premium consultancy services. Independent consultants are cheaper but this still causes delays and budget overruns • The implementation is never “done”, it evolves because customers expectations and technology evolve @albangerome
  7. 7. Multiple versions of the truth • 2 managers find discrepancies in analytics data and end up blaming the tool • Senior management recognises that the organisation needs a single version of the truth, i.e. a dedicated analytics team • Building a team from only internal or only external has advantages and disadvantages @albangerome
  8. 8. First reporting outage • IT developers pushed code changes live and broke analytics reporting in spite of following a rigorous testing procedure • Broken analytics tends not to produce errors, it pushes garbage data in the reports or even no data instead • IT believes that breaking analytics is no big deal but the analytics team should be the guardian of the data quality @albangerome
  9. 9. Rebuilding trust • The analytics department is expected to monitor a large and increasing number of metrics • The stakeholders beginning to ask for a lot of analytics data, a sign of regained trust • The stakeholders are no longer asking for actionable insight. The analytics team should welcome this with a sigh of relief, this is a bad sign @albangerome
  10. 10. First data analyst leaves • The analytics team is treated like a cost-centre. This means a small team without prospects of promotions or opportunities to develop team leadership skills • Each day will consist of monitoring hundreds of metrics, producing daily, weekly, monthly reports and huge data extracts that the stakeholders barely exploit • Head of analysts without analytics experience only counting the days until their next rotation, never read of a book on analytics, complete unknown in the analytics community @albangerome
  11. 11. Data cherry-picking • Pick only the data that confirms prior beliefs, discard the rest on grounds of potential implementation issues • Blaming how the data was captured is much easier than questioning one’s beliefs when the data contradicts them • Data cherry-picking makes them look data-driven but they are really data-justified, biased and they get away with it because the CXOs are either naïve or complacent and too few of them lead this by example @albangerome
  12. 12. Unwanted conservatorship • Spending ones resources for the benefits of another team at the expense of one’s own ideas and projects can only meet resistance • The incumbent teams want to retain autonomy of decision-making, claim credit and use the analytics purely as a support team • Implementing the analytics’ team ideas undermines their relevance and if given free rein, could usher something resembling holacracy @albangerome
  13. 13. Apple juice anyone? • If making apple juice is harder than expected, perhaps something is blocking at the bottom of the press • Collecting ever more data and fancy data visualisations may not be the solution • People make decisions based on emotions and then justify them with data but we are supposed to have a data-driven decision process @albangerome
  14. 14. Inspiration in strange places To create a fire, you will need three elements • Oxygen • Fuel • Heat You will not achieve self-combustion until you reach a certain heat threshold. Remove one of these three items and a fire will stop @albangerome
  15. 15. Accessible • John Gall wrote “Some complex systems actually work but building a complex one from scratch never works. You have to start over, beginning with a working simple system.” @albangerome
  16. 16. Accessible • John Gall wrote “Some complex data capture implementations actually work but building a complex one from scratch never works. You have to start over, beginning with a working simple implementation.” • With too much data captured, one faces an increased noise to signal ratio. Beware analysis paralysis • Gradually increase the number of data points you collect data for and only give access to that data to people who need that data. This allows people to become more data literate at their pace @albangerome
  17. 17. Actionable • The CXOs must lead the digital transformation by example and stop tolerating data-justified recommendations. They will strive to become a data-informed organisation rather than just a data-driven one • You need an analytics centre of excellence with a steer from the other teams. The analytics experts will focus on complex finding actionable insight and rotate between teams to gain operational experience • The incumbent teams will need “moles” or “canaries”, i.e. junior data analysts doing the reporting and metrics monitoring. Data quality is every teams’ concern @albangerome
  18. 18. Auditable • Although a lack trust can be a very handy excuse for not implementing recommendations, trust in analytics data is key • Implement alerts, introduce tests in the developers test suites to check the integrity of the analytics tracking, ask your “moles” and “canaries” to report any issues they spot • All recommendations must document why you are tracking these data points, the data sources, the date ranges, how the data was cleaned and processed to ensure reproducibility @albangerome
  19. 19. Acceptance • Dr Elisabeth Kübler-Ross wrote: “Any natural, normal human being, when faced with any kind of loss, will go from shock all the way through acceptance.” • An analytics project will inflict a shock to any organisation and this will come with a feeling of loss for many. Data-informed organisations still need years of experience and domain knowledge • An analytics programme will take an organisation on a journey, from doubt until finally reaching acceptance @albangerome
  20. 20. You need all four • The digital transformation must be accessible, actionable, auditable and achieve acceptance • Having all four will not usher the digital transformation until there is widespread acceptance all the way to the C-suite • If you lose one of the four A’s, you can wave your digital transformation goodbye @albangerome
  21. 21. “The most beautiful people we have known are those who have known defeat, known suffering, known struggle, known loss, and have found their way out of the depths. These persons have an appreciation, a sensitivity, and an understanding of life that fills them with compassion, gentleness, and a deep loving concern. Beautiful people do not just happen.” Dr Elisabeth Kübler-Ross @albangerome
  22. 22. The data-informed stakeholder does not just happen @albangerome
  23. 23. Thank you! @albangerome
  24. 24. AAAA – Cheat Sheet Problem Cause Answers Net Benefits 1 Vendors give the illusion that analytics is easy and downplay that implementing and operating their tools often requires experts To achieve the most sales for product licences, consulting services and premium support Address the expectations set by the vendors and explain that actionable insight requires effort and often starts with nothing more than an hypothesis which will then require A/B testing to verify A better alignment between expectations and capabilities and a better understanding of the resources required such as implementation experts and a web analytics team 2 The organisations decide to track everything and spend months capturing the requirements from the stakeholders The organisations do not know what to track but the cost of data storage keeps decreasing. Intelligence agencies are notoriously tracking with little discrimination and set a bad example. The organisations decide to track everything At first, focus on tracking very little data, demonstrate the value of a data-driven approach early, support a small number of stakeholders and then expand gradually by tracking more and supporting more stakeholders Traction, an earlier and faster implementation. Once implemented, the stakeholders will learn analytics at their own pace. Restraint in your tracking appetite will help your organisation comply with regulators 3 The implementation drags on and the initial key stakeholders are withdrawing their support and resources to support other projects with better chances of success The IT team may believe they can implement an analytics tool but they lack the skills in the analytics vendor proprietary code Hire an implementation expert. Large projects may require a team of implementation experts Traction, ensuring the continued support from the key stakeholders and their resources 4 At best, the first reports contain little actionable insight. At worse, the data is just garbage or contradicts the stakeholders' beliefs The IT team's lack of implementation expertise leads to collecting garbage. When the data is correct, it may paint a picture at odds with the stakeholders' beliefs Expect to hire implementation experts more than once as an implementation is never "done" Continued trust in the analytics data and an implementation that evolves with the organisation's needs © Alban Gérôme @albangerome
  25. 25. AAAA – Cheat Sheet Problem Cause Answers Net Benefits 5 Two stakeholders extract data that should match but it does not and by a wide margin too. The organisation does not know who to trust anymore. The stakeholders lack experience with the analytics tool and data literacy too You will also need a web analytics department A single version of the truth 6 The stakeholders hide correct but unflattering data. The organisation suspects that their stakeholders lack objectivity The stakeholders leverage the perception that analytics is scientific and therefore the whole and accurate truth The head of the analytics department should report to the C-suite directly, the COO or the CEO preferably Reporting impartiality 7 Who should be in the analytics department? Internal staff or external experts? Internal staff have operational knowledge often struggle with analytics and make mistakes when extracting data. External experts can use data but often do not know which data could have a commercial impact Create a hub and spoke model, the incumbent teams who need analytics data will nominate a junior web analyst. Rotate the junior analysts between teams if possible. In the analytics team, you need external talent who will start spending time in other teams to gain operational experience The actionable insight is better aligned with the business objectives and strategy. The incumbent teams will stop become responsible for their own reporting and free the hub to focus on analysis © Alban Gérôme @albangerome
  26. 26. AAAA – Cheat Sheet Problem Cause Answers Net Benefits 8 The IT developers keep breaking the analytics code and the stakeholders blame the analytics department The IT developers still lack implementation knowledge and do not know what to test for when pushing changes live and end up breaking the analytics code. Tracking garbage data or no data at all causes no errors Get your implementation expert to speak with the developers to include automated tests to check for the integrity of the analytics implementation in their testing suites Serenity through knowing that reporting outages due to IT changes will become less and less frequent and true partnership between IT and the analytics team will emerge 9 The analytics dedicate an increasing number of resources just to monitor metrics at the expense of finding actionable insight The analytics team is now responsible for the data quality With a hub and spoke model, delegate all reporting and monitoring to the junior analysts The junior web analysts will look at a smaller dataset and will be your canaries in the coal mine when a reporting outage occurs and the advocates for analytics in their team and the organisation when the hub members are not available 10 The analytics experts are expensive, hard to find and hard to retain When an analyst spends his or her time monitoring hundreds of metrics and reporting instead of searching for actionable insight and runnning tests, the analyst will have no trouble finding another job and give your organisation a bad reputation in the analytics community Analytics should be about increasing the company ROI by improving the customer experience, not about avoiding looking bad. Monitoring and reporting will only turn the analytics team into a cost centre and your organisation will earn a bad reputation in the analytics community Recruitment is faster, cheaper and retaining is easier when your analysts leverage their personal networks. Talented web analysts want to leave the companies where they do little else than reporting and monitoring data, watching for the outage caused by IT and analyse data instead © Alban Gérôme @albangerome
  27. 27. AAAA – Cheat Sheet Problem Cause Answers Net Benefits 11 The stakeholders keep making ever increasing demands rather than keeping their promise and implementing any actionable insight The stakeholders can borrow an idea from an external thought-leader, tweak it enough to claim credit and ownership. Looking data- driven, i.e. data-justified, is much easier than being genuinely data-driven and the C-suite either cannot tell the difference or tolerate it By growing your implementation rather than trying to track everything from the beginning, your stakeholders will have to make do with the data available and stop cherry-picking data The stakeholders can develop their experience with the analytics tools, their data literacy too 12 Many biases are standing in the way of buy- in such as belief persistence, confirmation bias and selective attention bias The stakeholders are trying to address the gap between their beliefs and what the data is telling The analytics team must review the stakeholder recommendations, protect and educate the organisation about theses biases, their causes and effects The organisation becomes wiser to the biases and protect them from potentially disatrous ideas 13 Making actionable insight easier to understand, memorable and other persuasion and influence techniques do not work Asking stakeholders to implement actionable insight without question is tantamount to conservatorship, puts their careers at risk and generates a strong but slient resistance The analytics team can also find interesting trends without potential commercial impact. They need a steer from the stakeholders. This guidance will provide the stakeholders with opportunities to claim some of the credit and the ownership for the actionable insight The seasoned stakeholders can regain pride in their experience, feel valued for guiding the analytics team and help them find real actionable insight © Alban Gérôme @albangerome
  28. 28. AAAA – Cheat Sheet Problem Cause Answers Net Benefits 14 The C-suite executives asks their stakeholders to be data-driven but they their organisation is data-justified instead and the digital transformation sees very little traction The C-suite executives do not lead by example and either fail to recognise the difference between data-driven and data- justified ideas or tolerate it. They also believe that the risk of disruption for their organisation is largely exaggerated Embrace data-driven decision-making, start with a business questions, collect data, find your answers by analysing that data Proactive organisations can gain compettive advantage by becoming genuinely data- driven, become more robust and keep potential disruptors in check. What could be scarier than your 30-year competitor suddenly becoming data-driven and you are not ready? © Alban Gérôme @albangerome