Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Acceptance, accessible, actionable and auditable


Published on

Many a stakeholder presented with actionable insight have expressed doubts about data quality, its relevance or potential impact. In other cases, the stakeholders will want a data analyst who also commands knowledge of the business or a similar unicorn. The web analytics practitioners would rise to the challenge, and the stakeholders will then want their own Hans Rosling and his dazzling data visualisation and raconteur sills. Your stakeholders are silently experiencing the data transformation like a conservatorship. How can you help them perceive your efforts like a temporary guardianship from which they will emerge as ready to face your data-driven competitors?

Published in: Technology

Acceptance, accessible, actionable and auditable

  1. 1. Acceptance, Accessible, Actionable and Auditable A model for the digital transformation and excellence in analytics Alban Gérôme Web Analytics Wednesday Copenhagen 4 October 2017
  2. 2. "Le doute, morne oiseau, nous frappe de son aile... Et l'horizon s'enfuit d'une fuite éternelle !..."
  3. 3. "Le doute, morne oiseau, nous frappe de son aile... Et l'horizon s'enfuit d'une fuite éternelle !..." "Doubt, dismal bird, beat us down with its wing... - And the horizon rushes away in endless flight!..."
  4. 4. "Le doute, morne oiseau, nous frappe de son aile... Et l'horizon s'enfuit d'une fuite éternelle !..." "Doubt, dismal bird, beat us down with its wing... - And the horizon rushes away in endless flight!..." Arthur Rimbaud, Sun and Flesh (Credo in Unam)
  5. 5. Barbarians at the gate!
  6. 6. Barbarians at the gate! No previous experience is an asset, leaving them no option but to innovate. They burnt their boats and even killed their horses! Our massive experience has become merely solidified concrete around our feet, making us sitting ducks for disruption!
  7. 7. You and whose army?
  8. 8. You and whose army? Our decades of experience and substantial brand recognition are impossible to replicate. Your data-driven ways, not so much. Once our data transformation is complete, we will buy your company and your data-servers! Real cheap too!
  9. 9. We must disrupt them before their digital transformation is complete! We must track everything and find actionable insight ASAP before they disrupt us! The heat is on! VS
  10. 10. Heroic times
  11. 11. Heroic times • The C-suite executives and stakeholders do not know what to track but… • Cost of data storage keeps going down • Let’s track everything, it might be useful one day • If it’s good enough for the NSA, it’s good enough for us
  12. 12. Heroic times • IT will implement the analytics data collection • No worries! It’s just Javascript! • We will setup user access and schedule reports • Your implementation will be delivered on X • It will cost Y on the first year and Z from next year onwards
  13. 13. IT cannot implement analytics
  14. 14. IT cannot implement analytics • A Javascript expert does not an analytics implementation expert make • Analytics library files are based on proprietary code • What’s a difference between a prop, an eVar, an event and a page view? • When is this implementation going to be “done”? • We are going to need an implementation consultant
  15. 15. Signal vs Noise
  16. 16. Signal vs Noise • The actionable insight is not jumping out at you like at the vendor road shows • The reports seems to tell a different story than what they believe to be true. It must be an implementation issue • Nicolas Cage film appearances and drownings are correlated? Who would have thought that Nicolas cage was so evil?
  17. 17. Analysis paralysis
  18. 18. Analysis paralysis • People have a finite absorption capacity for new data, skills and knowledge, says Dr. Mark Kelly of Imperial College, London. Data visualisation helps • Nearly 40 years of spreadsheet software has not made us significantly more data literate • People will focus on data that seems to make sense and even confirm their own beliefs
  19. 19. Thinking too fast
  20. 20. Thinking too fast • A sports shop sells tennis racquets and tennis balls • They sell a ball and racquet as a pack for 1,100 kroner, and the racquet costs 1,000 kroner more than the ball • How much does the ball cost?
  21. 21. Accessible
  22. 22. Accessible • Explain to your key supporters at the start of the project that you will not try to track everything • Tracking everything from the start leads to data-justified instead of data- driven decision-making • The implementation will grow in scope, complexity and number of stakeholders supported over time but it will allow you start faster • The stakeholders will avoid analysis paralysis and have time to absorb new skills and improve their data literacy
  23. 23. All I do is reporting…
  24. 24. All I do is reporting… • Finding actionable insight takes time and all everybody seems to want is reports! • The developers implement a change, we lose tracking and I get the blame! • Guardian of data quality? I cannot monitor 300 metrics all day!
  25. 25. Decentralised model
  26. 26. Decentralised model • Everybody has access to analytics data • Tendency to blow their own trumpet and sweep the bad news under the carpet • Poor data literacy and analytics tool knowledge leading to two people extracting what should be the same data and end up with two versions of the truth
  27. 27. Centralised model
  28. 28. Centralised model • Create a new team that will guarantee data quality and a single, objective version of the truth • This new team will need to support all the incumbent teams, which can cause backlog issues and delays
  29. 29. Centralised model • These people will have to be hired externally, they are costly, hard to retain and replace • They do not know the business well and their recommendations are often not aligned with the business
  30. 30. Hub and spoke model
  31. 31. Hub and spoke model • Hybrid of the decentralised and centralised models • A small central team of experts hired externally • Embedded junior web analysts in the other teams, existing members of their teams with domain knowledge, trained as web analyst • They do the reporting and monitoring for their own team’s remit • They act like the canaries in the coal mine and alert the hub earlier • They even rotate between teams!
  32. 32. A/B testing
  33. 33. A/B testing • Because it’s hard to argue against the voice of the customer • 4 steps: • Plan – We are trying to answer a business question, what is the hypothesis? • Do – Run a test where 10% of our visitors will another version of the content than the rest • Check – Which version of the content won? The new one or the old one? Have we collected enough data to be confident in the results? • Act – We serve the winner to 100% of the visitors and move on to the next A/B test
  34. 34. Actionable
  35. 35. Actionable • Implement a hub and spoke model with rotating embedded web analysts • If a team wants reports let their embedded web analysts do them. That team should stop quickly when they realise it’s a low value/high effort exercise • Let the embedded web analysts monitor their own metrics and be your canaries in the coal mine. It’s faster and more efficient • Run A/B tests before pushing layout changes live
  36. 36. sssssCan I trust analytics?
  37. 37. sssssCan I trust analytics? • The developers remove analytics code when they make changes • The testers are not capturing these issues because no errors fired • The web analysts spotted the issue only three weeks later • So much data, where do I start? How can I spot the moonwalking gorilla in the middle of all this and not focus on the data that suits me? • What’s a confidence interval? • What if a visitor uses Ad Blocking software, Ghostery, keeps clearing their cookies and block Javascript?
  38. 38. Reproducibility
  39. 39. Reproducibility • Any analysis should document • the business question we are trying to answer • the data sources used and where to find them • how the data was cleaned • The conclusions of the analysis • Are there better data souces? • Are there other methods to remove outliers? • Did we reach the same conclusions?
  40. 40. Alerts, model prediction errors
  41. 41. Alerts, model prediction errors • Embedded web analysts : must raise the issue when a reporting outage occurs • Hub analysts: • speak with the developers and testers to build a test suite to include in their Selenium or chromeless browser testing • document and communicate the reporting outages so your stakeholders should not have to remember before using historical data • Model prediction errors are a great source to drive analyses
  42. 42. Regulations and restraint
  43. 43. Regulations and restraint • GDPR will come into force in May 2018, with substantial fines for non- compliant companies • Machine learning and AI can lead to recommendations which could damage the brand: • 2012: Target stores in the US send baby clothes and cribs coupons to teenage girl • 2017: Uber fare prices x 4 during London metro strike
  44. 44. Auditable
  45. 45. Auditable • Reproducible research will ensure that all analyses start with a business questions rather than personal beliefs • It will also start the discussion on whether the conclusions remain the same with better data sources and other methodologies • Data collection outages should be • documented and communicated • lead to the creation of test suites running before going live • GDPR will make auditable analytics more relevant than ever • Companies should veto brand-damaging recommendations by machine learning and AI algorithms
  46. 46. Either with us or against us
  47. 47. Either with us or against us Imagine there is an opportunity for 10% cost reduction You have identified the opportunity after analysing the data and now you must convince a stakeholder to implement your recommendation Your stakeholder found the same idea, but you did not, in a business newsletter nobody else in the company reads. He can claim the credit of that idea all for himself
  48. 48. Conservatorship
  49. 49. Conservatorship • A new team was created to make recommendations on how I should run my business • I cannot claim credit for their recommendations • I am under no obligation to implement them • But I must be able to prove I use their data to make decisions
  50. 50. Conservatorship • I will overwhelm the analytics team with custom reports and stop them from meddling into my business • I will use their data when it supports my strategy, brand the rest as unreliable and probably the product of a bad implementation • The C-suite executives cannot tell the difference between a data-driven and a data-justified decision
  51. 51. Regency
  52. 52. Regency • One day, having to prove that analytics can deliver value at a company will be as stupid as having to prove that having electricity will make the lights in the office work. If analytics can deliver value it can deliver value anywhere • But for now the business to learn, improve their data literacy. Until then the analytics team will take credit but this is only a temporary situation
  53. 53. Regency These early successes will pale in comparison to the stakeholders’ when they are ready and the credit will be all theirs and rightly so
  54. 54. Five stages “The five stages – denial, anger, bargaining, depression, and acceptance – are a part of the framework that makes up our learning to live with the one we lost. They are tools to help us frame and identify what we may be feeling. But they are not stops on some linear timeline in grief.” “Any natural, normal human being, when faced with any kind of loss, will go from shock all the way through acceptance.” Dr. Elisabeth Kübler-Ross.
  55. 55. Acceptance
  56. 56. Acceptance • The stakeholders are feeling like being place under a conservatorship they cannot openly rebel against • Until the C-suite executives show the example by embracing a data- driven decision-making process, the stakeholders will play a game of superficial compliance
  57. 57. Acceptance • The stakeholders need to trust that their experience is not obsolete, far from it. It will make them unstoppable once they are truly data- driven • Until you have acceptance across the whole business, the more accurate implementation, the most adequate data visualisation, the smoothest relationships with your stakeholders is all for nothing
  58. 58. Self-ignition temperature
  59. 59. Self-ignition temperature • Fire requires fuel, oxygen and heat to sustain itself. Remove one and the fire stops • Below a certain temperature no fire can start even when these three elements are present • Digital transformation requires accessible, actionable and auditable analytics to sustain itself • Without acceptance your digital transformation cannot start even when you have nailed all three aspects of analytics
  60. 60. “The most beautiful people we have known are those who have known defeat, known suffering, known struggle, known loss, and have found their way out of the depths. These persons have an appreciation, a sensitivity, and an understanding of life that fills them with compassion, gentleness, and a deep loving concern. Beautiful people do not just happen.”
  61. 61. “The most beautiful people we have known are those who have known defeat, known suffering, known struggle, known loss, and have found their way out of the depths. These persons have an appreciation, a sensitivity, and an understanding of life that fills them with compassion, gentleness, and a deep loving concern. Beautiful people do not just happen.” Dr. Elisabeth Kübler-Ross “Death: The Final Stage of Growth”, 1975
  62. 62. Mange tak! @albangerome
  63. 63. Further reading • Accessible, Actionable, Auditable – originally from Eric Ries’ book “The Lean Start-up” but his paradigm was not directly related to analytics per se • Accessible • • waste-of-time • your-organization • analytics.pdf • • • • Actionable • • “Cult of Analytics” by Steve Jackson for the REAN framework and the Hub and Spoke model • “Thinking Fast and Slow” by Daniel Kahneman, 2002 Nobel Prize Winner in Economics Science • • Auditable • • Selective Attention Test by Daniel Simmons and Christopher Chabris: • PhantomJS: analytics-implementation • Google Chrome chromeless: • londoners-by-quadrupling-fares-amid-tube-strike-chaos-a3435891.html • teen-girl-was-pregnant-before-her-father-did/#46986a2c6668 • data-43900072 • Acceptance • • “Games People Play”, Dr. Eric Berne, especially the“Look how hard I’ve tried” game analysis • “Death: The Final Stage of Growth”, Dr. Elisabeth Kübler-Ross • and