The document discusses a model for digital transformation in analytics, emphasizing the need for accessible, actionable, and auditable analytics to support decision-making and overcome organizational challenges. It highlights issues like data tracking difficulties, the importance of avoiding analysis paralysis, and the necessity of improving data literacy among stakeholders. Additionally, the piece suggests implementing a hub and spoke model for analytics teams and underscores the significance of addressing compliance with regulations like GDPR.

1. Acceptance, Accessible,
Actionable and Auditable
A model for the digital transformation and excellence in analytics
Alban Gérôme
Web Analytics Wednesday Copenhagen
4 October 2017

2. "Le doute, morne oiseau, nous
frappe de son aile...
Et l'horizon s'enfuit d'une fuite
éternelle !..."

3. "Le doute, morne oiseau, nous
frappe de son aile...
Et l'horizon s'enfuit d'une fuite
éternelle !..."
"Doubt, dismal bird, beat us
down with its wing...
- And the horizon rushes away in
endless flight!..."

4. "Le doute, morne oiseau, nous
frappe de son aile...
Et l'horizon s'enfuit d'une fuite
éternelle !..."
"Doubt, dismal bird, beat us
down with its wing...
- And the horizon rushes away in
endless flight!..."
Arthur Rimbaud, Sun and Flesh
(Credo in Unam)

5. Barbarians at the gate!

6. Barbarians at the gate!
No previous experience is an
asset, leaving them no option
but to innovate. They burnt
their boats and even killed
their horses!
Our massive experience has
become merely solidified
concrete around our feet,
making us sitting ducks for
disruption!

7. You and whose army?

8. You and whose army?
Our decades of experience and substantial brand recognition are
impossible to replicate. Your data-driven ways, not so much.
Once our data transformation is complete, we will buy your company
and your data-servers! Real cheap too!

9. We must disrupt them
before their digital
transformation is complete!
We must track everything
and find actionable insight
ASAP before they disrupt us!
The heat is on!
VS

10. Heroic times

11. Heroic times
• The C-suite executives and stakeholders do not know what to track
but…
• Cost of data storage keeps going down
• Let’s track everything, it might be useful one day
• If it’s good enough for the NSA, it’s good enough for us

12. Heroic times
• IT will implement the analytics data collection
• No worries! It’s just Javascript!
• We will setup user access and schedule reports
• Your implementation will be delivered on X
• It will cost Y on the first year and Z from next year onwards

13. IT cannot implement analytics

14. IT cannot implement analytics
• A Javascript expert does not an analytics implementation expert make
• Analytics library files are based on proprietary code
• What’s a difference between a prop, an eVar, an event and a page view?
• When is this implementation going to be “done”?
• We are going to need an implementation consultant

15. Signal vs Noise

16. Signal vs Noise
• The actionable insight is not jumping out at you like at
the vendor road shows
• The reports seems to tell a different story than what they
believe to be true. It must be an implementation issue
• Nicolas Cage film appearances and drownings are
correlated? Who would have thought that Nicolas cage
was so evil?

17. Analysis paralysis

18. Analysis paralysis
• People have a finite absorption capacity for new data,
skills and knowledge, says Dr. Mark Kelly of Imperial
College, London. Data visualisation helps
• Nearly 40 years of spreadsheet software has not made us
significantly more data literate
• People will focus on data that seems to make sense and
even confirm their own beliefs

19. Thinking too fast

20. Thinking too fast
• A sports shop sells tennis racquets and tennis balls
• They sell a ball and racquet as a pack for 1,100 kroner, and the
racquet costs 1,000 kroner more than the ball
• How much does the ball cost?

21. Accessible

22. Accessible
• Explain to your key supporters at the start of the project that you will not
try to track everything
• Tracking everything from the start leads to data-justified instead of data-
driven decision-making
• The implementation will grow in scope, complexity and number of
stakeholders supported over time but it will allow you start faster
• The stakeholders will avoid analysis paralysis and have time to absorb new
skills and improve their data literacy

23. All I do is reporting…

24. All I do is reporting…
• Finding actionable insight takes time and all everybody seems to
want is reports!
• The developers implement a change, we lose tracking and I get the
blame!
• Guardian of data quality? I cannot monitor 300 metrics all day!

25. Decentralised model

26. Decentralised model
• Everybody has access to analytics data
• Tendency to blow their own trumpet and sweep the bad news under
the carpet
• Poor data literacy and analytics tool knowledge leading to two people
extracting what should be the same data and end up with two
versions of the truth

27. Centralised model

28. Centralised model
• Create a new team that will guarantee data quality and a single,
objective version of the truth
• This new team will need to support all the incumbent teams, which
can cause backlog issues and delays

29. Centralised model
• These people will have to be hired externally, they are costly, hard to
retain and replace
• They do not know the business well and their recommendations are
often not aligned with the business

30. Hub and spoke model

31. Hub and spoke model
• Hybrid of the decentralised and centralised models
• A small central team of experts hired externally
• Embedded junior web analysts in the other teams, existing members
of their teams with domain knowledge, trained as web analyst
• They do the reporting and monitoring for their own team’s remit
• They act like the canaries in the coal mine and alert the hub earlier
• They even rotate between teams!

32. A/B testing

33. A/B testing
• Because it’s hard to argue against the voice of the customer
• 4 steps:
• Plan – We are trying to answer a business question, what is the hypothesis?
• Do – Run a test where 10% of our visitors will another version of the content
than the rest
• Check – Which version of the content won? The new one or the old one?
Have we collected enough data to be confident in the results?
• Act – We serve the winner to 100% of the visitors and move on to the next
A/B test

34. Actionable

35. Actionable
• Implement a hub and spoke model with rotating embedded web
analysts
• If a team wants reports let their embedded web analysts do them.
That team should stop quickly when they realise it’s a low value/high
effort exercise
• Let the embedded web analysts monitor their own metrics and be
your canaries in the coal mine. It’s faster and more efficient
• Run A/B tests before pushing layout changes live

36. sssssCan I trust analytics?

37. sssssCan I trust analytics?
• The developers remove analytics code when they make changes
• The testers are not capturing these issues because no errors fired
• The web analysts spotted the issue only three weeks later
• So much data, where do I start? How can I spot the moonwalking
gorilla in the middle of all this and not focus on the data that suits
me?
• What’s a confidence interval?
• What if a visitor uses Ad Blocking software, Ghostery, keeps clearing
their cookies and block Javascript?

38. Reproducibility

39. Reproducibility
• Any analysis should document
• the business question we are trying to answer
• the data sources used and where to find them
• how the data was cleaned
• The conclusions of the analysis
• Are there better data souces?
• Are there other methods to remove outliers?
• Did we reach the same conclusions?

40. Alerts, model prediction errors

41. Alerts, model prediction errors
• Embedded web analysts : must raise the issue when a reporting
outage occurs
• Hub analysts:
• speak with the developers and testers to build a test suite to include in their
Selenium or chromeless browser testing
• document and communicate the reporting outages so your stakeholders
should not have to remember before using historical data
• Model prediction errors are a great source to drive analyses

42. Regulations and restraint

43. Regulations and restraint
• GDPR will come into force in May 2018, with substantial fines for non-
compliant companies
• Machine learning and AI can lead to recommendations which could
damage the brand:
• 2012: Target stores in the US send baby clothes and cribs coupons to teenage
girl
• 2017: Uber fare prices x 4 during London metro strike

44. Auditable

45. Auditable
• Reproducible research will ensure that all analyses start with a
business questions rather than personal beliefs
• It will also start the discussion on whether the conclusions remain the
same with better data sources and other methodologies
• Data collection outages should be
• documented and communicated
• lead to the creation of test suites running before going live
• GDPR will make auditable analytics more relevant than ever
• Companies should veto brand-damaging recommendations by
machine learning and AI algorithms

46. Either with us or against us

47. Either with us or against us
Imagine there is an opportunity for 10% cost reduction
You have identified the
opportunity after analysing the
data and now you must
convince a stakeholder to
implement your
recommendation
Your stakeholder found the
same idea, but you did not, in a
business newsletter nobody else
in the company reads. He can
claim the credit of that idea all
for himself

48. Conservatorship

49. Conservatorship
• A new team was created to make recommendations on how I should run
my business
• I cannot claim credit for their recommendations
• I am under no obligation to implement them
• But I must be able to prove I use their data to make decisions

50. Conservatorship
• I will overwhelm the analytics team with custom reports and stop them
from meddling into my business
• I will use their data when it supports my strategy, brand the rest as
unreliable and probably the product of a bad implementation
• The C-suite executives cannot tell the difference between a data-driven and
a data-justified decision

51. Regency

52. Regency
• One day, having to prove that analytics can deliver value at a company will
be as stupid as having to prove that having electricity will make the lights in
the office work. If analytics can deliver value it can deliver value anywhere
• But for now the business to learn, improve their data literacy. Until then
the analytics team will take credit but this is only a temporary situation

53. Regency
These early successes will pale in comparison
to the stakeholders’ when they are ready and
the credit will be all theirs and rightly so

54. Five stages
“The five stages – denial, anger,
bargaining, depression, and acceptance –
are a part of the framework that makes up
our learning to live with the one we lost.
They are tools to help us frame and identify
what we may be feeling. But they are not
stops on some linear timeline in grief.”
“Any natural, normal human being, when
faced with any kind of loss, will go from
shock all the way through acceptance.”
Dr. Elisabeth Kübler-Ross.

55. Acceptance

56. Acceptance
• The stakeholders are feeling like being place under a conservatorship
they cannot openly rebel against
• Until the C-suite executives show the example by embracing a data-
driven decision-making process, the stakeholders will play a game of
superficial compliance

57. Acceptance
• The stakeholders need to trust that their experience is not obsolete,
far from it. It will make them unstoppable once they are truly data-
driven
• Until you have acceptance across the whole business, the more
accurate implementation, the most adequate data visualisation, the
smoothest relationships with your stakeholders is all for nothing

58. Self-ignition temperature

59. Self-ignition temperature
• Fire requires fuel, oxygen
and heat to sustain itself.
Remove one and the fire
stops
• Below a certain
temperature no fire can
start even when these
three elements are
present
• Digital transformation
requires accessible,
actionable and auditable
analytics to sustain itself
• Without acceptance your
digital transformation
cannot start even when
you have nailed all three
aspects of analytics

61. “The most beautiful people we have known are those who have known
defeat, known suffering, known struggle, known loss, and have found
their way out of the depths. These persons have an appreciation, a
sensitivity, and an understanding of life that fills them with compassion,
gentleness, and a deep loving concern. Beautiful people do not just
happen.”

62. “The most beautiful people we have known are those who have known
defeat, known suffering, known struggle, known loss, and have found
their way out of the depths. These persons have an appreciation, a
sensitivity, and an understanding of life that fills them with compassion,
gentleness, and a deep loving concern. Beautiful people do not just
happen.”
Dr. Elisabeth Kübler-Ross “Death: The Final Stage of Growth”, 1975

63. Mange tak!
http://www.albangerome.com
@albangerome

64. Further reading
• Accessible, Actionable, Auditable – originally from Eric Ries’ book “The Lean Start-up”
but his paradigm was not directly related to analytics per se
• Accessible
• https://hbr.org/2013/01/why-it-fumbles-analytics
• https://hbr.org/2016/07/how-ceos-can-keep-their-analytics-programs-from-being-a-
waste-of-time
• https://hbr.org/2017/06/how-to-integrate-data-and-analytics-into-every-part-of-
your-organization
• https://assets.kpmg.com/content/dam/kpmg/xx/pdf/2016/10/building-trust-in-
analytics.pdf
• http://www.gartner.com/newsroom/id/3130017
• https://en.wikipedia.org/wiki/John_Gall_(author)
• https://hbr.org/2017/06/does-your-company-know-what-to-do-with-all-its-data
• Actionable
• https://hbr.org/2016/08/the-reason-so-many-analytics-efforts-fall-short
• “Cult of Analytics” by Steve Jackson for the REAN framework and the Hub and Spoke
model
• “Thinking Fast and Slow” by Daniel Kahneman, 2002 Nobel Prize Winner in
Economics Science
• https://hbr.org/2017/06/a-refresher-on-ab-testing
• Auditable
• https://hbr.org/2017/09/only-3-of-companies-data-meets-basic-quality-standards
• Selective Attention Test by Daniel Simmons and Christopher Chabris:
https://www.youtube.com/watch?v=vJG698U2Mvo
• PhantomJS: https://www.slideshare.net/AlbanGrme/using-phantom-js-to-qa-your-
analytics-implementation
• Google Chrome chromeless:
https://developers.google.com/web/updates/2017/04/headless-chrome
• https://www.standard.co.uk/news/transport/uber-slammed-for-ripping-off-
londoners-by-quadrupling-fares-amid-tube-strike-chaos-a3435891.html
• https://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-
teen-girl-was-pregnant-before-her-father-did/#46986a2c6668
• https://www.slideshare.net/Management-Thinking/infographic-the-virtuous-circle-of-
data-43900072
• Acceptance
• https://hbr.org/2017/04/how-companies-say-theyre-using-big-data
• “Games People Play”, Dr. Eric Berne, especially the“Look how hard I’ve tried” game
analysis
• “Death: The Final Stage of Growth”, Dr. Elisabeth Kübler-Ross
• https://en.wikipedia.org/wiki/Fire_triangle and
https://en.wikipedia.org/wiki/Autoignition_temperature