This document discusses common issues ("pathologies") that organizations face when adopting analytics and data-driven approaches. It summarizes these pathologies in 3 sentences or less: Organizations often treat analytics as a "black box" without understanding how it works due to the technical nature of analytics and lack of transparency in algorithms and methods. Many projects fail because organizations jump into analytics without properly preparing their data, validating results, or planning how insights will be implemented and drive business changes. To successfully adopt analytics, organizations must ask critical questions about data quality, intended use cases, and consequences of results in order to focus efforts and avoid wasting resources on initiatives that do not provide value.

1. Analytics and Witch Doctoring:
A Cure for the Black Box
Mentality
February 1, 2011
O’Reilly Strata Conference
J.C. Herz, Batchtags LLC
jc@tripledex.com

2. Analytics: An Intervention
• Original Sin
Enterprise
Pathologies Critical Questions

3. Analytics: Occult Phenomenon
Very powerful
Don’t understand it
Practitioners possess
arcane knowledge

4. Alchemy

5. Secret
Algorithms

6. Greek Letters Are Your
Kryptonite

7. High Status Helplessness
• If you understood the technology, you’d
be one of those people whose job it is
to make technology work.
• You know, underlings

8. Executive ADD - Kaching!
• Re-starts are where
consulting shops
make their money

9. Mid-Life Crisis

10. Shiny Pebble Syndrome
• Infoviz Porn: Visualization with no
use case
– Ex: Social Network visualization. Why?
– START with a use case and work
forward
• Demo Envy: just because it looks
slick doesn’t mean it’s possible, or
even advisable, to pipe your data
into it.

11. A Ballad of Spectacular
Information Display
• Time Magazine 1976
• Telex text routing:
information off the wire
goes to terminals, properly
foldered
• Z8 terminal display awes
executives
• Pneumatic system not
eliminated

12. Technical Reality vs.
Leadership Attention Span

13. Half Ass Syndrome
• Halfway into the project, jump off into
the next problem.
• Haven’t refined results or hypothesis
• Failure blamed on technology, but it’s
really loss of interest and desire for
instant gratification

14. Shelfware Syndrome
• The guy who was driving the program left...
• Approach-Avoidance conflict --> pilot-itis
• A US agency has $30M of software that
hasnʼt been installed…some of it with
maintenance contracts.
• Base Model vs. Fully Loaded
– One enterprise bought $12M worth of
Autonomy before figuring out that the add-ons
they needed would be another $22M.

15. Customization Before Testing

16. Critical Question: What is the
Validation Test?
• Formulating the validation test keeps
both the customer and the developer
focused - and honest
• Suggest pay for performance, and see if
the developer or vendor freaks out.
• Make sure validation is ongoing - in
case the ground is shifting

17. Data Due Diligence & Auditing

18. Shame
Shame

19. Ugly Babies

20. Critical Question: Data Quality
• How complete is it?
– Ex: 600 custom fields, only two have more than 50%
coverage
• How accurate is it? How do you know?
• How consistent is it?
– Good test: make three calls to different parts
of the company, to get an answer to a factual
question that doesn’t require calculation.

21. Critical Question: Half-Life of Data
• How long is the
data accurate?
• How long is the
data useful?

22. Critical Question: Real World Context
• Without real world data, “behavioral” metrics
are misleading
• Where is the transactional data that validates
insights from non-transactional data?
• How would you prove the magic analytics
WRONG?

23. • Are you prepared to spend painful
amounts of money cleaning up your data?
• Crack heads if people don’t share data?
• Make business units accountable for their
data?
• Play hardball to make sure data is not
stored in single-application proprietary
formats?
Data: Gut Check

24. Critical Questions: Workflow
• What workflow changes will this
proposed capability require?
• People hate changing their workflow,
even if it’s an improvement
• Never attribute to stupidity what can be
attributed to laziness
• What is your plan for changing
workflow? How do you enforce it?

25. The perfect application that
no-one uses is still worthless

26. Process

27. Politics

28. What are you prepared to do?

29. Critical Question: Consequences
• What actions are you willing to take on
the basis of validated analytic insight?
– Change your product?
– Change your marketing budget?
– Change people’s job descriptions?
– Re-allocate R&D budgets?
• What actions are you not willing to take?

30. Stakes

31. Critical Question: Tempo
• How fast will a decision be made on the
basis of analytic insight?
• Quarterly?
• Daily?
• Within seconds?
• Milliseconds?
• Never?
• Realtime vs. Continuous vs. Batch

32. Precision vs. Accuracy
When precision exceeds accuracy, you’re
setting yourself up for analytic failure

33. OODA Loop
Which of these does an analytic tool/technology do?

34. Before You Rip ‘n’ Replace:
What is the exit cost of this
technology?
Does “turnkey” mean
monoculture?

35. Business Payoff vs.
Intellectual Appeal
Social Network
Analysis
Operations
Research
Market
Segmentation
Competitive
Intelligence
Pilots to test new
analyst tools with
tiny amounts of
generic data
360º
Lead
Scoring
Validate Marketing
Effectiveness

36. Questions?
• J.C. Herz jc@tripledex.com (202) 213-3151

37. Base Model vs. Fully Loaded

38. “Never attribute to stupidity
what can be explained by
laziness.”

39. Ugly Babies & Pretty Babies