The document discusses the growth and challenges in the big data market, highlighting a $60 billion expenditure on information management software and a shortfall of 150,000 data scientists. It poses critical questions about generating insights from data and presents a framework for building an 'insight machine' through codification and technological investment. The author emphasizes the importance of compounding knowledge and learning from past ideas to improve data analysis outcomes.

1. Building an Insight Machine
Matthew Granade
Strata DDBD 2015
September 29, 2015

2. Big Data, Big Spending
50%
10%
• Market growing
at 10%1
• Big Data portion
growing at 50%1
• $60 billion1 on
information
management
software
• Shortfall of
150,000 data
scientists2
1 http://www.ovum.com/press_releases/ovum-forecasts-big-data-software-to-grow-by-50/
2 McKinsey & Company

3. Common Question
“I know clearly how much I’m
spending on my data initiatives – it’s
millions of dollars. But I don’t feel like
our company is that much smarter.
… Where’s all the insight I was
promised?”

4. How do I generate
insight from all this data?
Data Insight

5. Three questions
1. What is insight?
2. Where does it come from?
3. How do I build a machine to generate it?

6. What is
insight?
Better
understanding.
Vx Vx + 1

7. Where does
insight come
from?

8. Lone genius?

9. Lone genius?

10. Lone genius?

11. Lone genius?

12. “If I have seen further,
it is by standing on the
shoulders of giants.”
-- Sir Isaac Newton

13. Compounding
Where does insight come from?

14. 0
5000
10000
15000
20000
1 11 21 31 41 51
5% 6%
Compounding interest

15. “My wealth has
come from a
combination of
living in America,
some lucky
genes, and
compound
interest.”
-- Warren Buffett

16. Checklists

17. Academic Compounding

18. Compounding Code

19. “Reporters, investors, and
others are always asking
me for the secret
ingredient. There is no one
secret. We just worked
every day to make the
math better and express it
well in code.”
-- Early RenTech Employee
Compounding Investments

20. What if you don’t?
• People waste time doing things that have
already been done.
• It’s difficult or impossible to make things
better. There is no place where ideas go.
• Idea generation suffers because people
aren’t exposed to the thinking that came
before them … they have no one to stand
on.

21. Codify Do Learn
1
2
Invest in technology and related processes to
make analysts’ work accessible and reproducible
Build an Insight Machine: Two Steps

22. Codify Do Learn

23. Codify Do Learn
Write down
your
procedure
Follow your
procedure
Compare
outcomes,
diagnose,
improve

24. The Full Circle

25. Codify Do Learn
1
2
Invest in technology and related processes to
make analysts’ work accessible and
reproducible
Build an Insight Machine: Two Steps

26. Invest in technology and related processes to
make analysts’ work accessible and reproducible
2. Version control system
3. Create building blocks
1. Clarity standard

27. Building the insight machine

28. Follow on twitter: @dominodatalab
@matthewgranade
Questions / Feedback?
Booth at Strata NYC: P3
matthew@dominodatalab.com
dominodatalab.com