The document discusses big data science, comparing data to crude oil, and emphasizing its extraction and refinement for valuable insights. It outlines the differences between big data and traditional data analysis, illustrating how data science contributes to understanding data through various stages of processing. Lastly, it highlights the challenges and potential of big data across different sectors, underscoring the necessity of leveraging it effectively.

1. BIG DATA SCIENCE
“The price of light is far less than the cost of darkness”
Chandan Rajah [ @ChandanRajah ]

2. BENEFITS OF BIG DATA
COST SPEED
AGILITY CAPABILITY

3. Steps to the EPIPHANY
WHERE
WHAT WHY
DEMO

4. What is Big Data ?
Big Data ≠ Data Volume
Big Data = Crude Oil
Think of data like ‘Crude Oil’
Big Data is about extracting ‘crude oil’; transporting it in ‘pipelines’; storing it in ‘mega tanks’

5. What is Data Science ?
Data Science ≠ Statistical Analysis
Data Science = Oil Refinery
Data science is about ‘treating’ data; applying ‘science’ to the data;
Refine the data ‘results’; and combine to form ‘insight’

6. Knowns, Unknowns & DIKUW FTW!
known knowns
we know we know
known unknowns
we know we don’t know
unknown unknowns
we don’t know we don’t know
D
DATA
I
INFORMATION
K
KNOWLEDGE
W
WISDOM
U
UNDERSTANDING
raw what how to why when
numbers description experience cause & effect prediction
letters context tested proven what’s best
symbols relationship instruction
signals reports programs models
PAST FUTURE
Data Engineer Data Analyst Data Miner Data Scientist
known knowns
known unknowns unknown unknowns

7. Data Analytics to Data Discovery ?
data you know
data you don’t know
questionsyou’reasking
questionsyou’renotasking
Data Analyst
Data Scientist
Data
Analytics
Data Discovery
DATA MODELLING
Y  F( X, random noise, parameters)
ALGORITHMIC MODELLING
Y  [ BLACK BOX ]  X

8. DIVIDE
SCATTER
Split Data in Block
Replicate and Store
Petabytes of Resilience
CONQUER
EXPLORE
1000s of Parallel Threads
Explore Every Path
Machine Learning
INSIGHT
GATHER
Real Time Action
Periodic Dashboards
Iterative Evolution
What is the Big Idea ?

9. Divide = HDFS
Name Node
1 32
Client 1. Create Metadata
2. Put Blocks
Data Nodes
Control / Monitoring
1 1
2 2
3 3
WRITE
Name Node
1 1 1 2
2
2
3 3 34
4 4
Client 1. Get Metadata
2. Fetch Blocks
Data Nodes
Control / Monitoring
READ

10. Conquer = MapReduce

11. Insight = Functional Paradigm

12. Steps to the EPIPHANY
WHERE
WHAT WHY
DEMO

13. Why is Big Data needed ?
VOLUME VELOCITY VARIETY
Exponential growth; 2x in 2 yrs
PB (1000 TB) is now common
Event streams; never at rest
640k GB per internet minute
100s of data sources
85% not in a table

14. Where in the Value Chain ?
Generation Transport Knowledge Output Value
BIG DATA SCIENCE
Straddles all four Challenge Areas

15. Steps to the EPIPHANY
WHERE
WHAT WHY
DEMO

16. Big Data Heat Map – Gartner 2012

17. Big Data Potential by Sector – McKinsey for USBLS, 2011

18. Big Data Investment by Industry – Gartner, 2012

19. Top Big Data Challenges – Gartner, 2012

20. Survey on Big Data Investments – IDG Survey, 2013

21. Survey on Main Drivers to Invest – IDG Survey, 2014

22. Steps to the EPIPHANY
WHERE
WHAT WHY
DEMO

23. DEMO

24. RECAP OF BENEFITS
COST SPEED
AGILITY CAPABILITY

25. LAST WORDS OF WISDOM
NOT ALL ROADS LEAD TO ROME
TIME VALUE OF DATA KNOWLEDGE IS POWER
I AM AN INDIVIDUAL

26. “The price of light is far less than the cost of darkness”