The document discusses a webcast about data lakes, hosted by the Bloor Group, outlining fundamental components such as hardware, software, and data, and their interrelationships. It emphasizes the importance of data governance in handling big data and the evolution of technology layers impacting data storage and analytics. Additionally, it covers the significance of events and streaming data, highlighting various frameworks and architectures in the realm of big data processing.

1. Grab some
coffee and
enjoy the
pre-show
banter before
the top of the
hour!

2. The Data Lake Survival Guide
Exploratory Webcast | October 26, 2016
SPONSORED BY

3. Presenting
Robin Bloor
Chief Analyst, The Bloor Group
@robinbloor robin.bloor@bloorgroup.com
Host: Eric Kavanagh
CEO, The Bloor Group
@eric_kavanagh eric.kavanagh@bloorgroup.com
Dez Blanchfield
Data Scientist, The Bloor Group
@dez_blanchfield dez.blanchfield@bloorgroup.com

4. Findings Webcast
January 12, 2017
Data Lake Survival Guide
Roundtable Webcast
December 8, 2016
Exploratory Webcast
October 26, 2016

5. Data Lake
Survival
Robin Bloor, PhD

6. The Sequence of Topics….
1 Disturbance in the Force
2 What is a Data Lake,
exactly?
3 Streams and Events

7. 1
Disturbance
in the
Force

8. The Generic Dimensions of IT
q All IT involves 4 components (only)
q Users
q Software
q Data
q Hardware
q They all relate to each other
q Change any one of these and the other
three components have to adjust
q Aggregate these and you get a process
q Time will impose change anyway
q We can also consider:
q Staff
q Business Processes
q Business Information
q Facility
q And also
q People
q Information
q Human Activity
q Civilization (Stuff)
Four Fundamental (IT) Factors
Hardware
Users
Software Data
BusinessInformation
BusinessProcess
HumanActivity
AllInformation
Staff
Facility
People
Civilization
TIME

9. The Technology Layers
§ The buying impulse
descends through the
stack
§ The impact of
technology change rises
up the stack
§ This ensures the
eventual “legacification”
of all technology
The Buying
Impulse Goes
Down
Technology
Change Rises Up
The Technology
Layers

10. Disruption in the Technology Layers
§ Disruption (as
innovation) can happen in
any layer
§ Where it occurs it will
impact all layers above it
§ And it may also impact
the layers below it (but
less quickly)
§ There is no such thing as
future-proof; but some
technologies definitely live
longer
The Buying
Impulse Goes
Down
Technology
Change Rises Up
The Technology
Layers

11. § Mainframe Computer (Batch architecture)
§ On-line Interaction (Centralized
architecture)
§ PC (Client Server)
§ Internet (Multi-tier architecture)
§ Mobile (Service Oriented architecture)
§ Internet of Things (Event Driven
Architecture)
Tech Revolutions
Note that all of these disruptive changes
were driven by hardware innovation
Cloud
Centralized Computer Systems
PC Based Systems
Integrated Systems
Limited process power
Terminals only
Few applications
No external data sources
Extensive process power
PCs & Apps
Analytics capability
Wealth of applications
Many external data sources
Moderate process power
PCs
Spreadsheets & email
Many applications
Few external data sources

12. Parallelism: The Imp Out of the Bottle
u Multicore chips enabled
parallelism
u It has changed the whole
performance equation
u It enabled Big Data
u Big Data is really Big
Processing

13. The Impact of Parallelism
We used to see 10x performance
improvement every 6 years, now we
see 1000x (and that’s just an
approximation)

14. Hardware Factors
q CPUs, GPUs & FPGAs
q Cross breeding
q SoCs
q 3D Xpoint and PCM (and
memristor?)
q SSDs & parallel access
q Parallel hardware
architectures
Performance is accelerating
and costs continue to fall.

15. The Perfect Storm (Software)
q The triumph of Open
Source as a business model
q The dominance of Apache
q Hadoop, the platform
for data
q Spark, for speed
q Kafka, for connectivity
q The triumph of the cloud
and its dominance
q Little data is also big data
q Cost challenges

16. Then the Data
Lake evaporated
into the Cloud
2
What is a
Data Lake?

17. Everything in flux
u Hardware (network,
storage, servers)
u Data Sources
u Data Staging
u Data Volumes
u Data Flow
u Data Governance
u Data Usage
u Data Structures
u Schema definition
u Ingest Speeds
u Data Workloads

18. Hadoop Applications

19. The Scale Out Applications
§ Data Ingest & Staging
§ Data Governance
§ Software development
platform
§ Analytics environment
§ Database/Data
Warehouse
§ Data Archiving
§ Video rendering & other
niche apps
The Data Lake involves just
the first two and does not
necessarily involve Hadoop

20. Data Lake, Refinery, Hub, in Overview
Think Logical, Implement Physical

21. The Data Lake Analytics Picture
Data Sources
Analytics
Service
Mgt
Life Cycle
Mgt
MetaData
Discovery
MDM
MetaData
Mgt
Data
Cleansing
Data
Lineage
R
O
U
N
D
|
U
P
W
R
A
N
G
L
I
N
G
Staging Area
(Hadoop)
Data Warehouse
or other location
Data Streams
ETL
ETL

22. How Data Gets to be Wrong
u Accidentally born wrong
u Deliberately born wrong
u Defective sensor/data
source
u Murdered (truncated,
overwritten)
u Corrupted in flight (rare)
u Corrupted by bad code
(surely not!)
u Corrupted by bad DBA

23. Data Governance
If data governance was important
before Big Data, (and it was) it is
far more important in the era of
Data Lakes

24. What Needs To Be Governed

25. Data Governance
Data Flows and Data Storage
Security & Access
Data cleansing and
transformation
Data meaning
Data provenance and lineage
Data archive and disposal
Availability and performance

26. Analytics Is a Process Not an Activity
q Data Analytics is a multi-
disciplinary end-to-end
process
q Until recently it was a
walled-garden. But the
walls were torn down by…
§ Data availability
§ Scalable technology
§ Open source tools
q It is now becoming an
integrated process
Data Governance is a process,
not an activity!!

27. The Global Map and Data Options
u Move the data to
the processing
u Move the
processing to the
data
u Move the
processing and the
data
u Shard
All network nodes can be data
creators, data stores and
processing points.

28. Logical Data Lakes
Soon we will be speaking of a
logical data lake and multiple
physical data lakes

29. 3
Events
and
Streams

30. Big Data, Event Data – The Data of Everything
WHAT
IS BIG
DATA?
Business
data
Traditional
data
Log file
data
Operational
data
Mobile data
Location
data Social
network
data
Public data
Commercial
databases
Streaming
data
Internet of
Things

31. A TRANSACTION is a
MOLECULE of ATOMIC EVENTS
The ATOM of data has
become the EVENT
Events: Atoms and Molecules

32. It’s Become and Event Based World

33. Events
Think of events as drops of water.
They can live in streams, and they
can also live in data pools and data
lakes.

34. Two Data Flows

35. The Traffic Cop (Events)

36. Event Types
q Instantiation Event
q A State Report
q A Trigger Event
q A Correction Event
We also need to consider:
Data Refinement
Aggregations
Homogeneous Collections
Derived Data

37. § The pulse and the
threshold alert
§ Some of this involves
distributed processing
§ There are known apps
and unknown apps, so
analytical exploration
needs to be enabled
§ Only aggregations will
migrate
DepotDepot
Central
Hub
Source
Proc.
Depot
Proc.
Central
Proc.
Sensors, controllers, CPUs
Data Data
Data
Event Based IoT Architecture

38. u Time
u Geographic location
u Virtual/logical location
u Source device
u Device ID
u Actors
u Ownership/
Provenance
u Values
Events and Event Data

39. Spark, Storm, Flink & Kafka
u Spark has dethroned Hadoop as a platform and
has momentum, both for microbatch and
streaming
u Storm provides batch and streaming (event
processing capabilities) concurrently via the
lambda architecture
u Flink was purpose built for streaming
u Kafka is the pipe
u Lambda and Zeta Architectures…

40. In Summary
1 Disturbance in the Force
2 What is a Data Lake,
exactly?
3 Streams and Events

42. Questions?

43. THANK
YOU!
FIND OUT MORE at
InsideAnalysis.com