William McKnight's presentation discusses the role of data lakes in modern data architecture, emphasizing their importance in facilitating digital transformation and analytics. He outlines the decisions organizations must make regarding data storage types, placement, and workload architecture, as well as the impact of cloud databases on data warehousing. The presentation advocates for a strategic approach to integrating data lakes, highlighting their use in AI/ML and the future of analytical frameworks.

1. When and How Data
Lakes Fit into a Modern
Data Architecture
Presented by: William McKnight
“#1 Global Influencer in Data Warehousing” Onalytica
President, McKnight Consulting Group
An Inc. 5000 Company in 2018 and 2017
@williammcknight
www.mcknightcg.com
(214) 514-1444
Second Thursday of Every Month, at 2:00 ET
#AdvAnalytics

2. From Data Lakes
to Data Experiences
Joel McKelvey, Director, Product Marketing

3. 1
https://emtemp.gcom.cloud/ngw/globalassets/en/information-technology/documents/trends/gartner-2019-cio-agenda-key-takeaways.pdf
Digital-fueled Growth is the Top
Investment Priority For Technology Leaders.1
Rebalance your technology portfolio toward digital transformation
Percent of respondents
increasing investment
Percent of respondents
decreasing investment
Cyber/information security 40%1%
Cloud services or solutions (Saas, Paa5, etc.) 33%2%
Core system improvements/transformation 31%10%
How to implement product-centric delivery by percentage of respondents
DigitalTransformation
Business Intelligence or data analytics solution 45%1%

4. 1
https://www.forrester.com/report/InsightsDriven+Businesses+Set+The+Pace+For+Global+Growth/-/E-RES130848
Insights-driven businesses harness
and implement digital insights
strategically and at scale to drive growth
and create differentiating experiences,
products, and services.1
7x Faster growth than global GDP
30% Growth or more using advanced analytics in a transformational way
2.3x More likely to succeed during disruption

5. Looker Data Platform
Governed metrics | Best-in-class APIs | In-database | Git version-control | Security | Cloud
Integrated Insights
Sales reps at
Slack have the
metrics their
customers care
about most within
a pre-populated
slide deck
Contextual | Passive | Where you work
Sales reps have
more context on
customer calls
with valuable
usage data
embedded within
Salesforce
Data-driven Workflows
Operational | Time-sensitive | Task-driven
Reduce churn
with automated
alerts and email
follow ups for
success managers
based on
customer health
Increase ROI on
digital ad spend
by optimizing bids
in real-time with
ML ‘bid-bot’,
trained with
governed data
Custom Applications
Job to be done | Larger purpose
Custom
application
ensures ads are
sold for the
optimal price,
regardless of time
slot or market
Top Broadcaster
Maintain optimal
inventory levels
and pricing with
merchandising
and supply chain
management
application
Top Retailer
Modern BI & Analytics
Analytical | Exploreable | Data-centric
Namely customers
access reports
and dashboards
to better
understand their
staffing needs
and trends
Holistic
understanding of
customers with a
360-degree view
across channels:
web, apps, print,
and more
Data Lake

6. 1 in 2
customers integrate
insights/experiences
beyond Looker
2000+
Customers
5000+
Developers
900+
Employees
Santa Cruz
San Francisco New YorkChicago
Boulder Tokyo
Dublin London
Empower
people with
the smarter
use of data

7. Looker Recognized as Challenger in the Gartner 2020 Magic
Quadrant for Analytics and Business Intelligence Platforms
“The growing demand for tools that close
the gap between discovering insights and
taking action is creating a profound change
in the way we use data in the workplace. At
Looker our vision is to meet this demand by
enabling data experiences that go far
beyond traditional business intelligence.”
- Nick Caldwell,
Chief Product Officer at Looker
1 Gartner “Magic Quadrant for Analytics and Business Intelligence Platforms,” by James Richardson, Rita Sallam, Kurt Schlegel, Austin Kronz, and Julian Sun February 13, 2020
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or
other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties,
expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

8. William McKnight
President, McKnight Consulting Group
• Frequent keynote speaker and trainer internationally
• Consulted to Pfizer, Scotiabank, Fidelity, TD Ameritrade, Teva
Pharmaceuticals, Verizon, and many other Global 1000 companies
• Hundreds of articles, blogs, benchmarks and white papers in
publication
• Focused on delivering business value and solving business problems
utilizing proven, streamlined approaches to information management
• Former Database Engineer, Fortune 50 Information Technology
executive and Ernst&Young Entrepreneur of Year Finalist
• Owner/consultant: 2018 & 2017 Inc. 5000 Data Strategy and
Implementation consulting firm
• Brings 25+ years of information management and DBMS experience

9. McKnight Consulting Group Offerings
Strategy
Training
Strategy
§ Trusted Advisor
§ Action Plans
§ Roadmaps
§ Tool Selections
§ Program Management
Training
§ Classes
§ Workshops
Implementation
§ Data/Data Warehousing/Business
Intelligence/Analytics
§ Master Data Management
§ Governance/Quality
§ Big Data
Implementation
3

10. Analytic Data Stores

11. 3 Major Decisions
• Decision #1: The Data Store Type
– The largest factor for distinguishing between databases and file-based scale-out system
utilization is the data profile. The latter is best for data that fits the loose label of 'unstructured'
(or semi-structured) data, while more traditional data -- and smaller volumes of all data -- still
belong in a relational database.
• Decision #2: Data Store Placement
– You must also decide where to place your data store -- on-premises or in the cloud (and which
cloud). In the past, the only clear choice for most organizations was on-premises data. However,
the costs of scale are gnawing away at the notion that this remains the best approach for a data
platform. For more on why databases are moving to the cloud, please read this article.
• Decision #3: The Workload Architecture
– Finally, you must keep in mind the distinction between operational or analytical workloads.
Short transactional requests and more complex (often longer) analytics requests demand
different architectures. Analytics databases, though quite diverse, are the preferred platforms
for the analytics workload.
5

12. Whither the idea of the Data Warehouse?
Intake
Export
Files
Txn
App
Data
Full
Delta
Stream
Structured
Big Data
TIER 1
Access1..n
Regional and
Departmental
Views
ADS
Applications
& Engines
Operational
Analytics &
Hot Views
Data Marts
Independent
Dependent
Relational
Data
TIER 3
Conformed
Dimensions
Distribution
Common Summary
and Derived Values
Master Data
Reference Data Hub
Transaction
Data Hub
TIER 2
6

13. Data Warehousing
• Data Warehouses (still) have a
lower total cost of ownership than
data marts
• A data warehouse is a SHARED
platform
– Build once, use many
– Access at Data Warehouse
– Access by creating a mart off the DW
• Still A LOT cheaper than building from
scratch
“… a subject-
oriented, integrated,
non-volatile, time-
variant collection of
data, organized to
support
management
needs.” — Bill Inmon

14. Reasons for Analytic Architecture Change
• Take Advantage Of…
– Cloud Databases
– Get into a Columnar Data Orientation
– Get into the Data Architecture you want
– Cloud Storage
• Projects Requiring Consolidated Data
8

15. The Key is Right-Fitting Platforms
• THE Data Warehouse
– Value-Added Components: Modeling for Access,
Data Quality, Tooling, Conformed Dimensions,
Data Governance, Etc.
• A Dependent Data Mart (Fed from the Data
Warehouse)
• A Data Lake
• A Big Data Cluster
• An Independent Data Mart
• An Operational Hub
• An Operational Data Lake
9

16. Data
Lake
Usage Understanding by the Builders
D
a
t
a
C
u
l
t
i
v
a
t
i
o
n
Data
Warehouse
Data
Mart
Sensible Divisions of Analytic Platforms

17. The Post-Operational Ecosystem
Data Lake
DW
DM
DM
11

18. Usage Understanding by the Builders
D
a
t
a
C
u
l
t
i
v
a
t
i
o
n
Data
Warehouse
/Lake
What If?
Data
Mart

19. Deploying the Data Lake

20. Data Lake
Data Scientist Workbench and Data
Warehouse Staging
OLTP
Systems
Data Lake
Data Scientists
ERP
CRM
Supply
Chain
MDM
…
Data
Warehouse
Data Mart
Stream or
Batch
Updates
DI
Real-Time,
Event-Driven
Apps
14

21. Data Lake Patterns
• Data Refinery
– Do Data Warehouse ETL in the Data Lake
• Archive Storage
• Data Science Lab
• [Data Lake as the Data Warehouse]
15

22. Files
RDBMS
Streaming
Data
Sources
Ingest
Governance
Process
Central Data Store
Kafka, Pulsar
Snowball
Kinesis
QuickSight
HadoopCloud Storage
EMR
Glue
Catalog & User Interface Access Management
DynamoDB ElasticSearch Web
Interface
API Gateway IAM & Cognito
Analyze
Python
R
Machine
Learning
Data Lake Example Components
16

23. Data Lake Setup
• Managed deployments in the Hadoop
family of products
• External tables in Hive metastore that point
at cloud storage (Amazon S3, Google
Cloud Storage, Azure Data Lake Storage
Gen 2)
– To run SQL against the data
– HiveQL and Spark SQL require entries in the
metastore
17

24. Object Storage Instances
• Object Storage instances/clusters have local
storage, i.e., on the physical drives mounted to
the instances themselves, that is HDFS and
Hive
• Object Storage technologies access their
cloud vendor’s respective cloud storage—viz.:
– Amazon EMR accesses S3
– Dataproc accesses Google Cloud Storage
– HDI accesses Azure Data Lake Storage Gen2
• Local storage is used by the Object Storage
platform for housekeeping
18

25. The Data Warehouse of the Future
• Pair a lake with an analytical engine that
charges only by what you use
• If you have a ton of data that can sit in cold
storage and only needs to be accessed or
analyzed occasionally, store it in Amazon
S3/Azure Blob Storage/Google Cloud Storage
– Use a database (on-premise or in the cloud) that
can create external tables that point at the storage
– Analysts can query directly against it, or draw down
a subset for some deeper/intensive analysis
– The GB/month storage fee plus data
transfer/egress fees will be much cheaper than
leaving it in a data warehouse
19

26. Notes on the Data Warehouse of the Future
• More Achievable separate compute and storage architecture
• Compute resources (Map/Reduce, Hive, Spark, etc.) can be
taken down, scaled up or out, or interchanged without data
movement
• Storage can be centralized, but compute can be distributed
• Major players have mechanism to ensure consistency to achieve
ACID-like compliance
• Remote data replication to ensure redundancy and recovery
• Most of the query execution is processing time, and not data
transport, so if cloud compute and storage are in the same
cloud vendor region, performance is hardly impacted
20

27. Sample Cluster Configuration
Google BigQuery
Cloud Provider Google Cloud
Platform Version 3.6
Hadoop Version 2.7.3
Hive Version 1.2.1
Spark Version 2.3.2
Instance Type n1-highmem-16
Head/Master Nodes 1
Worker Nodes 16 and 32
vCPUs (per node) 16
RAM (per node) 104 GB
Compute Cost
(per node per hour)
$0.947
Platform Premium (per node per hour) $0.160
21

28. Tips
• If possible, configure remote data to be stored in parquet format, as
opposed to comma-separated or other text format
• As new data sources are added to cloud storage, use a code
distribution system—like Github—to distribute new table definitions
to distributed teams
• Use data partitioning to improve performance—but don’t forget new
partitions have to be declared to the Hive metastore when they are
added to the data
• Co-locate compute and storage in the same region
• Use AES-256 encryption on cloud storage bucket to ensure encryption
at-rest
• Hold the remotely-stored data to the same governance and data
quality standards you would if it were on-premise—consider a data
catalog or other metadata technique to keep the data organized and
easy-to-find for new compute engines
• Drop commonly used data in the lake, like master data from MDM
22

29. The Data Science Lab Role of
the Data Lake

30. Artificial Intelligence and Machine Learning
• Looming on the horizon is an injection of
AI/ML into every piece of software
• Consider the domain of data integration
– Predicting with high accuracy the steps ahead
– Fixing its bugs
• Machine learning is being built into databases
so the data will be analyzed as it is loaded
– I.e., Python with TensorFlow and Scala on Spark.
• The split of the necessary AI/ML between the
"edge" of corporate users and the software
itself is still to be determined
24

31. Training Data for Machine Learning &
Artificial Intelligence
• You must have enough data to analyze to
build models
• Your data determines the depth of AI you
can achieve -- for example, statistical
modeling, machine learning, or deep
learning -- and its accuracy
25

32. AI Data
• Call center recordings and chat logs
• Streaming sensor data, historical maintenance records and
search logs
• Customer account data and purchase history
• Email response metrics
• Product catalogs and data sheets
• Public references
• YouTube video content audio tracks
• User website behaviors
• Sentiment analysis, user-generated content, social graph
data, and other external data sources
26

33. When and How Data
Lakes Fit into a Modern
Data Architecture
Presented by: William McKnight
“#1 Global Influencer in Data Warehousing” Onalytica
President, McKnight Consulting Group
An Inc. 5000 Company in 2018 and 2017
@williammcknight
www.mcknightcg.com
(214) 514-1444
Second Thursday of Every Month, at 2:00 ET
#AdvAnalytics