Data Con LA 2020 Description The data warehouse has been an analytics workhorse for decades. Unprecedented volumes of data, new types of data, and the need for advanced analyses like machine learning brought on the age of the data lake. But Hadoop by itself doesn't really live up to the hype. Now, many companies have a data lake, a data warehouse, or a mishmash of both, possibly combined with a mandate to go to the cloud. The end result can be a sprawling mess, a lot of duplicated effort, a lot of missed opportunities, a lot of projects that never made it into production, and a lot of financial investment without return. Technical and spiritual unification of the two opposed camps can make a powerful impact on the effectiveness of analytics for the business overall. Over time, different organizations with massive IoT workloads have found practical ways to bridge the artificial gap between these two data management strategies. Look under the hood at how companies have gotten IoT ML projects working, and how their data architectures have changed over time. Learn about new architectures that successfully supply the needs of both business analysts and data scientists. Get a peek at the future. In this area, no one likes surprises. *Look at successful data architectures from companies like Philips, Anritsu, Uber, *Learn to eliminate duplication of effort between data science and BI data engineering teams *Avoid some of the traps that have caused so many big data analytics implementations to fail *Get AI and ML projects into production where they have real impact, without bogging down essential BI *Study analytics architectures that work, why and how they work, and where they're going from here Speaker Paige Roberts,Vertica, Open Source Relations Manager

1. CombineStrengthsofDataLakeandDataWarehouse
UNIFY ANALYTICS:
PaigeRoberts
VerticaOpenSourceRelationsManager

2. A Day in the Analytics Life
Applications and services that enable our data-driven world use both BI and data science.
Operations Optimization

3. Analytics Database
SQL
BATCH
Transactional data
Files
Application Data
Customer
Operational
Financial
Extract, Transform,
Load (ETL)
CRMERP
Billing
Reporting
Ad Hoc
Queries
Business Intelligence
BUSINESS
INTELLIGENCE
REPORTS,
VISUALIZATION
Data Warehouse Architecture

4. Data Warehouse Strengths
Reporting/
Business Intelligence
High Performance
High Concurrency
Reliability
Security
Governance
SQL
DATA
WAREHOUSE

5. Data Warehouse Strengths
Reporting/
Business Intelligence
High Performance
High Concurrency
Reliability
Security
Governance
SQL
BI Visualization tools
DATA
WAREHOUSE
Expensive to scale
Structured data only
ETL caused data to be stale
Business intelligence only
Can’t handle streaming data
WEAKNESSES

6. Mass Storage
(Data Lake)
HDFS
LOW LATENCY
Application data
Web clicks
Logs
Sensors
Operational metrics
User tracking
Geo-location
Data Lake Architecture
Distributed
Pub/Sub
Stream
Processing
ETL or ELT with
Transformation
in data lake
Object
Storage
AND / OR
Data Prep
Prepared
data in new
format
DATA SCIENCE
VISUALIZATION,
APPLICATIONS
SQL-Like
Query
Engine
Machine
Learning
Transactional data
Application Data
OLTP/ODS
BATCH
Contextual data
Weather
Geo
Files

7. Data Lake Strengths
DATA LAKE
Machine Learning/
Data Science
Unlimited Scale
Streaming Data
Semi-Structured Data
(JSON, AVRO, …)
Complex Data Types
(Maps, Structs, Arrays)
Schema on Read
Python, R, Jupyter

8. Data Lake Strengths
DATA LAKE
Machine Learning/
Data Science
Unlimited Scale
Streaming Data
Semi-Structured Data
(JSON, AVRO, …)
Complex Data Types
(Maps, Structs, Arrays)
Schema on Read
Python, R, Jupyter
Slow performance
Poor concurrency
Complex to build / maintain
Immature security, governance
WEAKNESSES

9. Mass Storage
(Data Lake)
HDFS
LOW LATENCY
Application data
Web clicks
Logs
Sensors
Operational metrics
User tracking
Geo-location
Cooperative Architecture
Distributed
Pub/Sub
Stream
Processing
Object
Storage
AND / OR
Data Prep
Cold data,
historical
data
Machine
Learning
Distributed
Analytics
Database
(Data Warehouse)
BUSINESS
INTELLIGENCE
DATA SCIENCE
+
Import,
Export,
Query
SQL
ETL or ELT with
Transformation
in data lake
Transactional data
Application Data
OLTP/ODS
BATCH
Contextual data
Weather
Geo
Files

10. Cooperative architecture strengths
DATA LAKE
Reporting/
Business Intelligence
High Performance
High Concurrency
Reliability
Security
Governance
SQL
BI Visualization tools
Machine Learning/
Data Science
Unlimited Scale
Streaming Data
Semi-Structured Data
(JSON, AVRO, …)
Complex Data Types
(Maps, Structs, Arrays)
Schema on Read
Python, R, Jupyter
DATA
WAREHOUSE
Complexity
Division of BI and Data Science
Focused on data location/format
WEAKNESSES

11. SQL
Transactional
databases
MySQL,
PostgreSQL
DATA SCIENCE
BUSINESS
INTELLIGENCE
+
BATCH
Cassandra
key value
database
Mass Storage
(Data Lake)
HDFS
LOW LATENCY
Applications data
Clickstreams
Location data
Schema
Enforced
Flattened,
modeled tables
DataManager/Manifest
DatabaseProxy
Ad Hoc Analytics:
• CityOps
• Data Scientists
• QueryBuilder –
Uber created
• DashBuilder –
Uber created
Applications:
• ETL/Modeling
• CityOps
• Machine Learning
• Experiments
Duplicate of
most used data
Infrequently
used data
Recent data
Cooperative Example 1:

12. LOW LATENCY
BATCH
SQLMass Storage
(Data Lake)
Third-party data
HDFS
EXTREMELY
LOW LATENCY
Real-time
bidding data
Contextual
data -
PostgreSQL
DATA SCIENCE
BUSINESS
INTELLIGENCE
+
BUSINESS
INTELLIGENCE
BUSINESS
INTELLIGENCE
SQL
SQL
Cooperative Example 2:

13. Unified Analytics Warehouse Architecture
BUSINESS
INTELLIGENCE
DATA SCIENCE
Model Evaluation
and Management
STREAM PROCESSING
LOW LATENCY
Application data
Web clicks
Logs
Sensors
Operational metrics
User tracking
Geo-location
STREAMING DATA
BATCH
Files
Weather
Geo
CONTEXTUAL DATA
Application Data
OLTP/ODS
TRANSACTIONAL
DATA
Batch ETL
OR
EL with T
ON-PREMISES, HYBRID, CLOUD OR MULTI-CLOUD
Object
Storage
HDFS
COMPLETED IN
WAREHOUSE
GCP
SQL
Isolated Workloads
Ingestion, ELT,
Data Prep
OR
+
Machine
Learning
Reporting
Ad Hoc
Queries
Unified Analytics Warehouse

14. Unified Analytics Warehouse strengths
DATA LAKE
Reporting/
Business Intelligence
High Performance
High Concurrency
Reliability
Security
Governance
Machine Learning/
Data Science
Unlimited Scale
Streaming Data
Semi-Structured Data
(JSON, AVRO, …)
Complex Data Types
(Maps, Structs, Arrays)
Schema on Read
DATA
WAREHOUSE
Unified
Analytics
SQL
BI Visualization tools
Python
Jupyter
R

15. SQL
DATA SCIENCE
BUSINESS
INTELLIGENCE
+
BATCH
LOW LATENCY
EON TV Service
On/Off, Channel
Change Data
Contact Center
Geo/Mapping
Customer
CX
Operational
Financial
Service Quality
TV Schedule
Extract, Load
CRM
ERP
Billing
Ingest,
Transform
Transformation
pushed down
Machine
Learning
Real-time ad
targeting
Customer profile
analysis for real-
time apps
Financial Reporting
Data-driven Apps
Reporting
Content Analytics
Reporting
Ad Hoc
Queries
UAW Example 1:

16. SQL
DATA SCIENCE
BUSINESS
INTELLIGENCE
+
Third-party data
EXTREMELY
LOW LATENCY
Real-time
bidding data
Contextual
data
BATCH
LOW LATENCY
Mass Storage
(Data Lake)
Data Ingest / ELT/
BI / Machine
Learning
Primary Sub-Cluster
Reporting
Ephemeral Sub-Clusters
Reporting
UAW Example 2:

17. The Only Constant is Change
18

18. New EMA White Paper:
The Race for a Unified Analytics Warehouse
Request your copy today:
www.vertica.com/resource/the-race-for-a-
unified-analytics-warehouse/
Learn More: academy.vertica.com
Try Vertica Free: vertica.com/try

19. Q&ALearn More: academy.vertica.com
Try it Free: vertica.com/try
Paige Roberts
Open Source Relations Manager
Paige.Roberts@vertica.com

20. Thank You!

21. Unified Analytics Warehouse
BUSINESS
INTELLIGENCE
DATA SCIENCE
Model Evaluation
and Management
STREAM PROCESSING
LOW LATENCY
Application data
Web clicks
Logs
Sensors
Operational metrics
User tracking
Geo-location
STREAMING DATA
BATCH
Files
Weather
Geo
CONTEXTUAL DATA
Application Data
OLTP/ODS
TRANSACTIONAL DATA
Batch ETL
OR
EL with T
ON-PREMISES, HYBRID, CLOUD OR MULTI-CLOUD
Object
Storage
HDFS
COMPLETED IN
WAREHOUSE
GCP
SQL
Isolated Workloads
Ingestion, ELT,
Data Prep
OR
+
Machine
Learning
Reporting
Ad Hoc
Queries

23. http://academy.vertica.com
https://www.vertica.com/data-disruptors-vertica-webcast-series/
Vertica Data Disruptors Webcast Series

24. Vertica Earns Top Leader Position in GigaOm’s Radar for
Evaluating Data Warehouse Platforms
Get the complimentary report today at:
https://www.vertica.com/resource/vertic
a-earns-top-position-in-gigaoms-radar-for-
evaluating-data-warehouse-platforms/

25. Modernized Data
Warehouse for IoT & More
Unified
Analytics for
Disparate
Data Stores
Operationalized Machine
Learning
Multi-Cloud,
On-Premise,
Hybrid, &
Embedded
Deployments
Vertica

27.  Eliminate overhead of data transfer
 Keep data secure with clear provenance
 Store and manage models and data together
 Serve hundreds of concurrent users
 Use highly scalable, high-performance
machine learning functionalities
 Avoid maintenance cost of a separate
analytical system
 Increase productivity with simple SQL calls
instead of coding everything
 Prep data faster
 Seamless Python and PMML integration
Advantages of Vertica’s in-database machine learning

28. Supporting the Entire Machine Learning Process
Business
Understanding
Data Analysis
&
Understanding
Data
Preparation Modeling Evaluation Deployment
Artificial
Intelligence
Speed
ANSI SQL
Scalability
Massively
Parallel
Processing
Deploy
Anywhere
Outer
Detection
Normalization
Imbalanced
Data
Processing
Sampling
Missing Value
Imputation
And More…
SVM
Random
Forests
Logistic
Regression
Linear
Regression
Ridge
Regression
Naive Bayes
Cross
Validation
And More…
Model-level
Stats
ROC Tables
Error Rate
Lift Table
Confusion
Matrix
R-Squared
MSE
In-Database
Scoring
Speed
Scale
Security
Pattern
Matching
Date/
Time Algebra
Window/
Partition
Date Type
Handling
Sequences
And More…
Sessionize
Time Series
Statistical
Summary
SQL SQLSQL SQLSQL
UNIFIED PLATFORM

29. 14 million
10 million
50 million
Vertica analyzes 10 billion
embedded RTMS timers per
month in milliseconds.
50 billion
Performance Scale
POWERED BY
Industry Leaders are Betting… & Winning… with Vertica

30. Predictive Maintenance Demo
31
Analyze sensor
data from cooling
towers across the
US , enabling
equipment
manufacturers to
predict and
prevent equipment
failure

31. 32
Vertica operates
at the “edge”
with flight track
detail. Sensor
data is collected
using a Raspberry
pi with radio
receiver and
antenna. Data is
loaded into
Vertica as
thousands of
records per
second and
builds to billions
of flight data
points collected
within a 250-mile
radius.
https://www.vertica.com/blog/blog-post-series-using-vertica-track-
commercial-aircraft-near-real-time/
Flight Tracker Demo

32. 33
Need to analyze over 1.5 PB of
customer, product sensor, and
performance metadata to fine-tune
leading-edge product development
 Embeds Vertica in InfoSight flash
storage software platform and
leverages predictive analytics to
deliver advanced customer insights
and 99.9999% availability
 Analyzes millions of sensors every
second to prevent problems
Predicting and solving
86% of customer issues
automatically

33. 34
Goal: Faster decisions and analytical insight on manufacturing
plant assets – but …
 Data is doubling every two years, and analyzing multiple ‘hot’
and ‘cold data streams too time consuming and ineffective in
environment where production quality issues are extremely
costly
Solution: Develop and deploy analytic models into
customer supply chains
 Collect, organize, store and process data from 50 billion
semiconductors and printed circuit boards / year
 Create a holistic view across both electronic system and
semiconductor component data for quick identification of the
root cause of a defect
 Analytics on 2 billion data points in < 1 minute
Analyzing quality and yield in
semi-conductor and
electronics manufacturing

34. 35
 Philips is moving from reactive to data-
driven, proactive maintenance, utilizing
new sources of sensor data along with
machine learning models to enable
scheduled, predictable and non-intrusive
service actions
 Collects and processes data from devices
and medical imaging systems using remote
monitoring and predictive analytics
Aiming for zero unplanned
equipment downtime with
IoT analytics

35. Anritsu provides products and services for the development,
manufacturing and maintenance of a range of communication
systems for mobile phones and internet connectivity. Anritsu is now
leveraging its experience with service assurance technologies to
develop solutions for 5G, M2M, IoT, and other wireline and wireless
communication markets.
Goal
▪ Deploy new customers faster to increase customer
satisfaction
▪ Deal with skyrocketing data volumes, while keeping
costs down and allowing the flexibility to scale both
immediately and in the future
▪ Increase employee productivity
Result
▪ Return on investment of 351%; payback period of 4
months; average annual benefit: $3,014,583
36
Anritsu Handles Large Scale
Data Input with Faster
Response Time
Faster performance on complex analytics