Date: 16th November 2017 Location: Data-Driven LDN Theatre Time: 11:10 - 11:40 Speaker: Mike Ferguson Organisation: Intelligent Business Strategies

1. The Logical Data Warehouse
– A Modern Analytical Architecture for the Smart Business
Mike Ferguson
Managing Director
Intelligent Business Strategies
Big Data LDN
November, 2017

2. 2
Copyright © Intelligent Business Strategies 1992-2017
Topics
 New data, new analytical workloads and new data stores
 The impact of new data
 Why is the classic data warehouse architecture no longer enough?
 What is a Logical Data Warehouse?
 How to implement one using data virtualisation
 Transitioning from a traditional data warehouse to a logical data
warehouse
 Simplifying access for self-service and operational BI

3. 3
Copyright © Intelligent Business Strategies 1992-2017
For Many Years The Traditional Data Warehouse and BI
Environment Has Been Used For Analysis & Reporting
Operational
systems
web
P
o
r
t
a
l
Employees
Partners
Customers
BI
Tools
Platform
Data
Integration
Reports &
analytics
Data warehouse &
data marts
DW

4. 4
Copyright © Intelligent Business Strategies 1992-2017
Sales
Product line n
Product line 4
Product line 3
Product line 2
Product/
service line 1
Marketing
Service
Credit
Verification
HR
Finance
Planning
Procurement
SupplyChain
Suppliers
Front Office BackOffice
Operations
New Data Sources Have Emerged Inside And Outside
The Enterprise That Business Now Wants To Analyse
Data volume
Data variety
Number of sources
E.g. RFID tag
sensor
networks
Data volume
Data velocity
Customers
weather data

5. 5
Copyright © Intelligent Business Strategies 1992-2017
Popular Types of Data That Businesses Now Want
to Analyse
 Machine data
• Clickstream data, e-commerce logs
• IVR logs, App Server logs, DBMS logs
 Connected things (Sensor data, Consumer & Industrial IoT)
• Product usage behaviour data, product performance data
• Location, temperature, movement, vibration, liquid flow, pressure
 Semi-structured data e.g., JSON, BSON, XML
 Social networks data (often unstructured e.g. Text)
 Open government data
 Weather data
…In addition to transactions and master data in data warehouses

6. 6
Copyright © Intelligent Business Strategies 1992-2017
Sales Force
automation apps
Customer facing
bricks & mortar apps
Front-Office Operations
Customer
service apps
Customers
Need to improve
customer engagement
Digital channels are
generating big data
E-commerce
application
M-commerce
Mobile apps
Social commerce
applications
E-commerce
application
M-commerce
Mobile apps
Social commerce
applications
Application
/ System data
customer app
Customer interaction – the new way
- Need access to low latency
operational data
- Need customer intelligent computer
systems
Application/
System data
Customer interaction – the old way
customer employee UI
Why New Data? - Digital Channels Are Becoming The Focal
Point For Customer Interactions In The Digital Front-Office

7. 7
Copyright © Intelligent Business Strategies 1992-2017
Why New Data? - Companies Need To Better Understand Their
Customers In A Market Where Customers Hold All The Power
Also customers are much more
informed before they buy and
can churn on a single click.
This means loyalty is cheap
The “Device Generation”
Prospects and customers are now
interacting with applications and
not people and so there is very little
time to engage them

8. 8
Copyright © Intelligent Business Strategies 1992-2017
The Requirement Now Is To Capture, Integrate And Analyse More
Data For Deeper Customer Insights AND Do It Quickly
OMNI channel analysis – analyse all
customer interactions across all channels
identity
data
behavioural
data
(on-line,
location,
product usage)
social
data
Customer “DNA”
transactional
activity
Needs to be integrated in near real-time for
maximise competitive advantage

9. 9
Copyright © Intelligent Business Strategies 1992-2017
Why New Data? – Sensor Data For Improving Process
Efficiency, Optimisation And Cost
 Supply/distribution chain optimisation
 Asset management and field service optimisation
 Manufacturing production line optimisation
 Location based advertising (mobile phones)
 Grid health monitoring
• Electricity, water, cell phone network…
 Smart metering (collect data every 15 minutes)
 Sustainability analytics e.g. energy optimisation
 Security and authorisation systems
 Fraud
 Healthcare – ITC vital signs, fit bits,….
 Traffic optimisation
E.g. RFID tag

10. 10
Copyright © Intelligent Business Strategies 1992-2017
Data Deluge – An Explosion Of Data Sources With
Data Arriving Faster Than We Can Consume It
Data.Gov
Enterprise
How do we cope
with this data
deluge?

11. 11
Copyright © Intelligent Business Strategies 1992-2017
Data Deluge
– How Can We Put It All In The Data Warehouse?
Data.Gov
Enterprise
Data Warehouse
RDBMS
EDW
Unstructured?
Streaming?
Schema variant?
Huge volume?

12. 12
Copyright © Intelligent Business Strategies 1992-2017
Needs to scale
Challenges – The Classic DW Architecture Is Not A Great
Fit For Some New Data E.g. Streaming Data
ETL
/ DQ DW
marts
ETL
SCM
CRM
ERP
staging
??
push-based
visualisation
In-flight
DI / DQ
C
R
U
prod cust
asset
D
MDM
BI tool
streaming data
ETL
Or
Copy
Classic DW Architecture
Too many databases
- takes too long to respond
In-flight
analysis
Decision
engine
DW or
Hadoop
filtered data

13. 13
Copyright © Intelligent Business Strategies 1992-2017
Data Deluge
– What If We Just Connect Business Users To All Of It?
Data.Gov
Enterprise
Self-service BI
Business
Analyst
Do we really expect all
users to just figure out
how to integrate all
this data themselves?

14. 14
Copyright © Intelligent Business Strategies 1992-2017
Challenges – In Reality Data Is Being Ingested Into Multiple
Types Of Data Store Both On-Premises And In The Cloud
Enterprise
cloud
storage
I
D N
A G
T E
A S
T
Data.Gov
C
R
U
prod cust
asset
D
MDM
NoSQL
DBMS DW

15. 15
Copyright © Intelligent Business Strategies 1992-2017
New Data Has Taken Us Beyond The Traditional DW
– New Big Data Analytical Workloads
1. Analysis of high velocity data in motion
2. Complex analysis of large volumes of structured data
3. Exploratory analysis of un-modeled multi-structured data
4. Natural language processing e.g. sentiment analysis
5. Graph analysis e.g. social networks, fraud
6. Deep learning on unstructured images
7. Data warehouse optimisation – offload DW ELT processing

16. 16
Copyright © Intelligent Business Strategies 1992-2017
An Analytical Ecosystem Has Emerged With Different
Platforms Optimised For Different Analytical Workloads
Multiple platforms now being used for analytical processing
Streaming
data
Hadoop
data store
Data Warehouse
RDBMS
NoSQL
DBMS
EDW
DW & marts
NoSQL DB
e.g. graph DB
Advanced Analytic
(multi-structured data)
mart
DW
Appliance
Advanced Analytics
(structured data)
Analytical
RDBMS
Traditional
query,
reporting &
analysis
Data mining,
model
development
Streaming
analytics
Real-time
stream
processing &
decision m’gmt
Investigative
analysis,
Data refinery
Graph
analysis
C
R
U
D
Prod
Asset
Cust
MDM
Master data
management
Data mining,
model
development

17. 17
Copyright © Intelligent Business Strategies 1992-2017
Data Access Challenge – How Do You Improve Agility
When Data Is Becoming More Fractured?
Streaming
data
Hadoop
data store
Data Warehouse
RDBMS
NoSQL
DBMS
EDW
DW & marts
NoSQL DB
e.g. graph DB
Advanced Analytic
(multi-structured data)
mart
DW
Appliance
Advanced Analytics
(structured data)
Analytical
RDBMS
Streaming
analytics
C
R
U
D
Prod
Asset
Cust
MDM
 Accessing data in different
locations and different data
storage technologies is hard
 Need different APIs and query
languages to access data
 Data in different data structures
 How do I know what it means when
different data definitions for the same
data exist in different data stores?
 Multiple copies of data – which one?
 How do I know if I can trust it?

18. 18
Copyright © Intelligent Business Strategies 1992-2017
Challenges – Data Management And Integration In A
Multi-Platform Analytical Ecosystem Is Very Complex
EDW
Graph
DBMS
DW
Appliance
Analytical DBMS
MDM System
C
R
U
D
Prod
Asset
Cust
XML,
JSON
social
Web
logs
ERP
CRM
SCM
Ops
NoSQL DB
Column Fam DB
Document DB
NoSQL DB
web
Data martsTransaction data
Cloud data (not
shown) is also
part of it
insights
Txns

19. 19
Copyright © Intelligent Business Strategies 1992-2017
Siloed Data & Analytics - Different Tools to Manage and
Access Data For Each Type of Analytical and MDM Store
Analytical
tools
Data
management
tools
EDW
mart
Structured data
CRM ERP SCM
Silo
DW & marts
Analytical
tools/apps
Data
management
tools
Multi-structured
data
Silo
DW
Appliance
Advanced Analytics
(structured data)
Data
management
tools
Structured data
CRM ERP SCM
Analytical
tools
Silo
Analytical
tools/apps
Data
management
tools
NoSQL DB
e.g. graph DB
Silo
Multi-structured &
structured data
Silo
C
R
U
D
Prod
Asset
Cust
MDM
Applications
Data
management
tools
Master data
management
CRM ERP SCM

20. 20
Copyright © Intelligent Business Strategies 1992-2017
Simplifying Data Management Via A Logical Data Lake
- A Common Platform For Managing And Integrating Data
EDW
mart
DW & marts
DW
Appliance
Advanced Analytics
(structured data)
Integrated Self-Service and IT Data Management Tools (Business & IT)
NoSQL DB
e.g. graph DB
C
R
U
D
Prod
Asset
Cust
MDM
feedsIoT
XML,
JSON
RDBMS Files office docssocial Cloud
clickstream
web logs web
services
NoSQL
Information

21. 21
Copyright © Intelligent Business Strategies 1992-2017
Simplifying Data Access Via A Logical Data Warehouse
- Use Data Virtualisation To Access Data In Multiple Stores
EDW
mart
DW & marts
Analytical
tools &
applications
DW
Appliance
Advanced Analytics
(structured data)
Integrated Self-Service and IT Data Management Tools (Business & IT)
NoSQL DB
e.g. graph DB
C
R
U
D
Prod
Asset
Cust
MDM
Logical Data Warehouse (Data Virtualisation with a common vocabulary)
feedsIoT
XML,
JSON
RDBMS Files office docssocial Cloud
clickstream
web logs web
services
NoSQL
common
metadata
Information

22. 22
Copyright © Intelligent Business Strategies 1992-2017
BI Tools
What Is Data Virtualization?
- Critical Technology In A Logical Data Warehouse
web
content
RDBMS
BI Tools
portal
Data is integrated at run time and is delivered to tools, portals and applications
without the need for data staging
On-demand
data integration
Spread
sheets
Web
service
Data
Virtualization
Server
application
virtual data
sources
Big Data
DW
Enterprise
layer
consumer
data services

23. 23
Copyright © Intelligent Business Strategies 1992-2017
Moving To An Logical DW Requires DW Modernisation
- Data Virtualization on Top of A Data Vault DW
Data
Warehouse
Image Source: http://bukhantsov.org/2012/04/what-is-data-vault/
Applications/
BI Tools
Data
Virtualization
Server
mart mart mart
Applications/
BI ToolsApplications/
BI Tools
SQL X/Query
Possibly convert to a Data
Vault data modelSpeed up development
via Data Warehouse
Automation Software

24. 24
Copyright © Intelligent Business Strategies 1992-2017
Data Virtualisation – The Logical Data Warehouse
Reducing Time To Value Using Data Virtualization to
Create The Logical Data Warehouse
Self-Service
BI tool
EDW
DW & marts
NoSQL DB
e.g. graph DB
mart
DW
Appliance
Advanced Analytics
(structured data)
Advanced
Analytics
Streaming
data
RT Analytics
C
R
U
prod cust
asset
master data

25. 25
Copyright © Intelligent Business Strategies 1992-2017
Reducing Time To Value Using Data Virtualization to
Create The Logical DW on A Logical Data Lake
EDW
DW & marts
NoSQL DB
e.g. graph DB
mart
DW
Appliance
Advanced Analytics
(structured data)
Advanced
Analytics
Streaming
data
RT Analytics
C
R
U
prod cust
asset
master data
DataVirtualisation
Logical Data Lake
Logical DWmart mart mart
Self-service BI
Smart
mobile
app
Reports
Consumer tables
Trusted logical
data lake tables
With acknowledgements to Rick van der Lans
Data Science

26. 26
Copyright © Intelligent Business Strategies 1992-2017
LDW Business Impact – Better Customer Segmentation
From On-Demand Integrated Customer Insight
EDW
DW & marts
NoSQL DB
e.g. graph DB
mart
DW
Appliance
Advanced Analytics
(structured data)
Advanced
Analytics
Streaming
data
RT Analytics
C
R
U
prod cust
asset
master data
Customer sentiment,
interactions,
online behaviour,
& new data
Customer
relationships*,
social network
influencers
Customer real-
time location,
product usage &
on-line behaviour
Customer
master data
Customer
purchase activity
& transaction
history
Customer predictive
analytical model
development
Customers
Improve customer
engagement
Analytics (Clustering)
Customer Segmentation
Now we can use rich integrated customer insight and analytics to more accurately
segment the customer base
Data
Virtualisation
Software The Logical Data Warehouse - Integrated 360o Customer Insight

27. 27
Copyright © Intelligent Business Strategies 1992-2017
LDW Business Impact
- Integrated Data and Deep Insights For Better Forecasting
EDW
DW & marts
NoSQL DB
e.g. graph DB
mart
DW
Appliance
Advanced Analytics
(structured data)
Advanced
Analytics
Streaming
data
C
R
U
prod cust
asset
master data
Customer sentiment,
interactions,
online behaviour,
& new data
Customer
relationships*,
social network
influencers
Customer real-
time location,
product usage &
on-line behaviour
Customer
master data
Customer
purchase activity
& transaction
history
Customer predictive
analytical model
development
Data
Virtualisation
Software
Forecast
The Logical Data Warehouse - Integrated 360o Customer Insight

28. 28
Copyright © Intelligent Business Strategies 1992-2017
LDW Business Impact - Smart Revenue Management
Using Consistent Dynamic Pricing Across All Channels
The Logical Data Warehouse - Integrated 360o Customer Insight
EDW
DW & marts
NoSQL DB
e.g. graph DB
mart
DW
Appliance
Advanced Analytics
(structured data)
Advanced
Analytics
Streaming
data
C
R
U
prod cust
asset
master data
Customer sentiment,
interactions,
online behaviour,
& new data
Customer
relationships*,
social network
influencers
Customer
master data
Customer
purchase activity
& transaction
history
Customer predictive
analytical model
development
Pricing
recommendation
service
Prescriptive
analytics
Demand driven
dynamic pricing
+ cross sell / upsell
Customer insights
Data
Virtualisation
Software
Sales Force
automation apps
Customer facing
bricks & mortar apps
Front-Office Operations
Customer
service apps
E-commerce
application
M-commerce
Mobile apps
Social commerce
applications
Customer real-
time location,
product usage &
on-line behaviour

29. 29
Copyright © Intelligent Business Strategies 1992-2017
The Logical Data Warehouse - Integrated Customer Insight
LDW Business Impact? - On-Demand Integrated Customer
Insight As A Service In All Front Office Channels
EDW
DW & marts
NoSQL DB
e.g. graph DB
mart
DW
Appliance
Advanced Analytics
(structured data)
Advanced
Analytics
Streaming
data
RT Analytics
C
R
U
prod cust
asset
master data
Customer sentiment,
interactions,
online behaviour,
& new data
Customer
relationships*,
social network
influencers
Customer real-
time location,
product usage &
on-line behaviour
Customer
master data
Customer
purchase activity
& transaction
history
Customer predictive
analytical model
development
Sales Force
automation apps
Customer facing
bricks & mortar apps
Front-Office Operations
Customer
service apps
Customers
Improve customer
engagement
E-commerce
application
M-commerce
Mobile apps
Social commerce
applications
Digital channels are
generating big data
e.g. In-store apps
In-branch apps

30. 30
Copyright © Intelligent Business Strategies 1992-2017
Conclusions
 The physical data warehouse is no longer enough
 The impact of new data on analytical systems
• We now have an analytical ecosystem consisting of multiple data
stores optimised for different analytical workloads
• Data management is much more complex and challenging
• Siloes of data and analytics exist and total cost is too high
• Data access is hard and users are forced to have to integrate data
 The answer is rationalisation with a need for
• A data management platform and an information catalogue to
manage a logical data lake of multiple data stores
• Data warehouse modernisation with data vault and virtual marts
• A logical data warehouse for common semantic layer, simplified
access to data, and data available as a service
 Result is plug & play data services for competitive advantage

31. 31
Copyright © Intelligent Business Strategies 1992-2017
Thank You!
Thank You!
Mike Ferguson is Managing Director of Intelligent Business Strategies
Limited. As an independent analyst and consultant he specializes in business
intelligence, analytics, data management and big data. With over 35 years of
IT experience, Mike has consulted for dozens of companies, spoken at
events all over the world and written numerous articles. Formerly he was a
principal and co-founder of Codd and Date Europe Limited – the inventors of
the Relational Model, a Chief Architect at Teradata on the Teradata DBMS
and European Managing Director of DataBase Associates.
www.intelligentbusiness.biz
mferguson@intelligentbusiness.biz
@mikeferguson1
(+44)1625 520700
About Mike Ferguson