Big Data LDN 2018: DATA MANAGEMENT AUTOMATION AND THE INFORMATION SUPPLY CHAIN – FROM DATA LAKE TO DATA MARKETPLACE

Mike Ferguson
Managing Director
Intelligent Business Strategies
Big Data LDN
London, November, 2018
Data Management Automation & The Information Supply Chain
- From Data Lake to Data Marketplace

2Copyright © Intelligent Business Strategies 1992-2018
Topics
§ Digital transformation
§ The thirst for new data
§ Data complexity and challenges caused by digital transformation
§ A siloed approach to managing and governing data
§ Conquering complexity, governing data and accelerating time to value
§ Data lake, information supply chain and data marketplace
§ Simplifying access to data
§ Conclusions

Sales
Product line n
Product line 4
Product line 3
Product line 2
Product/servi
ce line 1
Marketing
Service
Credit
Verification
HR
Finance
Planning
Procurement
SupplyChain
Front Office BackOffice
OperationsShipping
Digital Transformation
– Cloud, Mobile, Social, IoT And B2B Process Integration
Suppliers
Customers,
Partners
Employees
Things
Things(Smartproducts)
Sales
Product
line n
Product
line 4
Product
line 3
Product
line 2
Product/s
ervice
line 1
Market
ing
Servic
e
Credit
Verific
ation
HR
Financ
e
Plann
ing
Procure
ment
SupplyChain
Front
Office
BackOf
fice
Operatio
ns
Shippi
ng
B2B process integration

Customers
Need to improve
customer engagement
Digital channels are
generating big data
Application
/ System data
customer app
Customer interaction – the new way
- Need customer intelligent
applications
- Need access to low latency data
- Need to respond quickly as there is
very little time to engage
Application/
System data
Customer interaction – the old way
customer employee UI
Digital Channels Have Already Become The Focal Point For Customer
Interactions In The Digital Front-Office
Sales Force
automation apps
Customer facing
bricks & mortar apps
Front-Office Operations
Customer
service apps
E-commerce
application
M-commerce
Mobile apps
Social commerce
applications
E-commerce
application
M-commerce
Mobile apps
Social commerce
applications

The Requirement Now Is To Capture, Integrate And Analyse More
Data For Deeper Customer Insights To Drive Value
OMNI channel analysis – analyse all
customer interactions across all channels
identity
data
(master
data)
behavioural
data
(on-line,
location,
product usage)
social data
(opinion,
relationships)
Customer “DNA”
transactional
activity
(channel
purchases)
Needs to be integrated in near real-time to
maximise competitive advantage

The Demand For Data
– Popular Types of Data That Businesses Now Want to Analyse
Type Of Data Examples Uses
Traditional
structured data
• Master data
• Transaction data
• Customer, product, employee, supplier, site,…..
• Orders, shipments, returns, payments, adjustments..
Machine
generated data
• Clickstream web server logs
• IVR logs, App Server logs
• DBMS logs
• On-line behaviour analysis
• Cyber security
• Consumer IoT (Sensor data)
• Industrial IoT (Sensor data)
• Location, temperature, movement,
vibration, pressure
• Product usage behaviour
• Product or equipment performance
Human
generated data
• Social network data
• Inbound email
• Competitor news feeds
• Documents
• Voice interaction data
• Unstructured text , sentiment analysis
External data • Open government data
• Weather data
• Structured data
• Semi-structured data e.g., JSON, XML, AVRO,
• Sales impact, distribution impact

Data Virtualisation (Logical Data Warehouse)
Commonly understood, trusted data and insights
Analytical Digital Transformation – Different Platforms Optimised For
Different Analytical Workloads Are Now In Use
Data Warehouse
RDBMS
EDW
DW & marts
mart
DW
Appliance
Advanced Analytics
(structured data)
Analytical
RDBMS
C
R
U
D
Prod
Asset
Cust
MDM
Streaming
data
Streaming
analytics
NoSQL
DBMS
Hadoop
data store
NoSQL
Graph DB
Advanced Analytics
(multi-structured data)
Cloud
storage
Self-service BI
& Analytical
Tools
IT developed
queries,
reports &
dashboards
Data mining,
model
development
Real-time
streaming
analytics &
decision m’gmt
Investigative /
Exploratory
analysis
Graph
analysis
Data mining,
model
development
Big Data analytical workloads have resulted in multiple platforms now
being used for analytical processing

Challenges –Data Is Being Ingested Into Multiple Types Of Data
Store Both On-Premises And In The Cloud
Enterprise
cloud
storage
Data.Gov
C
R
U
prod cust
asset
D
MDM
NoSQL
DBMS DW
I
D N
A G
T E
A S
T

Problems Caused By Greater Complexity
– Finding, Organising and Governing Data
MDM
C
R
U
D
Prod
Asset
Cust
Data marts
Cloud object
storage
ECM Staging
areas
RDM
C
R
U
D
Code
sets
Archived
DW data
Hive
tables
feedsIoT
XML,
JSON
RDBMS Files office docssocial Cloud
clickstream
web logs web
services
NoSQL
Text /
Image/
Video
Filtered
sensor
data
Published
trusted
data
In-progress
data
DWDW
staging
area
ODS
ODS
ODS
NoSQL
NoSQL
Data resides in multiple data stores
Data duplication, overlapping subsets and multiple versions of data
No idea what data is located where or what it means
No idea of data quality, where sensitive data, access security….
No view of what data is processed where and by whom….

What Have We Been Doing To Date?
- The Story So Far Is A Siloed Approach To Managing and Governing Data
Analytical
tools
Data
management
tools
EDW
mart
Structured data
CRM ERP SCM
Silo
DW & marts
Analytical
tools/apps
Data
management
tools
Multi-structured
data
Silo
DW
Appliance
Advanced Analytics
(structured data)
Data
management
tools
Structured data
CRM ERP SCM
Analytical
tools
Silo Silo
C
R
U
D
Prod
Asset
Cust
MDM
Applications
Data
management
tools
Master data
management
CRM ERP SCM
Streaming
data
Analytical
tools/apps
Silo
Analytical
tools/apps
Data
management
tools
NoSQL DB
e.g. graph DB
Silo
Multi-structured &
structured data

Challenges - With 000’s of Data Sources, How Can Business Help
IT Prepare Data At Scale To Avoid A IT Bottleneck?
IT
OLTP
systems
Web
logs
web
DQ/DI
job
DQ/DI
job
DQ/DI
job
Open data
IoT
machine data
social & web
C
R
U
prod cust
asset
D
MDM
DW
Data
warehousing
cloud
Data virtualization
Can business analysts &
Data Scientists help?
Are there any tools aimed
at business professional?
Self-service
data prepSelf-service
data prepSelf-service
data prep
???
Bottleneck?
Should IT be expected
to do everything?
Big Data

Challenges – How Do You Govern Self-Service Data Preparation To
Avoid Chaos In The Enterprise?
social
Web
logs
web cloud
sandbox
Data Scientists
sandbox
Data Scientists
sandbox
Data Scientists
HDFS
ETL
/ DQ
Self-service
BI tools with data prep
ETL
new
insights
SQL on
Hadoop
DW
ETL
/ DQ
DW
marts
ETL
SCM
CRM
ERP
Data
prep
marts Self-service
BI tools with data prep
Data
prep
Built by IT
data
prepData
prepData
prep
Governance?
Everyone is blindly integrating data with
no attempt to share what we create !!

Data Challenges – It’s Not Just About Data Governance…
The Data Strategy Has To Also Deliver New Value
Competitive Advantage!

The Requirement - How Can You Achieve Data Governance While Also
Enabling Agility And Reducing Time To Value?
Vs
Data Governance
Freedom to get ahead of the competition
+ =
Reduce time to value
Why not both?

How Do You Conquer Complexity, Govern Data And Accelerate Time to Value?
1. Establish a common vocabulary for trusted data and insights you intend to share
2. Establish a common data management platform for ingesting, discovering, preparing and
governing data
3. Organise to become data driven
• CDO, programme office, information producers and information consumers
4. Define data and analytical projects aimed at achieving objectives in your business strategy
5. Create an organised data lake from which to manufacture ‘ready to use’ data and analytical
products (assets) for use across the enterprise
6. Create an information supply chain to ‘manufacture’ data and analytical assets
• Ingest -> Discover -> Catalog -> Prepare -> Publish -> Analyse -> Decide -> Act
7. Create an enterprise data marketplace to publish ready to go data and analytics services
8. Establish an agile analytics lifecycle to develop, deploy, monitor and manage models
9. Simplify access to data via a logical data warehouse
10. Deliver value - Consume and use everywhere

16
Copyright © Intelligent Business Strategies 1992-2018
The Foundation for Smart Data Management, Governance And Creating Value
Is A Common Vocabulary and Lineage
Shared
Business
Vocabulary
Standard JSON
representation
Common data names
Common data definitions
Common integrity rules
Standard XML
representation
Logical Data
Structure
Representation
….
Acts as the foundation for sharing data across systems irrespective of whether those
systems are on-premises or in the cloud - It is fundamental to getting rid of complexity
Prevents confusion
and builds trust

17
Why A Common Vocabulary? - Data and Analytics Have Moved to
the Centre of the Enterprise for Use Everywhere
Partners
Customers
Suppliers
Employees
Things
The Intelligent
Business
Data &
Analytics
HR
Sales
Marketing
Service
Finance
Procure
-ment
Operations Distribution
Access to commonly
understood, trusted data,
insights and analytics across the
business is critical to success

18
Integrate Business Glossary With Other Technologies To Push The
Common Vocabulary Everywhere For Consistency
Self-service
Data Prep tools
Software
Development
Tools (services)
C
R
U
D
prod cust
asset
Enterprise Service Bus
& IPaaS software
metrics,
dimensions
Data Science
Workbenches
BI ToolsUML
DW/BI System(s)
Operational
app services & APIs
Business
Glossary
e.g. Hive
Tables
MDM
Data &
Metadata
Relationship
Discovery
Tools
Data
Quality
Profiling &
Monitoring
Tools
Data
Modelling &
Tool
Data
Cleansing
&
Matching
Tools
Data
Integration
Tools
metadata
Data Management Platform – A suite of DM tools sharing a common repository
Business
Glossary
Tool
Data Governance/Management Console & Process Workflows
IDE
plug-in
data virtualisation
e.g. Swagger
publish
DW
mart mart
Information Catalog

Rationalise Siloed Data & Analytics Initiatives To Establish A Common
Approach To Managing And Governing Data
Analytical
tools
Data
management
tools
EDW
mart
Structured data
CRM ERP SCM
Silo
DW & marts
Analytical
tools/apps
Data
management
tools
Multi-structured
data
Silo
DW
Appliance
Advanced Analytics
(structured data)
Data
management
tools
Structured data
CRM ERP SCM
Analytical
tools
Silo Silo
C
R
U
D
Prod
Asset
Cust
MDM
Applications
Data
management
tools
Master data
management
CRM ERP SCM
Streaming
data
Analytical
tools/apps
Silo
Analytical
tools/apps
Data
management
tools
NoSQL DB
e.g. graph DB
Silo
Multi-structured &
structured data
We Need To Rationalise!

EDW
mart
DW & marts
DW
Appliance
Advanced Analytics
(structured data)
C
R
U
D
Prod
Asset
Cust
MDM
Streaming
data
NoSQL DB
e.g. graph DB
Simplify Data Management Via A Multi-Purpose Logical Data Lake
- A Common Platform For Managing & Integrating Data Across All Data Stores
Integrated Self-Service and IT Data Management Tools (Business & IT)
feedsIoT
XML,
JSON
clickstream
web logs web
services
NoSQL
The ‘Logical’ Data Lake is when data in the lake is exists in multiple data stores
but the data lake is managed, operated and governed as if it were centralised
Logical Data Lake
Information

Data virtualization services
The ‘Logical’ Data Lake - An Organised Collection of Raw, In-Progress and Trusted
Data In Multiple Data Stores
DW
MDM
C
R
U
D
Prod
Asset
Cust
Data marts
Cloud object
storage
Refinedtrusted&integrateddata
Stronggovernance
Rawuntrusteddata
somegovernance
ECM Staging
areas
ODS
RDM
C
R
U
D
Code
sets
Archived
DW data
Hive
tables
feedsIoT
XML,
JSON
clickstream
web logs web
services
NoSQL
ODS
ODS
DW
Text /
Image/
Video
Filtered
sensor
data
Published
trusted
data
Search
indexes
In-progress
data
Logical Data Lake
(not a data store but a collection of stores)
Data sources and ingested lake
data are all known to the catalog
Info
CatalogEnterprise Data Management Platform & Analytical Services
(Agility, Control/Governance, Centralised Metadata, Common Approach)

The Information Catalog - Automatically Discover, Catalog And Map Disparate
Data To Your Common Vocabulary
Cloud
object
storage
ECM
Staging areas
ODS
Archived DW
data
Hive tables
feeds
IoT
XML,
JSON
RDBMS Files
office docs
social Cloud
clickstream
web logs web servicesNoSQL
ODS
ODSText /
Image/
Video
Filtered sensor
data
Published trusted
data
In-progress data
DW
Info
Catalogmetadata
Crawlers can be
scheduled to keep the
catalog up to date
Hybrid Logical Data Lake
The catalog crawls analytical ecosystem
data stores and OLTP system data stores
and IoT Schema registries on the cloud and
on premises to discover and inventory data
m
etadata
Business Glossary
inside
MDM
C
R
U
D
Prod
Asset
Cust
RDM
C
R
U
D
Code sets
Schema
registry
Edge devices
metadata

Organise To Become Data Driven
- Information Producers And Information Consumers
§ Need to make use of
• A business glossary and information catalog
• Re-usable services to manage and process data
• Collaboration and social computing to manage, process and rate data
• Role-based data management tools aimed at IT AND business
clean &
integrate
service
raw data
trusted data
Information
catalog
BI tool or
application
search
find
shop
order consume
provision
data scientist
IT professional
information producers
clean &
integrate
service
raw data
publishdata
publish a
service
business analysts
information consumers
like a
“corporate
iTunes” for
data
reuse data

Organise To Succeed – Create Business Strategy Aligned Data Curation And
Analytical Project Teams Tasked With Producing Specific Data And Analytics
Data Science Project
Data Science Project
DW Project
MDM ProjectDW Project
Business
Strategy Objectives
Customer
Risk
Operations
Finance
• What business problems
are you trying to solve?
• What data do you need?
• What kind(s) of analytic
workload are needed?
Data lake management software
should include project capability
Collaborative teams of
business & IT personnel
Data projects and
analytical projects
Share!

Organise Data And Analytics Producers Into Business Strategy
Aligned Project Teams
What’s in a project?
Project Elements Comments
Team members • Business & IT professionals
Connections • Internal and external data sources
Data assets • Files, tables, web services, spreadsheets….
Data preparation flows • ETL workflows
• Self-service data preparation jobs
Analytical assets • Machine learning workflows
• Streaming analytics workflows
• Data science notebooks (Jupyter, RStudio..)
• Analytical models
Collaborations • Discussions on data
….
Ø You need to track it all and keep it organised across the data lake
Ø You should be able to add things to a project e.g. data assets
E.g. Customer engagement projects, operations optimisation projects, risk…

Create An Information Supply Chain To Curate ‘Business Ready’ Data And
Analytical Assets For Use By Consumers
sandbox
Trusted
Data Zone
Raw Data
Zone
master
ref data
DW archive
sandbox
Refinery zone
(prepare & analyse
data)
In-progress data
Refined
data &
Insights
zone
Data lake management
ETL/
Data prepDQ
Data Ingestion
zone
(transient data)
IoT
RDBMS
office docs
social
Cloud
clickstream
web logs
XML,
JSON
web
services
NoSQL
Files
DW data
streams
Logical Data Lake
Exploratory analysis
The information supply chain is a manufacturing process that produces trusted
data assets (products) and analytical assets to publish in a marketplace
information
consumers access
the data
marketplace to
shop for ready-to-
go data and
analytical assets
shop
for
data
Info
Catalog
Data
marketplace
Information Supply Chain (Data Curation Process)Project

Trusted Business Ready Data In A Marketplace For Users To
Consume

28
Shorten Time To Value Using A Component Based ‘Production Line’ Approach
To Curating Trusted Data And Analytical Assets
data
source
Data
Integration
publish Info
catalog
trusted data
as a service
publish Info
catalog
trusted, integrated
data ad a service
subscribe
Analyse
(e.g.score)
consume
publishAnalytics
catalog
New predictive
analytic pipelines
(as a service)
consume
subscribe
Visualise
Decide Act
other e.g. embed
analytic applications
consume
subscribe
publish
Solutions
catalog
New prescriptive
analytic pipelines
publish New analytic
applications
use
crawl,
discover,
profile,
publish
Info
catalog
discovered
data
Data Preparation
(clean, transform, filter)
ingest

Information Catalogs Should Also Be Able To Automatically Discover ETL &
Data Prep Jobs, BI Artefacts, Analytical Models, Data Science Notebooks etc.
Information Catalogue
BI Tools
RDBMS
e.g. DW, MDM
Cloud object
storage
ETL & self-
service data
prep tools
Catalog
metadata
metadata metadata metadatametadatametadata
Mining
Tools
metadata
Data
Science
Notebooks
metadata
A catalog needs to know about much more besides data
The catalog should allow you to share data, pipelines, models
and insights across the enterprise to break down the silos

sandbox
Trusted
Data Zone
Raw Data
Zone
master
ref data
DW archive
sandbox
Refinery zone
(prepare &
analyse data)
In-progress data
Refined
data &
Insights
zone
Data lake management
ETL/
Data prepDQ
Data Ingestion
zone
(transient
data)
IoT
RDBMS
office docs
social
Cloud
clickstream
web logs
XML,
JSON
web
services
NoSQL
Files
DW data
streams
Logical Data Lake
Exploratory analysis
Info
Catalog
Data Management Platform Workflow Links Components In A Pipeline
To Produce Data And Analytic Assets To Publish in a Marketplace
Info
Catalog
Data
marketplace
Prepare DecideVisualiseAnalyse
Pipeline
(workflow)
Pipeline (workflow)
Remote service
Pipeline
(workflow)
ActCapture DiscoverProject
Info
Catalog

An Example Of An AI Based Data And Analytic Pipeline To
Improving Voice Of Customer Analysis - Amazon
Source: aws

Logical Data Lake
Logical Data Lake
EDW
mart
DW & marts
DW
Appliance
Advanced Analytics
(structured data)
C
R
U
D
Prod
Asset
Cust
MDM
Streaming
data
NoSQL DB
e.g. graph DB
Simplifying Data Access Via A Logical Data Warehouse Architecture
- Use Data Virtualisation To Access Data In Multiple Stores
Integrated Self-Service and IT Data Management Fabric Business & IT)
feedsIoT
XML,
JSON
clickstream
web logs web
services
NoSQL
Data&InsightCuration
Analytical tools &
applications
Logical Data Warehouse (Data Virtualisation with a common vocabulary)
shared
metadata
Information
Commonly understood, trusted data and insights

33
Conclusions
§ Digitalisation has introduced complexity and many new challenges
§ We now have an analytical ecosystem with multiple data stores optimised
for different analytical workloads
• Data management is much more complex
• Siloes of data and analytics exist and total cost is too high
• Data access is hard and users are forced to have to integrate data
§ The answer is rationalisation with a need for
• Common vocabulary - needed in all tools producing data and insights
• A logical data lake of multiple data stores with integrated self-service data preparation
and ETL tools sharing metadata
• An information supply chain with strategy aligned projects to produce data assets
• Plug & play data and analytical services to accelerate time to value
• An data marketplace to publish data and analytical assets for consumers to use
• A logical DW (data virtualisation) to simplify access to trusted data

Thank You!
www.intelligentbusiness.biz
mferguson@intelligentbusiness.biz
@mikeferguson1
(+44)1625 520700
Thank You!
Mike Ferguson is Managing Director of Intelligent Business
Strategies Limited. As an independent analyst and consultant he
specializes in business intelligence, analytics, data management and
big data. With over 35 years of IT experience, Mike has consulted
for dozens of companies, spoken at events all over the world and
written numerous articles. Formerly he was a principal and co-
founder of Codd and Date Europe Limited – the inventors of the
Relational Model, a Chief Architect at Teradata on the Teradata
DBMS and European Managing Director of DataBase Associates.

Big Data LDN 2018: DATA MANAGEMENT AUTOMATION AND THE INFORMATION SUPPLY CHAIN – FROM DATA LAKE TO DATA MARKETPLACE

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Big Data LDN 2018: DATA MANAGEMENT AUTOMATION AND THE INFORMATION SUPPLY CHAIN – FROM DATA LAKE TO DATA MARKETPLACE

Similar to Big Data LDN 2018: DATA MANAGEMENT AUTOMATION AND THE INFORMATION SUPPLY CHAIN – FROM DATA LAKE TO DATA MARKETPLACE (20)

More from Matt Stubbs

More from Matt Stubbs (20)

Recently uploaded

Recently uploaded (20)

Big Data LDN 2018: DATA MANAGEMENT AUTOMATION AND THE INFORMATION SUPPLY CHAIN – FROM DATA LAKE TO DATA MARKETPLACE