Fathoming Data for Competitive Advantage

CW
IN
CAPGEMINI
WEEK OF
INNOVATION
NETWORKS
Fathoming Data for
Competitive Advantage
Gururaj Joshi, Bangalore, Sep 26th 2018

The Evolving 3rd Platform
At-Scale PersonalizationExponential Change
Autonomy
Data as a Service
Reference : https://www.idc.com

Data Deluge
By 2024 machine-to-machine connections will grow to 27 billion
By 2020, a quarter of a billion cars will be connected to the Internet
By 2020 Smartphone users alone are predicted to number over 6 billion
By 2020, devices that connect to the Internet are 50 billion.
By 2025; the amount of analyzed data that is “touched” by cognitive systems will grow by a factor 1.4ZB
By 2025, more than a quarter of data created will be real time in nature, and real-time IoT data will
make up more than 95% of this.
By 2025, an average connected person anywhere in the world will interact with connected
devices nearly 4,800 times per day
By 2025, nearly 20% of the data will be critical to our daily lives and nearly 10% of
that will be hypercritical.
Reference : https://www.idc.com

The Power of Data
Healthcare
investing billions in developing
new biometric sensors and
wearable technology that
tracks health and fitness.
Sports
Retail
retailers are constantly
finding cutting-edge ways
to draw insights from the
ever-increasing amount of
information available
about their customers
Government
and public
sector
services
Financial
services,
banking
and
insurance
Are adopting to the data first
approach to driving business
growth and enhancing its
services for customers.
Energy
Transportat
ion and
logistics
From the weather to the
condition of vehicles and
machinery, and data
analytics
enables businesses to drive
significant efficiencies
Agriculture
and farming
it’s possible to take more
than a million readings –
vastly increasing the
amount of data gathered
during exploration
several data-enabled
services that let farmers
benefit from crowdsourced,
real-time monitoring of
data collected
An increasing number of
cities are currently piloting
data analytics with the aim
of turning themselves into
‘smart cities’
Most elite sports have now
embraced data analytics &
its hard to think of any
area of sport that isn’t
embracing data
Businesses built
on data
A glance at the 10 most
valuable Fortune companies ,
Proves that their business
model are built on data, or are
heavily investing in data

every business should be a data business
data is becoming a key business asset, central to the success of every company, big or small.
Data is a valuable
asset and, as a
result, companies
are more hungry
for data than ever
before
As the world
becomes smarter
and smarter, data
becomes the key to
competitive
advantage.
every tiny piece of
data may very well
be valuable to some
extent or another.
every business
therefore needs a
robust data strategy,
Those without risk
being left behind.

The Path Of Wisdom

© 2018 Capgemini. All rights reserved.Fathoming Data for Competitive Advantage | Gururaj Joshi | Sep 26,2018
Pathways to Big Data adoption: A strategic value-drivers based
approach
Strategic Value & Differentiation:
Data Monetization
Personalization & Intimacy:
Data Science
Scale, Variability and Flexibility:
Data Lakes 1
2
3
4
1
2
3
4
Speed and Responsiveness:
Data Streams
ValueDriversandBigDatalevers
The Data Explosion in the digital universe is transformational – and can unlock answers to
critical business needs and opportunities

Validity
TRUST OF
DATA
Volatility
RETENTION OF
DATA

What we seek about data …
S No Source Details
1 Volume – What is the volume of data coming for each source
1.1 Overall Size of dataset in One year (GBs)
1.2 Size of dataset in upload (Small, Medium, Large, Extra Large)
1.3 Avg number of Records of Dataset
1.4 Does dataset contains binary data (Capture limit of binary dataset)
2 Variety – Different types of data set
2.1 Types of Structured Data
2.1.1 RDBMS
2.1.2 XML
2.1.3 JSON
2.1.4 CSV
2.1.5 Capture the structure
2.2 Is the data in Semi-structured form?
2.3 Is the data in Completely unstructured?
2.4 Is the data in binary format?
2.5 Location of source data (On-premises, Private Cloud, Public Cloud, Hybrid)
2.6 What should be source provenance
2.7 On boarding timestamp
2.8 Source Identification( exact file name formats)
2.9 Source data header/ trailer format's available
2.10 Are there any delimiters at different places available
2.11 Does Business metadata captured
2.11.1 Business names
2.11.2 Descriptions
2.11.3 Tags
2.11.4 Quality
3 Velocity – Rate of data ingestion, transformation & visualization
3.1 Is the dataset expected to be real time
3.2 Is the dataset expected to be one time bulk ingested
3.3 Is the dataset expected to be ingested in incremental sizes
3.4 Is the dataset expected to be ingested in batch mode (Repeated small chunks of dataset)
3.5 Does data ingested by pull-based refreshes/ push-based refreshes
3.6 Frequency of Data Ingestion
3.7 What is expectancy of availability of data at Transient Data layer
3.8 What is expectancy of availability of data at Immutable Raw Data layer
3.9 What is expectancy of availability of data at Enriched Data layer
3.10 What is expectancy of availability of data at Trusted Data layer
3.11 What is expectancy of availability of data at Discovery Data layer
3.12 What is expectancy of availability of data at Visualization Layer
4 Veracity –
4.1 Are there any known anomalies in data set
4.2 Are there any known data cleaning activities required
4.3 Are there any known data formatting activities required
4.4
Are there any known data to be tokenized or masked to protect
personally identifiable information (PII)
4.5 Are there any known data to be tokenized or masked for sensitive data.
4.6
Data Classification (Open, Organizational, Internal, Restricted, Need
basis)
4.7 Data correctness and accurate for the intended use.
4.8 Are the data sets validated by owners for correctness
4.9 Are any specific Metadata to be captured
4.10
Does the data source make the data available easily or specific connector
needs to be built?
5 Volatility - Refers to shelf life of Data
5.1 How long is data valid
5.2 How long data needs to be stored
5.3 Is there point of data irrelevance available
5.4 Capture Data Audit (Lineage)
6 General
6.1 Sample data to be prepared / made available by the data provider .
6.2
Does Mapping Document (mappings, transformations and joins provided )
& is verified with source
6.3 Will there be any control file for each of the source files?
6.4
In a Scenario where the process is down for multiple days?
Is it required to load the backlogs of files or just the latest file?
6.5
The delimiters or special characters to be used should be confirmed in
appearance and their ASCII value provided.
6.6 What validations need to be Performed for source file names?
6.7
Do the file contains columns which indicates the start and end of field
values
6.8 Is it required to validate the header and trailer details?
6.9 Details of data upload failure intimation / Process followed
7 Security
7.1
Are there any known users groups identified (User classified in separate
groups)
7.2 Are there any known users identified as Owners (Full access to data)

Components of data architecture
Batch Ingestion /
Data Store
Batch Processing
Real Time Ingestion Stream Processing
Analytical Data Store Analytics & ReportingData Source
On-Premise
RDBMS
Static Files
Real-Time
Cloud Storages
Devices
Social
Events
Sensors
Shared
File, Queue, Blob Process Long-running /
Large Data Sets
Store / Buffer/ Reliable
delivery / real-time
messages
filtering, aggregating,
preparing the data
serve data for
analysis in
structured format /
low-latency NoSQL /
Interactive
distributed data
•Data Modeling Layer,
•Self-service BI
•Visualization tech
•Interactive data
exploration
Automated Workflows, Transform , Move between, Load processed, Push results to report or dashboardOrchestration
Securing access to dataSecurity Details on dataMeta Data Guidance for dataCatalog Right dataQuality
Data Store
Build/ train models /
NLP
Machine Learning
Data Services
(APIs)
Data Processing
Algorithms

Zones of Data
Data is organized into zones that serve specific functions. Each zone data is accessible and viewable
Why: Record Incoming Data Snapshot
Where : Transient Data (staging area)
What :
•Data first comes into the data lake.
•Basic quality checks
•Create new and different transformed data sets
•Create Data Catalog
•Create Meta Data (Automated )
•Provide access on need basis
How :
•Data can come by Manual, Stream, Batch
•Raw data is categorized by Source
Why: Source snapshot for Analytics &
Discovery
Where : Raw Data (Immutable Raw data
area in original form)
What :
•Data is immutable
•Data can be tokenized or masked to protect
personally identifiable information (PII) or other
sensitive data.
•Data is in original format
•Restricted / Very Limited Access is provided
How :
•Data can come from source & in format for
analytics
•Raw data is categorized by Source
Why: Provide insights
Where : Enriched Data (Refined Data )
What :
•Create refined data sets from raw data / Trusted
data
•Perform data enrichment
•Perform data quality checks
•Define new structures for common data models
• Democratizing access
How :
Data is added after transformation and processing
Enriched data is categorized by Destination
Why: Single Source of Truth
Where : Trusted Data
What :
•Create Trusted data sets from raw data/ Enriched Data
•Create Master Data / Reference Data
•Democratizing access
How :
Data is added after transformation & processing
Trusted data is categorized by Destination
Ingestion Storage
Trusted Data
Enriched Data
Data
Discovery
Transformed Data
Reference Data
Master Data
Raw Data
Immutable
Manual
Stream
Batch
Transient Data
Data ingestion
snapshot
Transient Data
Why: Value Discovery
Where : Data Discovery Zone (Sandbox Area)
What :
•Create Trusted data sets from raw data/ Enriched Data
•Play with Data
•More Analytical Models (by Data Scientist & Machine
learning)
• Limited Access
How :
•Data is in multiple formats
•Slice N Dice
•Data is categorized on need basis
Mask /Tokenize to
protect sensitive data,
Data ingestion
snapshot

Folder Structure
Raw data is categorized by Source
Enriched and Trusted data is categorized by Destination
Raw_Data_Zone
Org_Internal
Business Unit
India
Consumer
Legacy
2017
Jan
01 (Day)
02
Feb
2018 Jan
01 (Day)
02
Order
Products
Unclassified
USA
Business_unit
Org_External
Org_Collabrati
on
Open_Collecti
on
DataZone Source Classification Platform
Geography
Entity Year Month DayBusiness Unit

Metadata structure
Why Metadata:
•Choose data from the right place and time.
•Different data sets require different types of preparation
•Identify any data causing error in any stage from ingestion through to the processing
•Restricted access to sensitive data.
•Use by downstream programs
•Create an environment of continuous quality improvement.
Technical:
Technical metadata captures about the nature of data, which is easier to find and understand attributes.
Type of Data (Text, JSON, Avro, Parquet, XML, etc)
Schema Structure (Fields and their types)
Size
Operational:
Operational metadata captured from processes about the data genealogy, which is easier to find and understand course of flow.
Data Source
Filename
Time of creation
Time of acquisition
File size
MD5 Hash (Redundancy checks make sure the transmission was not corrupted)
Watermark / special identifiers
Lineage
Quality
Profile
Provenance
# of Rejected Records
Job Status (Success/ Failure/ Partial)
Number of bad fields or bad records
Job Process Time
Business:
Business metadata captures what the data means to the end user to make data fields easier to find and understand
Business names
Descriptions
Tags
Quality
Masking rules

Data Catalog Data catalog serves as system of registration and system of discovery for enterprise data assets
Annotate
Data
Assets
Remove
Data
Assets
Discover
Data
Assets
Register
Data
Assets
Connect
Data
Assets
Manage
Data
Assets
Provision
Data
Catalog
One data catalog per
organization Register key structural
metadata such as names,
types, experts, & locations
from the data source
Discover data assets
with easy search & filter
for the indexed metadata
(Any property in catalog,
annotations etc)
Remove data assets
from the data catalog.
Provide annotations ( information
as in descriptions , tags,
taxonomy , documentation ) for
the data assets.
Open data assets in
integrated client tools
& non-integrated tool
Control the visibility
to specific users or to
members of specific
groups.
Data Sources
Enable Data
Democratization
Connect People to
data and empower
them

Data Quality Dimensions
Data Quality
Completeness
Uniqueness
Timeliness
Validity
Accuracy
Consistency

17Fathoming Data for Competitive advantage | Gururaj Joshi | Sep 26,1018 © 2018 Capgemini. All rights reserved.
Private
Security as a Critical Foundation
▪ Information requiring
the highest security,
such as financial
transactions,
personnel files,
medical records, and
military intelligence
▪ Information that the
originator wants to
protect, such as trade
secrets, customer
lists, and confidential
memos
▪ . Account information
that, if breached,
could lead to or aid in
identity theft
▪ Information such as
emails that might be
discoverable in
litigation or subject to
a retention rule
▪ Information such as
an email address on a
YouTube upload
Compliance-
driven
Confidential CustodialLockdown
The percentage of data requiring security will be near 90% by 2025, and this data falls into five categories

INFRASTRUCTURE

Open Source Tools….

Advanced Analytics

Cognitive Tools…..

Business models for data economy
Store & Host
Filter/Refine
Enhance / Enrich
Simplify Access
Consult /Advise
Collect & Supply
Gather and sell raw data
Hold onto someone else’s data for them
Strip out problematic records or data fields or release interesting data subsets
Blend in other datasets to create a new and interesting picture
Help people cherry-pick the data they want in the
format they prefer
Provide guidance on others’ data efforts

A Checklist for Success
• Business-Benefit Priority List
• Architectural Oversight
• Security Strategy
• Compute , I/O and Memory Model
• Workforce Skillset Evaluation
• Operations Plan
• Disaster Recovery Plan
• Communications Plan
• Monetization Plan
finding ways for data to make our lives better that we didn’t imagine even a few years ago

Questions & Answers

CW
IN
CAPGEMINI
WEEK OF
INNOVATION
NETWORKS
Thank You!
Phone: +91 9590019491
Gururaj.Joshi@capgemini.com
Gururaj Joshi
Enterprise Architect
@mgururaji
Speaker 1
Photo

This message contains information that may be privileged or confidential and is
the property of the Capgemini Group.
Copyright © 2018 Capgemini. All rights reserved.
A global leader in consulting, technology services and digital transformation, Capgemini is
at the forefront of innovation to address the entire breadth of clients’ opportunities in the
evolving world of cloud, digital and platforms. Building on its strong 50-year heritage and
deep industry-specific expertise, Capgemini enables organizations to realize their business
ambitions through an array of services from strategy to operations. Capgemini is driven
by the conviction that the business value of technology comes from and through people.
It is a multicultural company of 200,000 team members in over 40 countries. The Group
reported 2017 global revenues of EUR 12.8 billion.
About Capgemini
Learn more about us at
www.capgemini.com

Fathoming Data for Competitive Advantage

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Fathoming Data for Competitive Advantage

Similar to Fathoming Data for Competitive Advantage (20)

More from Capgemini

More from Capgemini (20)

Recently uploaded

Recently uploaded (20)

Fathoming Data for Competitive Advantage