Real time analytics in Big Data

Intended for Knowledge Sharing only
PREDICTIVE ANALYTICS & BUSINESS INSIGHTS SUMMIT
Mar 2016

Disclaimer:
Participation in this summit is purely on personal basis and not representing VISA in any form or
matter. The talk is based on learnings from work across industries and firms. Care has been taken to
ensure no proprietary or work related info of any firm is used in any material.

Quick recap of what it is
REAL TIME ANALYTICS

AS THEY ARE ENVISIONED TODAY…
Intended for Knowledge Sharing only 4
SPEED PRECISION POWER

…BUT IT HAS GROWN TO
SPEED PRECISION POWER
DISTANCE
PAYLOADS
RE-USABLE
MISSION
LONGEVITY

OH MY…
HOUSTON,
WE HAVE A
PROBLEM!

ARE YOU SURE IT’S POSSIBLE IN BUSINESS WORLD?

AN EXAMPLE FROM OUR BUSINESS WORLD
8
...sync with business hours, predictive alternative means, nearby businesses instead,
book an online appointment for future, mail/call instead, suggest virtual interaction,
discovery

LET’S SEE IT IN ACTION…

ADOBE CAPTURED IT PERFECTLY…
10Intended for Knowledge Sharing only

HOW COULD IT HAVE BEEN AVOIDED
No Knee jerk reaction
Statistical significance
Cross validation across multiple data sources
Explanation of the drivers
Proper response mechanism
11Intended for Knowledge Sharing only

HOW REAL IS REAL TIME ANALYTICS?

UNITED BREAKS GUITAR

OK AGREED, BUT WHAT ARE THE OTHER USE CASES?
OPERATIONAL
FRAUD
PRODUCT LAUNCHES
• System downtime, users experience issues, API failures, load
times, etc. – by regions, products, browsers, devices, etc.
• Fraud rates, types, amount, hacking, system compromise,
gaming/misuse, etc.
• New Product/Flow/App/Feature/Plug-ins performance, issues
• User Behavioral changes
FUNCTIONS TYPICAL USE CASES
MARKETING CAMPAIGNS • Campaign usage & inventory management– popular/flop/gaming
SALES
• Recommendation engines – Cross/Up sell
• New Product sales
• Inventory Management
BRAND MANAGEMENT
• Social Media Monitoring – VOC, NPS, SOV (a Trending issue or
opportunity)
14

HOW DO WE PULL IT OFF?

Setting up
right
Analytical
Framework
Data
Collection &
Preparation
Analysis Action
CURRENT ANALYTICAL FRAMEWORK NEEDS END-TO-END OPTIMIZATION…
Problem
Statement
1 Strategy
 Type of functional use case
 Objective & strategic measurements (&
impact on Corporate KPI)
 Analyses, Alert thresholds, impact sizing
2 Execution
 Command-Control (Working Group)
 Communication protocols & methods
 Response Framework (Approvals)
 Fall back options, alternatives, ramps
3 Organizational
Transformation
 People-Process-Technology-Culture

Data
Collection &
Preparation
Analysis Action
Problem
Statement
Setting up
right
Analytical
Framework
Type of reporting: Statistical Process Controls (Deviation from mean, median, expected
values, benchmarking)
Other techniques required: A/B Testing, VOC, Social Media Monitoring, Mining of
patterns, etc.
Sizing & Prioritization of issues depending on impact on corporate KPIs
Types of alerts based on metric: Statistical Significance of deviation, consistency (VOC,
Social), absolute count thresholds (statistical significance calculation based), benchmarking
Level of explanation required: Multi level drilldown, early warning indicators and data
points to cross validate with

Analysis Action
Problem
Statement
Setting up
right
Analytical
Framework
Data
Collection &
Preparation
Data ingestion: Volume, Variety (OLTP, Clickstream, Social, Server Logs, Campaign,
Industry, Search traffic, Devices, Regions), Velocity & Value
Data blending: Ability to manage fast, at scale mix to come up with complete view
Data Governance: Data Quality (monitoring to ensure data feed is reliable, sensible and
not an issue), Data Lineage (ability to back track & understand the data is what it is
supposed to be) and Data Understanding (indicates the right usage that it was intended
for).

Action
Problem
Statement
Setting up
right
Analytical
Framework
Data
Collection &
Preparation
Analysis
Reporting: Depending on required analytical framework, audience, use case
A/B Testing: Analyze multiple variations and/or benchmark with current experience
Sizing & Investigation: Estimation of impact on Corporate KPI, Prioritization, ability to
explain numbers and evolving patterns
Investigation: Cross Validation, Continued trends, benchmarking

Problem
Statement
Setting up
right
Analytical
Framework
Data
Collection &
Preparation
Analysis Action
Mode of communication: Email/Text alerts, App Notifications, Calls?
Content: (post investigation– cross validated, continuing, benchmarking)
-What has happened: Bands breached, Statistically Significant size, Threshold counts,
trending topic)
-Where & for whom: Region, Product Type, Flow, Browsers, Customer Segment
-How big: Dollar impact, impact on Corporate KPI
-Possible drivers: Based on data analyses, Domain expert input, working group
-Recommendation
Response Type: Approval to stop/continue/ramp/alternative – over mail/app/calls
Feedback Loop: Learning needs to be fed back into mainstream analytics

TECHNOLOGICAL FRAMEWORK

DATA PROCESSING PIPELINE
22
Ingest /
Collect
Store
Process /
Analyze
Consume
/ Visualize
DATA
Answers

DATA CATEGORIZATION
23
HOT WARM COLD
Data Volume MB-GB GB-TB TBs
Item size B-KB KB-MB KB-TB
Latency Millisec-sec Minutes – hour Hrs, Day
Durability Low-Medium High Very High
Maintenance Very High High Low
Applications Real-time, Alerts
Analysis and
reporting
Deep dive analysis
and Machine
learning

DATA EVOLUTION (MASLOW HIERARCHY OF NEEDS)
24
Batch PredictionReal-time
Reports Alerts Forecast

LAMBDA ARCHITECTURE
25
Aims to satisfy the needs for a robust system that is fault-tolerance, both against hardware failures
and human mistakes, being able to serve wide range of workloads and use cases, and in which
low-latency reads and updates are required. The resulting system should be linearly scalable.
1. All data entering the system is dispatched to both batch layer and speed layer for processing.
2. The batch layer has two functions: (1) managing master dataset (an immutable, append-only) (2) to pre-
compute batch views.
3. The serving layer indexes the batch views so that they can be queried in low-latency
4. The speed layer compensates for the high-latency of updates to the serving layer and deals with recent data
only.
5. Any incoming query can be answered by merging results from batch views and real-time views
Reference : http://lambda-architecture.net/

LAMBDA ARCHITECTURE
26
New data
stream
HADOOP
All
data(HDFS)
Enriched data
SPARK
Data Stream
Increment
Views
Query
Batch Layer
Access Layer
Speed Layer

LAMBDA ARCHITECTURE – WITH BENCHMARKS
27
New data
stream
HADOOP
All
data(HDFS)
Enriched
SPARK
Data Stream
Alerts
Benchmarks
(rules engine)
Benchmarks
(rules engine)
Data Stream
Batch Layer
Access Layer
Speed Layer

IN CONCLUSION…

WHY DO WE THINK THE TIME IS NOW?
Evolution in the value prop of Real Time Analytics:
What/where/how much (Descriptive) -> what can happen (Predictive) -
>what should we do (Prescriptive) ?
Audience has broadened (From Operational to other key functions)
Demands on RoI have gone up
Data Mining is maturing enough to be used to answer “Real time Pattern
identifications”
29
KPI of Analytics has changed from Turn-Around-Time (TAT) to Time-to-
Action (TTA)

KEY TAKEAWAYS
30
• “Know” that Real Time Analytics is a need not luxury
• “Must have” a strong Strategic, Tactical & Organization framework
• “Ensure” Cross validation, Sizing & Prioritizing
• “Develop” Command-Control Structure & Working Group to ensure “rapid but
right” response
• “Prepare” for evolution of Real Time Analytics closer towards Artificial Intelligence

Appendix

Disclaimer:
Participation in this summit is purely on personal basis and not representing VISA in any form or
matter. The talk is based on learnings from work across industries and firms. Care has been taken to
ensure no proprietary or work related info of any firm is used in any material.
Director, Insights at Visa, Inc.
Enable Decision Making at the Executives/
Product/Marketing level via actionable
insights derived from Data.
RAMKUMAR RAVICHANDRAN
Data Warehouse Architect at Visa, Inc.
Architect a data-shop in Hadoop to get 360-
degree view of the interaction. Technology
interface for the Data Stakeholder Community.
BHARATHIRAJA CHANDRASEKHARAN

THANK YOU!
Would love to hear from you on any of the following forums…
https://twitter.com/decisions_2_0
http://www.slideshare.net/RamkumarRavichandran
https://www.youtube.com/channel/UCODSVC0WQws607clv0k8mQA/videos
http://www.odbms.org/2015/01/ramkumar-ravichandran-visa/
https://www.linkedin.com/pub/ramkumar-ravichandran/10/545/67a
https://www.linkedin.com/in/dataisbig
http://bigdatadw.blogspot.com/
BHARATHIRAJA CHANDRASEKHARAN
RAMKUMAR RAVICHANDRAN
33

34
SOURCES OF VARIOUS IMAGES
Images from:
https://www.google.com/search?q=f16&source=lnms&tbm=isch&sa=X&ved=0ahUKEwjT2ZKytr_LAhVM12MKHZvtAngQ_AUIBygB&biw=1366&bih=599#i
mgrc=W6qpeXNuNSm1lM%3A
https://www.google.com/search?q=fast+and+furious&biw=1366&bih=599&source=lnms&tbm=isch&sa=X&sqi=2&ved=0ahUKEwjBgqfZt7_LAhXkJJoKHb8R
DrsQ_AUIBigB#imgdii=cDHYaybkEHafyM%3A%3BcDHYaybkEHafyM%3A%3BW2D1W4BUx3boGM%3A&imgrc=cDHYaybkEHafyM%3A
https://www.google.com/search?q=sandra+bullock+astronaut+movie&biw=1366&bih=599&source=lnms&tbm=isch&sa=X&ved=0ahUKEwj23PKPvb_LAhV
E92MKHSiiD1kQ_AUICSgD#imgrc=lKmxS5CNElGmPM%3A

Real time analytics in Big Data

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (13)

Similar to Real time analytics in Big Data

Similar to Real time analytics in Big Data (20)

Recently uploaded

Recently uploaded (20)

Real time analytics in Big Data