Presentation delivered to IBM's Insight 2015 conference.

1. © 2015 IBM Corporation
Session 1977: Implementing
Advanced Analytics Platform –
Successes & Architecture Decisions
Dr. Arvind Sathi
Mathews Thomas
Thomas Eunice
Richard Harken

2. • IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal
without notice at IBM’s sole discretion.
• Information regarding potential future products is intended to outline our general product direction
and it should not be relied on in making a purchasing decision.
• The information mentioned regarding potential future products is not a commitment, promise, or
legal obligation to deliver any material, code or functionality. Information about potential future
products may not be incorporated into any contract.
• The development, release, and timing of any future features or functionality described for our
products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks in a
controlled environment. The actual throughput or performance that any user will experience will vary
depending upon many factors, including considerations such as the amount of multiprogramming in the
user’s job stream, the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve results similar to those stated
here.
Please Note:
2

3. Overview
• Motivation for Advanced Analytics Platform
• Business Use Cases
• Application Architecture
• Data Science Discussion
• Data Engineering Discussion
• Q&A
2

4. Motivation for Advanced Analytics Platform
in the Cognitive Era
Key Disruptive Trends:
• Growing interest in applying the results of advanced analytics to improve business
performance
• The rapid growth in available data, particularly new sources of data — such as unstructured
data from customer interactions and streaming volumes of machine-generated data.
• Increasing requirements for higher data and decision velocity
• Shortage of data science skills – how do we leverage small number of data scientists for
increasing number of applications
• Limitations in the use and scaling of existing BI tools
• Open sourced platforms

5. Data Sources and sizes
4
Data Source
Daily
Volumes
Data types available for
Customer Experience
Analytics
CRM / Billing 100s of
Gigabytes
Subscription and demographics
Call Detail
Records / Web
Logs
Terabytes Voice and SMS usage, Web
interactions
Product Usage
Data
10s of
Terabytes
Data and video usage
IoT Data 100s of
Terabytes
Driving data for connected
cars, connected home events

6. • Real-time Decision Engines – need the real-time data right
away, and require real-time scoring engines to rank order and
select candidates.
• Operational Dashboards – require data in near real-time
across large cross-section of the enterprise.
• Advanced analytics (data scientist) users – require raw data for
complex statistical and text-analytics sandbox.
• Business Analysts – require curated batch data for standard
and ad hoc reporting.
• Stewards – require source data to make governance
decisions.
Emerging Users
5

7. Overview
• Motivation for Advanced Analytics Platform
• Business Use Cases
• Application Architecture
• Data Science Discussion
• Data Engineering Discussion
• Q&A
6

8. Subscriber Profiling
& Enrichment
How can I uncover new
insights from subscriber
data for better marketing,
customer care & network
operations?
Subscriber Analytics
(Segmentation)
How do I create subscriber
micro-segments based on
subscriber usage, channel
interaction and mobility
patterns?
Social Media Insight
How can I gain insights
on brand, product &
service reputation,
marketing campaign
impact on various
customer segments?
Proactive
Care
How can I improve
revenue from call
center and lower costs?
Counter Fraud
Management
How can I better predict,
detect and investigate
voice and data fraud?
Network Analytics Based on
Customer Insight
How can I innovate and
improve my network for
better subscriber
experience?
Internet of Things Analytics
and Usage
How can I capitalize on
insights gathered from
IoT to offer personalized
value-added services?
Customer Data
Location
Monetization
How can I monetize
subscriber data for
higher revenue &
profits?
Using our catalog of industry use cases, we have prioritized the following use cases
for industry solutions.
Key Telecom Business Value Cases
Innovate
Business
Models
Transform
Business for
Higher
Efficiency
Improve
Subscriber
Insight
KPI Correlation
How do I drive new and
deeply correlated
insights on key
measures enabling
new value:
NPS, Churn, Cross Sell
Customer Experience
Management
How can I measure and
improve subscriber quality
of experience across all
channels and services?
Vertical Analytics
Integration
How do I partner to build
value added offerings
for other industries?
Retail
Transportation
Financial
Proactive Marketing
& Sales
How can I deliver targeted
marketing campaigns for
higher acceptance rate?
How can I improve
customer care?
NBA, NBO, PBA, PBO,
Omni-Channel
Accelerate Digital Transformation

9. Real Time Actionable Insight (Value Roadmap)
DECISIONSINSIGHTS OUTCOMES
Measure
Results
Historical
Data
SUBSCRIBER
PROFILING &
ENRICHMENT
• Hangout
• Location
• Trends
• Behavior
• Lifestyle
Go
tha
m
Cit
y
Nig
ht
Ow
ls
PREDICTIVE
ANALYTICS (SCORES)
Spor
ts
Fans
Lunch
Crowd
KPI-DRIVEN
ACTIONABLE
INSIGHTS
• NPS
• Churn
• Upsell
• Cross-
Sell
BUSINESS DECISIONS
Upgra
de
Phone
Bad
Device
Lo
w
NP
S
Wron
g
Plan
DATA SOURCE
COLLECTION &
EXTRACTION
DATA / VALUE
SOCIAL
NETWORK
TROUBLE
TICKETS BILLING
DEVICES
APPs
OPERATIONS
TRANSFORMATI
ON
• Proactive Care
• Enhanced Sales &
Marketing
• Fraud & Security
• Revenue Assurance
• Insights Monetization
• New Business Models
BUSINESS OUTCOMES
Business Maturity
INDIVIDUAL
SUBSCRIBER
EXPERIENCE
• Device
• Usage
• Customer type
• Network
• Service
Experience
CUSTOMER PROFILE
(INSIGHTS)
iPhone
5C
Congest
ed 3G
Cell
Hea
vy
Netfli
x
User
s

10. Issue Resolution
• Solve
• Steps to solve
• None
Next Best Action (NBA)
• compare w/ similar
• Tier 2 support
• Tier 3 support
Next Best Offer (NBO)
• Sales
• Up-sell / Cross-sell
Inbound
Resolve
Individual
Call
Reactive Care
Customer can’t
access Netflix video
on their
smartphone, so they
ring a customer care
agent
Monitor
Trends
outbound Communicate
to impacted
Subscribers
Netflix
congestion
issue
Proactive Care
Mobile
App Push
Mobile
Web Push
1
Omni Channel Outbound Communication
How is care different?

11. Psycholinguistic
NPS
Usage
Mobility
CRM
Experience
Interest
Other
Unstructured
Other
Structured
Customer Insights

12. Networking Insights
Sample Insights
 Quality of video/data
 Number of dropped calls
 Number and type of users
 Normal changes vs.
abnormalities
 Trending spots
 Mobility Pattern
Target Segments
 Heavy Video users
 Regularly at cell tower
 Propensity to Churn
 High Value Customer

13. Track
Results
Notify
• High Value
Customer
• Watches Video
• Impacted by
Networking
Issues
Network
congestion
issue
Customer
Insights
1
Monitoring
Trends
Real Time Analytics
Business process
Rules management
Proactive Care – Network
Upsell
Segmentation
Campaign
management

14. Key Flows
Data Sources
Real Time Analytics
Predictive Models
Operational Decisions
Management
Business Process Management
Campaign Management
Mobile Channel
Dashboard

15. Overview
• Motivation for Advanced Analytics Platform
• Business Use Cases
• Application Architecture
• Data Science Discussion
• Data Engineering Discussion
• Q&A
14

16. Advanced Analytic Platform (AAP) - Architecture
Overview
Data Lake
Descriptive &
Predictive
Modeling
Real Time
Analytics
TDR
TransactionsUsage
Real-time Action
Marketing
Customer Care
NOC/SOC
Network
Planning
...
Analyst
Workbench
Batch Action
CRM
Billing
Care
BatchETL
SmartFilter
Conversations
Social
Media
Chat
Network
Engg
CDR
Call Center
ContinuousIngest/Parsing
Unification
Intelligent
Campaigns
Data
Governance
Proactive
Care
Counter
Fraud
Real-time
Dash Boards
Segmentation
Network
Configurations
Dash Boards
Reports
Visualization
Stream and Mediation
Analytics
Data Mart
SQL
Accees
1
2 3
6
5
4
7
8
10
9

17. Architecture Walk Through
Step Description
1 Continuous Ingest / Parsing: CDR data is parsed from ASN.1 format.
2 Unification: CDR & TDR data is unified into a common format and identified with subscribers.
3 Smart filter: Data is filtered for real-time, predictive and descriptive analytics. All data is sent
to the lake.
4 Batch ETL: Source data from transactions and conversation is ingested and sent to the lake
after appropriate transformations.
5 Data Governance: Transactional data is organized into master data, with data quality and
matching. Conversation data is aligned to master data.
6 Descriptive & Predictive Modeling: Creates aggregations, derived attributes and scoring
models.
7 Real-time analytics: Various counters are used for real-time aggregations. Scoring engines
are used for predictive scoring.
8 Real-time Action: Real-time aggregations and scores are sent to respective action engines
and real-time dash boards
9 Batch Action: Tables with aggregate data and derived attributes are made available to batch
consumers.
10 Analyst Workbench: Governed data, aggregations and derivations are made available to
analysts for reports, visualizations and dash boards.

18. Architecture Decision – Bring expertise to data
• In high velocity or high volume situations, data can not be
moved across many tools.
• Many filtering decisions have to be done closer to the source to
bring down false positives.
• These filters must be dynamic and changed by business users.
Source Filter Target
Filter criteria

19. Architecture Decision – Identity Resolution
• Identity resolution provides a way to connect various facts about an entity and resolve
differences.
scrila34@msn.com
Job
Applicant
Identity Thief
Top 200
Customer
Criminal
Investigation

20. Architecture Decision - Feedback and Machine
Learning
Predictive models can be compared for their success and fine tuned
using the following steps:
Step 1 – Many predictive models are developed simultaneously
Step 2 – These models are tested using test or real data
Step 3 – Results are compared and used for fine tuning the models
Sensor
Predictive Modeler
Scorer
Analytics Engine
High Velocity
High Volume

21. Drive
Interact with the
customer to seek
permission to use location
information and send
campaign, record
interaction and results.
Discover
Collect historical
behavioral data, past
acts, and success rates.
Analyze historical data
to formulate patterns
and changes required
to campaign selection
and design rules.
Decide
Use background
information, past
campaigns, privacy
preferences, customer
reaction to past
campaigns, purchase
intent, preferences
expressed in social
media to design
campaign.
Detect
Detect in real time if a
transaction relates to
targeted consumers.
Identify, align, score,
and send for further
processing (e.g., a
targeted customer
driving towards mall)
Architecture Decision – Integration
Detect observations
about a target
Take action in real
time – when it
matters
Find new targets by
analyzing historical
data
Identify patterns
over time and
actions required
Drive
Detect
Discover
Decide
Target
Subscriber
20
Filter
definitions
Filtered
Data
Decisions
Feedback
Interrogations

22. Overview
• Motivation for Advanced Analytics Platform
• Business Use Cases
• Application Architecture
• Data Science Discussion
• Data Engineering Discussion
• Q&A
21

23. Advanced Analytics using Hadoop Lake.
Streams and SPSS
22
Reference
Data
Changes to
Reference
Data
Event Data
Event Data
Data
Integration
Movement
Hadoop
Lake
Local
Appliance
Infosphere
Streams
SPSS
Analytics
Server
SPSS
Modeler
Server
Real Time
Analytics
SPSS
Modeler
Client
Data at Rest
(Historical Data)
Data in Motion
(Real Time)
Real-time Models

24. Location Analytics
CDRs
Location
Affinity
Common locations
by time of day and
day of week
2-6 weeks of
CDRs with
location info
High speed
aggregations and
calculations on big data
Preferred
Locations
Location
algorithms using
SPSS
Home, Work,
Weekend, Locations

25. Mobility Profiling Outputs
Usage Profiles
 Heavy Voice
 SMS Mostly
 No Data
Quality of Service
 Individual QoS measure
 Detailed relation to ARPU and
CLV
Sentiment analysis
 From surveys and comments
 Contact center data
 Social media
Usage Direction
 Declining / Increasing
 For each service
 e.g. Increasing Data, declining
Voice
Personality Profiles
 Commuter
 Homebody
 Night Owl
Interests and Preferences
 OTT Messaging Travel, Shopping,
Betting
 App preferences e.g. Travel, Games
 Handset prefs
Preferred Locations
 Hangouts for groups
 Popular Home and Work locations
 Mode of Travel; train, car, walk
 What is the profile of persons in each hangout
Social Networks and Best
Buddies
 Who calls who
 Who hangs out with Whom?
 Who are the influencers

26. How it Works – Buddy Model
Physical Relationship Social Network
 Seven pass algorithm creates a sparse matrix of
all events within space time boxes (defined as Cell
Masts and 2 minute intervals)
 Subscriber pairs in the same space time box are
counted as a “hit”, then ranked by hits.
 Subscriber pairs that have many hits in many
locations or time frames are kept (above a
threshold for a coincidental relationship
connection)
 The resulting pairs and hit counts are passed to
IBM's SNA algorithm to create the final networks
 Input: 2+ weeks of xDR data for a large
metropolitan area

27. Profile Name Description
Night Owl Primarily active at night
Homebody Does not visit many locations
Delivering the goods Visits many locations during the day (Delivery truck
driver, postman, etc)
Commuter/Daily Grinder
A daily commuter, home → office → home
Predictable/Norm Peterson Activity inside the 2nd standard deviation*
Active Active at many times of the day with no clear pattern
IBM Mobility Lifestyle Definitions
* from the Television show, “Cheers”. Norm was an accountant who went to the same pub every night

28. Discovery of Mobility Lifestyle
• A typical discovery uses statistical tools to identify pattern in data.
• Discovery may contribute new derived attributes for further analysis or reporting.
Night Owls at Night
Delivery People
During the Day
Quiet Weekday people
go for dinner on weekends
Almost no Homebodies any time

29. Hangout Analysis
 What are the most common Lifestyle Profiles at these places

30. Overview
• Motivation for Advanced Analytics Platform
• Business Use Cases
• Application Architecture
• Data Science Discussion
• Data Engineering Discussion
• Q&A
29

31. Illustrative Data Engineering Requirements
• Security by user role
• Classification of data
• Taxonomy, Semantics
• Auditability
• Scalability
• Lineage - metadata and data, relationships across business and
technical
• monitoring
• Auto discovery
• Multitenant and enterprise class (separation of orgs, or sub orgs)
• Policies, leverageable , implementable policies, and business rules
• Integrateable, open API, integration, publishable
• Harvest/ingest metadata from various sources

32. Illustrative Data Engineering Requirements, cont.
• Regulatory compliant - like auditable by dodfrank, hipaa etc
• automate as much of this as possible
• Support transactional an analytic workloads
• Realtime updates
• Sensitive data protection
• Needs to support structured and unstructured data
• Search
• History ( point in time relevance)
• versioning
• Workflow - can't force it thought through policy and procedure
• Support for MDM
• Open APIs (see content from ING slide below)

33. Illustrative Data Engineering Requirements, cont.
• Support for Json, cobol, Nested models, non relational models, all
data is not defined as relational
• Characterize data auto classify for self services, also do this for
data lineage
• Usability Improvements

34. Analytics/Reporting/Consumption
Data Sources
Information Fabric
Operational
Data
Reporting/
Warehouse
Data
Services Layer
3rd Party
Data
Information
Governance
Catalog
/
ATLAS
Cognos SPSS R ML
Spark
Information
Server
Streams
Landing
Zone
Discovery
Zone
Harmonized
Zone
Optim
Guardium Optim
Security (LDAP, Kerberos, HTTPS, Certificates)
1
2
3
5
4
Information Governance

35. Overview
• Motivation for Advanced Analytics Platform
• Business Use Cases
• Application Architecture
• Data Science Discussion
• Data Engineering Discussion
• Q&A
34

36. Reading Material
• IBM Developer Works
 Explore the advanced analytics platform, Part 1: Support your business requirements using big data
and advanced analytics
 Explore the advanced analytics platform, Part 2: Explore use cases that cross multiple industries
using the advanced analytics platform
 Explore the advanced analytics platform, Part 3: Analyze unstructured text using patterns
 Explore the advanced analytics platform, Part 4: Analyze location data to determine movement
patterns using a mobility profile pattern
 Explore the advanced analytics platform, Part 5: Deep dive into discovery and visualization
 Explore the advanced analytics platform, Part 6: Dive into orchestration with a combination of SPSS,
Operational Decision Management (ODM), and Streams using care and fraud management case
studies
• IBM Data Magazine
 Mining Data in a High-Performance Sandbox - Fulfill data analysts’ dreams with data warehouse
appliances for in-database analytics and data mining
 Target Behavior in Real Time for Effective Outcomes: Part 1 - How real-time, adaptive architectures
can drive management decisions for specific use cases
 Target Behavior in Real Time for Effective Outcomes: Part 2 Drive marketing and business
management decisions using a real-time, adaptive architecture
• Books
 Big Data Analytics: Disruptive Technologies for Changing the Game
 Engaging Customers Using Big Data: How Marketing Analytics Are Transforming Business

37. Engaging Customers using Big Data by Arvind Sathi
36
BIG DATA IS RAPIDLY TRANSFORMING HOW
COMPANIES MARKET TO THEIR
CUSTOMERS.
Dr. Sathi uses a series of examples across many
industries, such as retail, telecommunications, financial
services, electronics, high tech, and media, to describe
how each marketing function is undergoing fundamental
changes: how personalized advertising is delivered using
online channels where the marketers identify the specific
customer and tailor their messaging based on customer
behavior, context, and intention; how customer behaviors
are collected from a variety of sources across many
industries and combined to identify micro segments; and
how online and physical stores collaborate to provide a
unified shopping experience and deliver product
information.
Engaging Customers Using Big Data provides the tools
and techniques necessary to effectively implement big
data into your marketing strategy, including statistical
techniques, qualitative reasoning, and real-time pattern
detection, and more.
Come and collect a signed copy of the
book at the Book store – Monday
October 26, 4:30 to 5:00 PM.

38. We Value Your Feedback!
Don’t forget to submit your Insight session and speaker
feedback! Your feedback is very important to us – we use it
to continually improve the conference.
Access the Insight Conference Connect tool at
insight2015survey.com to quickly submit your surveys from
your smartphone, laptop or conference kiosk.
37

39. 38
Notices and Disclaimers
Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form
without written permission from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for
accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to
update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO
EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO,
LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted
according to the terms and conditions of the agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as
illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other
results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services
available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the
views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or
other guidance or advice to any individual participant or their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the
identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the
customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will
ensure that the customer is in compliance with any law.

40. 39
Notices and Disclaimers (con’t)
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly
available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance,
compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the
suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to
interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights,
trademarks or other intellectual property right.
• IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document
Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM
SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON,
OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®,
pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ,
Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of
International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be
trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at:
www.ibm.com/legal/copytrade.shtml.

41. © 2015 IBM Corporation
Thank You