Monitizing big data at Telcos

1. Monetizing Big Data at Telecom
Service Providers
Juergen Urbanski
Tech Alpha

2. Hadoop Makes Shareholders Happy
The World's Largest Telcos are Driving Business
Performance with Hadoop at the Center of an
Enterprise-Wide Modern Data Architecture
Juergen Urbanski
CEO, Tech Alpha
Board Member Big Data & Analytics, BITKOM (German IT Industry Association)

3. Agenda
• Telco Data Management Challenges
• Hadoop Business Value
• Data Lake Business Value
• Data Lake Reference Architecture
• 21 Telco Use Cases for Hadoop
– Network Infrastructure
– Service and Security
– Sales and Marketing
– New and Adjacent Business
3

4. Enterprise Data Management Challenges
Limited Insight:
• Schema On Write
• Data In Silos
Limited Scale:
• Not Designed to Scale
• Not Affordable at
Scale
Physical
Infrastructure
Presentation &
Application
Data
Access
Data
Management
Engineered
Systems
Shared Storage
Systems
OLTP OLAP
Traditional
Analytics
=
=
– 4 –

5. Business Value of Hadoop
Data Access Layer
Data Management Layer
Hadoop Core Capabilities:
Broader Insights:
• Allows simultaneous access by and
timely insights for all your users across
all your data
• Irrespective of the processing engine,
analytical application or presentation
• Enabled by schema on read and
enterprise-wide pool of data
Unlimited Scale:
• Allows to acquire all data in its original
format and store it in one place, cost
effectively and for an unlimited time
• Affordable and performing well into the
100+ petabyte scale
=
=
– 5 –

6. A New Approach for Broader Insights
HADOOP
Iterate over structure
Transform and analyze
Hadoop Approach
• Apply schema on read
• Support range of access patterns to
data stored in HDFS: polymorphic
access
Batch Interactive Real-time
Right Engine, Right Job
In-memory
Traditional Approach
• Apply schema on write
• Heavily dependent on IT
Determine list of questions
Design solution
Collect structured data
Ask questions from list
Detect additional questions
Single Query Engine
SQL
– 6 –

7. Compelling Economics Allow Scale
0 5 10 15 20 25 30 35 40
SAN
EDW / MPP
Engineered System*
NAS
HADOOP
Cloud Storage
Min
Max
Fully Loaded Cost per Raw TB Deployed
US$ ‘000s
Hadoop Provides
Highly Scalable Data
Storage at 5% of the
Cost of Alternatives
36 to 180
20 to 80
12 to 18
10 to 20
0.250 to 1
0.1 to 0.3
* E.g., Oracle Exadata
– 7 –

8. 5 Capabilities of Hadoop 2.x Enable the Data Lake
– 8 –
Data
Integration &
Governance
Integrate with
existing
systems.
Move data into,
within and out of
the environment
Security
Provide layered
approach to
security
Operations
Deploy and
manage a
multi-tenant,
environment
easily, using
existing tools
where possible
Environment and
Deployment Model
Run anywhere
Data Lake Functional Requirements
1 3
2
4
Data Access = Insight
…ask questions later (or in the
moment)
Data Management = Scale
Store first…
Presentation & Application
Enable existing and new
applications
5

9. Data Lake Reference Architecture
– 9 –
Deployment
Model
Environment
Data
Integration &
Governance
Data
Access
Security Operations
Data
Management
Storage: HDFS
(Hadoop Distributed File System)
Multitenant Processing: YARN
(Hadoop Operating System)
Online
HBase
Accumulo
Real-
Time
Storm
Others
Commodity HW
Linux Windows
Appliance
On Premise Virtualize
Cloud/Hosted
Authentication
Authorization
Accountability
Data Protection
across
Storage: HDFS
Resources: YARN
Access: Hive, …
Pipeline: Falcon
Cluster: Knox
Provision,
Manage &
Monitor
Ambari
Scheduling
Oozie
Data Workflow
Data Lifecycle
Falcon
Real-time and
Batch Ingest
Flume
Sqoop
WebHDFS
NFS
Batch
Map
Reduce
Script
Pig
SQL
Hive
In-
memory
Spark
Metadata Management
HCatalog
Presentation & Application

10. Multiple Use Cases and Tools Run on Hadoop as a
Shared Service
– 10 –
Hadoop 2.x:
Shared Service = Data Lake
Hadoop 1.x:
Dedicated Project Silos = Data Ponds
BU2 BU3
BU1
Customer
Intimacy
Hbase
Opera-
tional
Excellence
Lucene
New
Business
Storm
Risk
Manage-
ment
Map-
Reduce
BU4
Customer
Intimacy
Hbase
Opera-
tional
Excellence
Lucene
New
Business
Storm
Risk
Manage-
ment
Map-
Reduce
Enterprise-wide
• Poor resource management
• Limited governance
• Batch processing, no streams

11.  Shared service operational benefits similar to infrastructure cloud
 Speed of provisioning and de-provisioning for capacity and users
 Fast learning curve and reduced operational complexity
 Consistent enforcement of data security, privacy and governance
 Optimal capital efficiency driven by scale and load balancing
 Value grows exponentially as data from more applications lands in
one Hadoop 2.x data lake
 Marginal cost of retaining data is less than marginal value
 Able to run a broader range of analyses
 More data in one place usually leads to better answers
 Results is order-of-magnitude better insights
Data Lake Business Rationale
– 11 –

12. Technical and Business Drivers
– 12 –
Foundation for a
modern data
architecture
New data types
Sensors
Machine
Generated
Geolocation
Documents, Em
ail,
Voice to Text
Social Networks
Web Logs,
Click Streams
Operational
excellence
E.g., Network
Maintenance
Compliance &
Risk Mgt.
E.g., Fraud
Reduction
Customer
Intimacy
E.g., 360
o
View of
Customer
New Business
E.g., Data as a
Product
Business drivers

13.  Network capacity planning
 Network upgrades
 Network maintenance
 Network performance management
 Network traffic shaping
21 Telco Use Cases for Hadoop
– 13 –
Use Case
Network Infrastructure
Function
 Customer experience analytics
 Contact center productivity
 Field service productivity
 Data protection and compliance
 End-user device security
Service and Security
 360-degree view of customer value
 Personalized marketing campaigns
 Upselling and cross-selling
 Next-product-to-buy (NPTB)
 Churn reduction
Sales and Marketing
 New product development
 Actionable intelligence serving:
 Advertisers
 Merchants/retailers
 Payment processors
 Federal governments
 Local governments
New and Adjacent Business
Network
Care
Sales
New Biz

14. Hadoop in Network Infrastructure
– 14 –
Business Problem
 Network capacity planning
 Network upgrades
 Network maintenance
 Network performance management
 Network traffic shaping
 Hadoop is used to optimize the rollout of
4G coverage in time and space to
match the likely pick-up in service
revenue, allowing an operator to defer
more than 10% of capex for the same
resulting revenue.
 Hadoop helped detect that only a small
number of congested cable network
nodes were responsible for the majority
of churn, and could thus be prioritized
for maintenance and upgrades.
 Network function virtualization, software
defined networking and unified all IP
networks vastly increase the amount of
machine and log data relevant for
trouble shooting. Hadoop helps with
root cause analysis and may even be
used to reason on the data in real-time.
Value Realized
Network
Care
Sales
New Biz

15. Network Infrastructure –
Network Capacity Planning
– 15 –
Business Problem
 The consumption of services and
resulting bandwidth in a particular
neighborhood may be out of sync with a
telco’s plans to build new towers or
transmission lines in that same
neighborhood.
 This leads to a mismatch between
expensive infrastructure investments
and the actual revenue from those
investments.
 Examples:
 4G (LTE)
 FTTC (fiber to the curb)
 FTTH (fiber to the home)
 One European carrier used Hadoop to
optimize the rollout of 4G coverage in
time and space to match the likely pick-
up in service revenue, based on detailed
cell tower traffic data of the last few
years.
 With their prior, less informed
approach, they would have had to
spend 10% more capex for the same
outcome.
Value Realized
Network
Care
Sales
New Biz

16. Network Infrastructure –
Network Upgrades
– 16 –
Business Problem
 Hadoop is used for targeted network
maintenance and upgrades by cable
companies.
 One large US cable MSO was unsure
how cable network congestion affects
churn, and where exactly network
upgrades produce the most incremental
revenue.
 The result was that only a small number
of nodes were responsible for the
majority of the negative customer
experience, and could therefore be
prioritized for upgrades.
Value Realized
Network
Care
Sales
New Biz

17. Hadoop in Network Infrastructure –
Network Upgrades Improve the Customer
Experience
– 17 –
• Correlate
network
congestion and
customer
experience
• 11 different data
sources
• 4m subscriber
records, 12m
work orders, 9m
calls, 42m
IPDRs, 20m
Tivoli NPMs
• Finding: Only a
few nodes
responsible for
most of the
negative
customer
experience
Network
Node
TNMP
CMTS
Performance
Network
Sensors
IPDR
Cable
Modem
Usage
Competitive
Spend
Data
Household
Household
Master
Subscriber
Record
Marketing
Demo-
graphics
Caller
Experience
Work Orders
Mobile
Devices
Customer
Premise
Equipment
Online
Transactions
Social Media
Interactions
SOURCE DATA
Network
Care
Sales
New Biz

18. Network Infrastructure –
Network Maintenance
– 18 –
Business Problem
 Radio access networks provide the air
interface between a mobile provider and
the end user mobile devices.
 Maintenance and repair of radio access
networks poses substantial logistical
challenges. In most countries, mobile
networks cover more than 95% of a
country’s surface area.
 Many transmission towers are in remote
and difficult to access locations.
 In high-density areas, pico- and femto-
cells optimize local coverage, but in turn
require coordination with the building
owner for maintenance.
 Hadoop improves a provider’s ability to
service equipment proactively, which is
always cheaper and less disruptive than
the replacement of equipment that has
already failed.
Value Realized
Network
Care
Sales
New Biz

19. Network Infrastructure –
Network Performance Management
– 19 –
Business Problem
 Existing network management platform
meant to diagnose poor cellular service
such as dropped calls or poor audio
quality.
 Overwhelmed by data volume, ingesting
10 million messages per second
 Each analysis was limited to a 24-hour
time window and only one-fiftieth the
surface area of the United States.
 Same customer issue may generate
multiple support calls, but the operator’s
team cannot see relationships between
multiple variables across time.
 Is the problem with the customer’s
device? Is it their neighborhood or
proximity to a tower? Is it because of
how they use their phone?
 With more history, they are able to
explore root causes that they have
never been able to identify by reviewing
just one day’s data, allowing them to to
improve cell phone service.
Value Realized
Network
Care
Sales
New Biz

20. Hadoop in Service and Security
– 21 –
Business Problem
 Customer experience analytics based
on call detail records (CDRs)
 Contact center productivity
 Field service productivity
 Data protection and compliance
 End-user device security
 With Hadoop, one operator detected
that 25% of callers were contacting the
call center merely to have their late fees
on the monthly bill waived. Clearly a
case for call deflection to interactive
voice recognition and online self-
service.
 Contact center agents had insufficient
ways of diagnosing what was wrong
with customers, leading to many
unnecessary truck rolls. Hadoop
helped avoid these.
 3% of smartphones account for 10-15%
of traffic because of malware (notably
on Android phones) and some fair use
violations. Hadoop helps detect that so
operators can take remedial action.
Value Realized
Network
Care
Sales
New Biz

21. Service and Security –
Customer Experience Analytics Based on
Call Detail Records (CDRs)
– 22 –
Business Problem
 A typical mobile service provider
generates >1 billion CDRs per
day, ingesting millions of CDRs per
second.
 System holds >100 billion records, half
a petabyte added every month!
 Due to the cost of existing solutions, the
data expires after 60 days
 CDRs need to be analyzed and archived
for compliance, billing and congestion
monitoring.
 Example: forensics on dropped calls
and poor sound quality.
 High volume makes pattern recognition
and root cause analysis difficult.
 Often those need to happen in real-
time, with a customer waiting for
answers.
 With Hadoop the carrier can to retain
some data for up to three years
 Hadoop provides both a cost advantage
– Hadoop provides storage 20x cheaper
than enterprise-grade storage – and
better insights.
 Better analysis to continuously improve
call quality, customer satisfaction and
servicing margins.
Value Realized
Network
Care
Sales
New Biz

22. Service and Security –
Contact Center Productivity
– 23 –
Business Problem
 A US-based mobile provider struggled
with a combination of high costs but low
customer satisfaction related to
customer care.
 An increasing share of support cases
are related to mobile data usage and
associated charges.
 Traditionally, contact center agents did
not have granular insights into a
particular customer’s data usage, hence
were unable to provide effective call
resolution.
 With Hadoop, one operator detected
that 25% of callers were contacting the
call center merely to have their late fees
on the monthly bill waived.
 The provider was able to off-load these
cases to online self-service and
interactive voice recognition.
 Frees up the agents to focus on more
valuable customer interactions.
 The provider is now extending this
solution to focus on issue resolution.
Value Realized
Network
Care
Sales
New Biz

23. Service and Security –
Field Service Productivity
– 24 –
Business Problem
 A provider’s contact center agents had
insufficient ways of diagnosing what was
wrong with customers, leading to many
unnecessary truck rolls.
 In particular, the agents were not able to
triage network vs. home-based
problems accurately enough.
 Therefore, technicians were dispatched
to the customer premises for problems
that reside within the network.
 The provider was able to avoid a large
number of “false positive” truck rolls.
 With each truck roll costing about $150
fully loaded, the provider was able to
save several million dollars already in
the first year.
Value Realized
Network
Care
Sales
New Biz

24. Service and Security –
End User Device Security
– 26 –
Business Problem
 A mobile operator needed to identify
real-time malware threats from non-
trusted application stores and contain
their impact on customers.
 3% of smartphones account for 10-15%
of traffic because of malware (notably
on Android phones) and some fair use
violations.
 Hadoop helps detect that so operators
can take remedial action, thus
eliminating a disproportionate share of
network tonnage.
 Options ranged from notifying an
affected customer all the way to
blocking certain URLs for the whole
network.
Value Realized
Network
Care
Sales
New Biz

25. Hadoop in Sales and Marketing
– 27 –
Business Problem
 360-degree view of customer value
 Personalized marketing campaigns
 Upselling and cross-selling
 Next-product-to-buy (NPTB)
 Churn reduction
 Telesales revenue increase by 50% by
tracking competitors web-sites visited
and counter offers to products searched
 +20% conversion rate increase by
optimizing and personalizing the path-
to-transaction
 $1.65 ARPU increase for 1 million
customers boosts topline by $20 million
per year.
 Reducing cable subscriber churn (“cord
cutting”). Every 100,000 subscribers
equates to customer lifetime value of
$1 billion
 Churn model quality increase
 Price related churn down by 40%
Value Realized
Network
Care
Sales
New Biz

26. Sales and Marketing –
360 Degree View of Customer Value
– 28 –
Business Problem
 Telcos and cable companies interact
with customers across many channels
and points in time.
 Data about those interactions is stored
in silos.
 Difficult to correlate data about
customer purchases, marketing
campaign results, and online browsing
behavior.
 Problem is exacerbated by recent
acquisitions and a proliferation in the
volume and type of customer data.
 Merging that data in a relational
database structure is slow, expensive
and technically difficult.
 Enterprise-wide data lake of several
petabytes
 360-degree unified view of the customer
(or household) life time value based on
usages patterns across time, products
and channels.
Value Realized
Network
Care
Sales
New Biz

27. Sales and Marketing –
Personalized Marketing Campaigns
– 29 –
Business Problem
 Marketers have long sought ways to
tailor their marketing campaigns to the
needs of each individual customer.
 Telcos are uniquely positioned to deliver
on that goal because mobile phones not
only follow their owners everywhere, but
also reveal a lot about their owners’
interests through browsing behavior and
the applications present on the phone.
 Telcos are looking for ways to mine that
information.
 Provider risked losing substantial
revenue as prepaid customers were
starting to switch to a competitor as a
result of a particularly effective
marketing campaign.
 The provider used Hadoop to pinpoint
those individual customers most at risk
of churning, and then built a highly
targeted campaign to retain the
remaining customers in that segment.
 A churn alarm system was established
and revenue leakage was minimized.
 Telesales revenue increase by 50% by
tracking competitors web-sites visited
and counter offers to products searched
 +20% conversion rate increase by
optimizing and personalizing the path-
to-transaction
 $1.65 ARPU increase for 1 million
customers boosts topline by $20 million
per year.
Value Realized
Network
Care
Sales
New Biz

28. Sales and Marketing –
Up-selling and Cross-selling
– 30 –
Business Problem
 The provider needed to find an
approach to upsell smart phones into a
user base that was still largely on legacy
feature phones.
 The operator converted many hundred
thousand feature phone users to smart
phones with associated data plans.
Value Realized
Network
Care
Sales
New Biz

29. Sales and Marketing –
Next Product to Buy (NPTB)
– 31 –
Business Problem
 As telco product portfolios grow more
complex, there are ever more
opportunities to sell additional services
to the same customer base.
 Many sales reps however are
overwhelmed with that complexity and
struggle to translate the breadth of the
product portfolio into incremental sales.
 Confident NPTB
recommendations, based on data from
all its customers, empower sales
associates and improve their
interactions with customers pre-
transaction.
Value Realized
Network
Care
Sales
New Biz

30. Sales and Marketing –
Churn Reduction
– 32 –
Business Problem
 A North American provider faced the
following challenge: 50% of new
customers churned off within 6 months
of acquisition.
 The average customer life time in this
segment was 13 months, well short of
the 18 months needed to break even.
 The provider increased the “right”
customer acquisitions by 27% and
decreased subsequent churn in this
segment by 50%.
 Price related churn down by 40%
 Reducing cable subscriber churn (“cord
cutting”). Every 100,000 subscribers
equates to customer lifetime value of
$1 billion
Value Realized
Network
Care
Sales
New Biz

31. Hadoop in New and Over-the-Top / Adjacent
Businesses
– 33 –
Business Problem
 New product development
 Actionable intelligence serving:
 Advertisers
 Merchants/retailers
 Payment processors
 Federal governments
 Local governments
 Hadoop-as-a-Service
 Telcos are well positioned to provide big
data as a service to retail, hospitality
and logistics customers. This can
generate $50-100m in annual revenue
for each medium-sized country.
Value Realized
Network
Care
Sales
New Biz

32. New and Adjacent Businesses –
New Product Development
– 34 –
Business Problem
 Mobile devices produce large amounts
of data about where, when, how and
why they are used.
 This data is extremely valuable for
product managers, yet much of it is out
of reach. Either it is never captured or
never converted into business insight.
Its volume and variety make it difficult to
ingest, store and analyze at scale.
 One provider who logged 27m devices
with more than 1bn events per month
has developed more than 20 projects
and pilots within 18 months after
launch, leading to increased revenue
and profitability.
Value Realized
Network
Care
Sales
New Biz

33. New and Adjacent Businesses –
Actionable Intelligence Serving Advertisers
– 35 –
Business Problem
 Europe’s leading real estate
marketplace Scout24 – a subsidiary of
Deutsche Telekom – features more than
one million properties for rent or sale at
any given time, and has facilitated more
than 20 million property transactions
over the last few years.
 The company wanted to drive more
market share to Scout24 by offering
advertisers – typically real estate agents
and brokers – an even better service.
 A small team consisting of a product
manager, a data scientist and a few
developers was able to make a
meaningful contribution to revenue
growth.
Value Realized
Network
Care
Sales
New Biz

34. Big Data as a Product:
ImmobilienScout (Deutsche Telekom)
– 36 –
Network
Care
Sales
New Biz

35. New and Adjacent Businesses –
Actionable Intelligence Serving Merchants
– 37 –
Business Problem
 A French mobile service provider is a
great example for how location
information per customer segments can
be used to optimize promotions and
point-of-sale locations of bricks-and-
mortar retailers.
 The retailers were able to increase their
reported same-store-sales through
better campaign management and in-
store optimizations. They also gained
valuable insights to optimize their store
network.
Value Realized
Network
Care
Sales
New Biz

36. New and Adjacent Businesses –
Actionable Intelligence Serving
Payment Processors
– 38 –
Business Problem
 Credit card issuers experience
increasing fraud when their card
members are travelling abroad.
 95% of travelers opted into the SMS
alerting service, resulting in a
substantial decrease in fraud related to
card use in foreign countries.
Value Realized
Network
Care
Sales
New Biz

37. New and Adjacent Businesses –
Actionable Intelligence Serving
Federal Governments
– 39 –
Business Problem
 The Eastward expansion of the
European Union has resulted in a longer
and more porous border to non-EU
member states.
 This has made it more difficult to protect
the EU against a stream of illegal goods
and refugees, which often travel over
land from the EU’s Eastern and South-
Eastern neighbors.
 Law enforcement agencies are able to
target their scarce resources much
more effectively, for instance choosing
to intercept suspicious cars traveling in
certain directions at speeds above
130km/h.
 This radically increases their hit rate per
mission.
Value Realized
Network
Care
Sales
New Biz

38. New and Adjacent Businesses –
Actionable Intelligence Serving
Local Governments
– 40 –
Business Problem
 In a large French city, traffic to large
events regularly caused massive
congestion on the city’s streets and
highways.
 The city identified and implemented
dozens of specific traffic management
measures, relieving congestion around
major events.
 They are also exploring how to use
these insights for environmental impact
studies, city planning and disaster
management.
Value Realized
Network
Care
Sales
New Biz

39. • Makes capital investments more efficient
• Leads to a better customer experience
• Lowers churn
• Increases conversions
• Strengthens security
• Opens up new markets
Hadoop Drives Business Outcomes for the
World’s Telcos and Cable Companies!
– 41 –

40. Questions?
Email juergen@techalpha.com for a copy of the presentation.
LinkedIn: juergenurbanski
Download 200-page BITKOM / Forrester Guide to Big Data
Technologies (in German):
http://www.bitkom.org/files/documents/BITKOM_Leitfaden_Big-Data-
Technologien-Wissen_fuer_Entscheider_Febr_2014.pdf