Case Study : MVNO MVNE Data Warehouse Transatel (2008 version)

Case study : MVNO / MVNE Data Warehouse Seminar for EFREI engineering school Nov 2008 Mathieu DESPRIEE – BI Manager [email_address] www.transatel.com

Agenda

Part 1 – MVNO / MVNE / Transatel

Part 2 – Data-warehouse & Business Intelligence :

Emergence of the need

Part 3 – Business Intelligence :

Tools & concepts

Part 4 – Project organization

Part 1 MVNO / MVNE / Transatel

Introduction / Definitions

MVNO : Mobile Virtual Network Operator

An MVNO is a Service Provider offering a mobile service under its own brand without having a mobile radio network

GSM operators are referred to as MNO : Mobile Network Operator

An MNO has a license on radio frequency ranges (GSM, UMTS…), and all the radio infrastructure

MVNE : MVNO-Enabler

An MVNE is a service company delivering tools & services to companies wishing to market their services over a mobile network, and thus becoming MVNOs

MVNO and MVNE responsibilities Radio Network Core Network OSS BSS Customer Service Life-cycle Management Marketing Distribution

GSM

GPRS

UMTS…

HLR

SMS-C

MSC

GGSN

Interco …

Service activation

CDR collect

CRM

Billing

Processes

Operations

Call-centre

Support…

Brand

Offer definition

Communicat°

Marketing mix

Sales

Distribution network animation

MVNO responsibility MNO responsibility MVNE capabilities

Transatel Company introduction

Transatel

Transatel is an European MVNE (Mobile Virtual Network Enabler)

Founded in August 2000

Backed with institutional funds such as Natexis and Credit Agricole

80+ employees

HQ in France with subsidiaries in UK, Belgium, Netherlands and Luxembourg

Serving MVNOs & GSM operators

Established brands and businesses: turn key solution to become MVNO

GSM operators: infrastructure & services to easily integrate new MVNO and/or build specific co-branded offer

A innovative approach to mobile service differentiation and USP

Team & Infrastructure

Dedicated team with know-how (business, regulatory, technical dev & op.)

IT & Telecom platforms

Several MVNO launches in several countries

Over 200k managed subscribers

Transatel skills & operational capabilities

Telecom Business Plan set-up: Consulting

Mobile offer definition & positioning, business plan

Interco model / cost structure / pricing

Regulatory issues

Relationship with GSM operators (MNO):

Airtime procurement in selected European countries

Support in negotiation

Mobile Service management: Development and Operations

Order entry / Activation / Back office / Customer Management / Call centre

CDR mediation / Real-time IN control / Billing (post-paid & prepaid)

Fraud detection / Money collection / Bad debt Management

Value added services: Development & Operations (incl. « live » prototyping)

SIM cards management, personalisation and procurement

Voicemail, Unified Messaging, IVR, Follow-me

WISP over GPRS

Fixed / mobile convergence – Voice / data convergence – GSM / Wifi (VoIP convergence)

M2M secured connectivity

Pan-European footprint with centralised & mutualised platform

Turn key solution for MVNOs in Europe from Marketing & Regulatory support to Technical Implementation & Operations

Service portfolio & Operations capabilities Customer Acquisition & Order Entry Provisioning (activations, portability) Billing Customer Care & CRM Value added services SIM cards Order-entry tool web enabled for end-users, distributors Prov. & workflow MNO, service platforms, billing system CRM system web enabled for call-centre & self care Prepaid reload management Prepaid engine IN based Post-perso process for small batches SIM profile definition for large batches SIM mngt & OTA server SIM applets Customisation & branding, call mngt, security Switch & Service node UMS, follow-me, call termination, SMS-GW WISP APN for user authentication & mobile portal mngt + WAP gtway + MMS + remote handset configuration HLR / GMSC (roadmap) Marketing / Business / Regulatory support (offer definition, economic modelling, business processes, regulatory requirements) Logistics (3 rd party) Call centre (3 rd party) Post-paid, invoicing, balance management Payment Collection Revenue Assurance

Transatel interconnection with GSM operators

Transatel MVNE operating models MVNO TRANSATEL MNO End-User MNO End-User MVNO MVNE - ASP E X E M P L E Aggregator TRANSATEL

Vision of a possible efficient market structure Mass market MVNOs Segmented / niche MVNOs Ethnic Sports SME Switchless resellers MVNA ( Aggregator ) M2M Media Distributor Fixed telco MVNE MVNE MVNE MVNO mass market MNO MVNO mass market MVNO niche MVNO niche MVNO niche MVNO niche MVNO niche MVNO niche MVNO niche MVNO niche MVNO niche MVNO niche

EasyBorder : Transatel offer for frequent travellers & cross- border population (FR, BE, NL, LU)

Dedicated offer for high roamers

30% to 100% price reduction on roaming

Better continuity & dedicated services

Commercialised in FR , BE , NL , LUX

Niche marketing

Direct marketing

Flyers & billboards in int’l train stations

Advertising in targeted newspapers & websites

Active referral program

Partnership with frequent traveller programs

Sales: telesales & web

www.transatel.com

Eurokeitai (post-paid) – France www.eurokeitai.com

Transatel international reach Roam Triple Wireless

IT systems to support CRM & billing Billing Customer Management Provisioning CRM (sales & mkt) Order-entry / Customer service Real-time rating IN call-control CDR mediation Prepaid top-up mngmt Post-paid mngmt & invoicing WWW IVR SMS PDF email WWW print-shop Bank interface Reporting & data w/h Customers Distributors MVNO back-office IT interface INAP / CAMEL CDR files from 3 rd parties GSM network Other P/F & systems MNO interface IN P/F Service P/F 3 rd parties, etc. Web Front End (O/E & C/S) API Account Status & Network features Trouble ticketing Subscribers & Catalogue Resources (SIM & MSISDN)  Accounting reports

Transatel Service infrastructure

Part 2 Data-warehouse & Business Intelligence : Emergence of the need

Reporting needs

Information is one of the biggest asset of the enterprise. Business users NEED to analyze that data.

Finances :

Analysis of revenues, costs, profitability, customer/supplier accounting, human resources, financial audits…

Revenue assurance

Analysis of costs and revenues, detection of revenue leaks

Fraud detection

Sales & marketing :

Sales analysis, orders evolution, customer bills, CRM reports, marketing channels efficiency, offers appropriateness, calls analysis (destination, duration, distribution along time)

CRM & Customer support :

Statistics on incidents, delay of incident resolution, hotline statistics…

Production

SLA control. Ex : 99% of billing data processed in 24h

CEO, Board

Overall performance dashboards, KPI = Key Performance Indicators…

Reporting needs are …

Repetitive

Finances and marketing will share lots of requirements, but expressed differently

Subtly different

Finances would like a report starting on Jan, 1 st .

Marketing would like the same report, starting on Dec, 10 th (Christmas offer…)

Complex

Marketing would like to know the revenues generated by an offer (billing data), but with differentiation of the different acquisition channels (CRM data)

Expressed by business users

Business vocabulary : they won’t use the name of table and columns !

They don’t care about the naming conventions in data-models

They don’t care about technical constraints

First approach

Custom SQL queries

Queries against the different databases of the different applications

Use Excel sheets (with embedded queries) to federate data, and produce charts

Use existing applications to deduce stats (from displayed grids in a search screen for instance…)

Enterprise IT « in real life »

Many systems

Many data sources

Many different data models

Different references

Duplicates

 Problems in data consolidation

Complexity

Queries are complex

Samples :

mvno_activation_monitoring.sql

mvno_dump_accounts.sql

mvno_financial_report.sql

process_monitoring.sql

This kind of queries :

Use many tables, sometimes from several databases, sometimes from several systems

Needs many indexes

Produce large result sets

Are long to execute

Volumes

1 MVNO = 10,000 to 100,000 subscribers

Around 200 CDRs (call data records) per subscriber per month

(Up to 800 CDRs per month for some specific offers with a lot of data traffic)

100,000 subs * 200 CDRs * 12 months = 240 M CDRs / year

800 bytes of storage per CDR (including indexes) =~ 190 GB of storage per MVNO per year

 doing a SELECT query over the whole data of the year (for instance, sum of all traffic) would require a lot of time of execution, and a lot of resources (disk, CPU) on production server

1 MVNO = 5 tariff-plans = 10,000 rates implemented in billing

Call destination analysis tables (numbering plans) = 600,000 destinations in the world

No reactivity to change

One report = one query  catastrophic !!

Each set of reports evolution needs a project organization :

specification by business,

technical specification (complexity to interpret business requirements)

testing,

move to production

Even for minor changes !!

OLTP models are not adapted

OLTP is the common modeling for business applications

transaction-oriented

data is highly normalized (cf. 3 normal forms)

OLTP models are not designed for large queries

Performance impact, database locking

OLTP models are not designed to maintain history

Often, OLTP systems store a snapshot of reality. For instance, they store the current status of some entity, but not all the history of statuses for that entity and the dates of each change.

Keeping all activity for a long time in OLTP systems would have an important impact :

Difficult to administrate indexes

Performance impact

Time to backup & restore database (because systems fail one day or another)

OLTP models are not adapted to reporting

Data normalization decompose data into many tables and joins, resulting in complex queries

The result :

90% of time spent in developing and producing the reports, 10% spent to analyze them

Use of user-interfaces not designed for reporting

Too few people know how to produce the key reports

Performance impact on production systems

Reports are incomplete, no sufficient history

“ I have data everywhere, and sometimes I forgot some data-source in my reports”

“ I get data from many different systems to fill-in my Excel sheet”

“ I’m not totally confident in my reports”

“ I make a lot of hypotheses in my simulation, but I cannot verify them”

All these problems = the field of BI

Difficulties in data acquisition

Need dedicated tools to collect data from different systems, with cleaning and consolidation capabilities

Difficulties in data storage with OLTP on production systems

Need different data-models to store history

Need different databases to store history

Need different engine to do the analysis and calculations

Difficulties in data restitution with SQL

Need tools usable by business users

BI = Business Intelligence = Field of technologies that facilitate acquisition, analysis and restitution of data to help in taking decisions

Part 3 Business Intelligence : Tools & concepts

Business Intelligence tools & concepts Production systems Other sources CRM Billing Activation systems Other DBs… Files ETL DataWarehouse OLAP cubes Business Users Querying & Reporting tools Acquisition Storage / Archiving Restitution Datamarts

Business Intelligence tools & concepts (2)

ETL

Extract : from various production data stores

Transform : transform data to match target data-model, merge the different sources

Load : in a target database (the data-warehouse)

Data Warehouse

Repository of the whole enterprise’s historical data

Subject oriented : business data

Time variant : every change is dated

Non-volatile : read-only, never deleted nor changed once committed

Integrated : related data from different production systems are gathered

Datamart

A small repository dedicated to a business subject.

Example : one datamart dedicated to cost &revenues analysis, another dedicated to traffic analysis.

In the case of Transatel, our data-warehouse is simply constituted of a set of datamarts.

OLAP cubes : explained next slide…

Reporting tools

Querying tools that can be used by business users, non computer specialists (such as excel !)

Reports, dashboards

OLAP & Datamarts

Datamart creation : the OLAP approach

Reorganize data in an hypercube

N-dimensional cubes

Facts : the data to be aggregated

Data quantifying the business activity (facts are also called “measures” in microsoft tools)

Price, revenue, cost, duration of calls, size of data packets, currencies exchange rates, number of visits on web-site, number of incidents…

Aggregation operations = sum, average, min, max…

Dimensions : the information structuring the facts

Time, time, time !

Customers, offers, currencies, countries, type of charge, type of call (SMS, voice, data), destinations of calls, marketing channels, …

Hierarchies :

Time : Year > Quarter > Month > Day

Geography : Zone > Country > Region > City

Organization : Business Unit > Line of business > Market

Time Geography Offers Cells contains facts data

OLAP : On-Line Analytical Processing

Cubes are the way OLAP technology organizes summary data into multidimensional structures.

Dimensions and their hierarchical levels reflect the queries that can be asked to the cube.

Aggregations are precalculated and stored in the multidimensional structure in cells at coordinates specified by the dimensions.

Sophisticated algorithms in modern OLAP engines minimize storage

Calculating all possible aggregations would require substantial processing time and storage, which is unnecessary. As a result, there is a tradeoff between storage requirements and the percentage of possible aggregations that are precalculated.

Different OLAP storage modes for different needs :

MOLAP : multidimensional OLAP : fastest calculation, larger storage, not updated real-time

ROLAP : relational OLAP : smaller storage overhead, slowest calculation, real-time

HOLAP : hybrid mode

OLAP engines use specific querying language (MDX)

OLAP client tools generate MDX queries

MDX can also be written by developers for advanced querying

OK, I need a cube, where do I start ?

The star data-model (or snowflake)

Facts Dimension Dimension Dimension Dimension

Dimensions hierarchies

Example of hierarchy for a time dimension

The cube is configured so that the OLAP engine knows the hierarchy structure, and can precalculate aggregates at different dimension levels

Real world example 1

Cube dedicated to revenue analysis from post-paid billing

Advanced point :

Transatel billing produces bills for customers in several countries, and in several currencies.

Currency is a special point of interest, because we can’t sum prices in different currencies.

Currency conversion rate variations along time are stored in a dedicated fact table

A “reporting currency” dimension has been created

The business user chooses in which currency he wants the aggregates

Cube has been configured with advanced calculation rules to help in correct processing

Real world example 2 : cube for billed charges

The datamodel (snowflake)

The cube structure

The cube structure Main Facts (charges) Measures : charge, sum minutes, sum SMS, sum MBytes …

The cube structure Measure : Subscriber count

The cube structure Measure : {subscriber x month} counting an artificial measure to ease some computations conceptually it’s a crossjoin, in this case, it’s better to create it from the datamodel, with a dedicated SQL query

The cube structure Measure : the currency exchange rate .. to display the data in €, even if the bill is in another currency (conversion is done by an advanced MDX script)

The cube structure Dimensions the Date dimension is used several times (role-playing dimension)

The cube structure Relationships a « regular » relationship ie : a BilledCharge has a DateKey for the BillingDate dimension

The cube structure Relationships a « fact » relationship ie : the same table (subscriber) is used as a dimension (organizing attributes) and as a fact (for counting)

The cube structure Relationships a « referenced » relationship ie : the Country dimension is not directly related to facts. We have to go through another dimension (customer) to get it

The cube structure Relationships a « many-to-many » relationship ie : subscriber measures (count) are not related to products. We have to go through the billed charges to know that. This is a many-to-many join : Dim Product  Fact Charges Fact Charges  Dim Subscriber Dim Subscriber  Fact Subscriber

ETL

Extract

Multi-source

Batch oriented : usually ETL batches are ran when load is low on production systems (by night)

Connectivity to different technologies, to various DBRMS implementations

Transform

Manipulate complex data

Merge from different sources

Implement business rules.

Cleanup, sorting, deduplication…

Optimized for performance and large volumes

Load

Load data in fact tables and in dimension tables

Produce logs to files or database for traceability

Others

Controls execution flows : manage dependencies between tasks, control execution, manage errors, logs

Examples of transformation available in SSIS

Fuzzy Lookup Transformation

Look up values in a reference table using a fuzzy match

Example : the field “how did you heard about Transatel” is a free-text field on the website. The data contains a lot of variants for same words (“friend”, “a friend”, “freind”, etc). The fuzzy logic can help in merging all the variants

Slowly Changing Dimension Transformation

Help in managing a the update of a slowly changing dimension

Example : Information about the customers change continuously, but we want to keep some history.

A change in customer address  we don’t keep the old value (override)

A change in company VAT Number  we ‘duplicate’ the customer entry, to keep old financial data aligned with the old VAT Number (for annual reports)

This is called a surrogate key

ETL : control the execution !

Data is very sensitive, quality of reports rely on the quality of data

Loading a datawarehouse can imply very complicated rules on extraction, transfer and transformations

Business users won’t trust a datawarehouse, if they’re not confident on how it has been loaded.

So, one of the key aspects in ETL process implementation is the control of how runs the data transfer and transformation

ETL solutions come with a lot of tools to control process execution, and with alerting systems (eg: mail sent to the DBA) to raise the error.

Microsoft SSIS organize ETL packages in control-flow and data-flows.

Cf next slide

ETL package example in SSIS

Control flow and data flow

Restitution & Reporting

Reports topology Dashboards Adhoc reports Mass reporting executives operational teams

KPIs, trends

5% of users

data and trends analysis

need of interactivity

need of drill-down, drill-through

20% of users

static reports designed for specific purposes

75% of users

Reporting : OLAP cube browsing

Users can browse data through pivot charts

Colums and rows are taken amongst available dimensions

Content of pivot-chart is measures (attributes of facts)

Excel is a good low-end tool for cube browsing

Demo

Reporting : Control access to data

Security issues :

Security of access and segregation of data is important : anybody may not access to any data.

Especially in the case where reports are produced for external partners / customers : they can connect to a secured web-site and download reports they’re interested in.

Some business data may also be confidential, and destined to top-executives only.

Reporting tools and OLAP engines provide the ability to define roles and permission on data, at every level (dimension, attribute of dimension, cell, etc.)

Simplification for business users :

Cubes may contains a lot of dimensions, dimension attributes, and facts. In some tools, it’s possible to define a subset of a cube, and restrain some users to access only at this part of data (“perspectives” in microsoft tools)

Part 4 Project organization

Collect the business needs

The best way is to start from existing reports, and existing excel sheets frequently used by managers & executives

Organize reporting needs by functional domain, and if possible by data-source

Define at a macro level the model you need

Determine dimensions

Determine facts

Look at data access permissions

What users want

What they specify

“ Put all information you have, we will select things afterwards”

What you’ll manage to do

What they really need ( thanks to Yves Cointrelle for the joke idea)

Select requirements and organize phases

Talk with business users, again, again and again !

Organize requirements in a matrix of choice :

Eliminate complex and non-urgent needs, and organize project phases

complexity business priority P1 P2 P3

Define granularity of data

A radar diagram can help you to :

Identify the needed dimensions and their hierarchies

Decide the level of granularity you need to keep in your datamarts

Learnings from experience…

Dimension design : beware of cardinality !

Example : Time dimension

First design : one table for all granularities, from year to second

365*24*3600 = up to 31’536’000 rows by year !!!

This design was catastrophic on performance of everything : loading, pre-processing, querying, custom calculations (advanced MDX)

In fact, bad analysis of business need

Day granularity was enough for most of time dimension usage (date of bill-cycle, date of call, date of subscriber creation, etc.) = 365 rows per year

Second granularity was useful in a unique case : time of call placement, only 86400 rows and rarely used.

2 different time dimensions : a “Date” dimension (used a lot), and a “Time” dimension (used once in the model)

Result : Pre-processing time improved by a factor of 20…

Learnings from experience…

ETL flows design

6 first months of loading of the datawarehouse after each bill-cycle : the ETL flows NEVER finished successfully !

There are always little problems or unexpected inconsistencies in data

Restarting a whole ETL process is long and costly for hardware resources

Don’t hesitate to separate the 3 parts of E-T-L

Design your flows so you can restart a phase without restarting everything, using intermediary storage if necessary

Datamart development methodology

Be AGILE !

Estimate the quality of data

Estimate constraints on data (ETL rules)

Design core part of the data model

Show to the users what is possible

Help users to define what they need

Do a prototype if necessary (with manually loaded data)

Implement ETL flows, rules, and data clean-up

Implement the data model and the cubes

Produce the first reports

Control the results with pre-existing reports

Discuss the first results and problems with users

Adapt and improve ETL rules and flows

Improve cube model to adapt to users comments

Add the advanced computed indicators (MDX scripts)

Refine the aggregation designs to optimize cube performance from real usage statistics

Project methodology

Start small !!

Build progressively, use iterative approach with frequent delivery to production

Benefits : give a quick feedback to business users

Validate the technical architecture

Organize a weekly project committee

Involve final users

Get trained

Developers should be trained specifically on the chosen BI tools

Business users have to be trained to use the new tools (and avoid a reject)

Buy consulting from specialists of necessary

Issues not to under-estimate

It can take some time to stabilize the flows and the rules in the ETL

With some applications, data is never totally clean.

You may need additional rules or even some manual processing from time to time

You may spend some time in operations with the first versions of your cubes

To adapt the aggregation configuration from the real usage statistics

To adapt the indicators you compute to what the users want

You need to spend time in the modeling phase

With a bad model, cube may be technically impossible to optimize. You can even crash you server with a bad query

With a bad model, you can obtain false calculations

MDX is a very difficult language

Business users will become greedy of your reports !

One day behind schedule to produce a sensitive report, and all the management will be on your back !!

One false statistic about the business, and everybody will hate you … Do a lot of testing to stay confident !

Some vendors of BI software…

ETL

Informatica

IBM

SAS

Cognos

DataMirror

Microsoft SSIS

Sunopsis

Business Objects

Oracle

Open Source : Talend, Enhydra

Storage

Generalists : Oracle, Microsoft SQL Server

Specialized :

Red brick

Teradata

Sybase IQ

SAS

Netazza

Datallegro

Open Source : MySQL

OLAP

Microsoft Analysis Services

Hyperion

Oracle

SAP BW (dedicated to SAP solutions)

SAS

Reporting

Business Object / Crystal Report

Cognos

Hummingbird

Hyperion Solutions

Microsoft reporting services

Information Builders

Oracle

SAS

Informatica

MicroStrategy

Actuate

Open Source : JasperReports, Pentaho

Bibliography

Yves Cointrelle

BI training session – Homsys Group

Miscrosoft MSDN for Analysis Services

http://msdn2.microsoft.com/en-us/sql/Aa336310.aspx

Wikipedia

Thank you ! electronic version www.slideshare.net/mathieu.despriee feel free to contact me for any question [email_address]

Jobs and Internships at Transatel

Case Study : MVNO MVNE Data Warehouse Transatel (2008 version)

0 comments

Notes on slide 1

3 Favorites