Demography estimation case study presented at Data Day Texas (http://datadaytexas.com/). Demonstrates how to apply graph analytics on telecom data.

1. Lynx Analytics © Confidential 1
Lynx Analytics © Confidential
Demography Estimation from Graph
A Large Telco Social Network Case Study
Gergely Svigruha – gergely.svigruha@lynxanalytics.com
Data Day Texas, Austin, 01/14/2017

2. Lynx Analytics © Confidential 2
Ideation
Academic origin with an
INSEAD professor and two
PHD students with
Mathematical and Computer
Science background
2010
Incorporation
LynxKiteTM
2013
R&D development team in
Hungary
2016
Opened office in
New York
2016
Capital increase with
strategic investor
Lynx Analytics History

3. Lynx Analytics © Confidential 3
Our team presence
Data Science
Lynx’s team of data scientists ensures the most
advanced technologies developed for our clients
Infrastructure & Deployment
Dedicated infrastructure engineers with extensive
experience in enterprise deployment
Banking & Telecom expertise
Key experienced experts from Telecom and Banking
industry enabling strict alignment to clients’ needs.

4. Lynx Analytics © Confidential 4
Estimate demographic data of telecommunication clients
• Demography data (e.g. age, gender or level of education) is invaluable for Telco companies for product design,
direct marketing or churn prevention
• However, the data is not always reliable or available, especially for prepaid customers
Task: Use existing Demography + Behavioral data to predict missing data
?
?, 30 Male, ?
Female, 40SMS SMS
Call

5. Lynx Analytics © Confidential 5
Social Networks can be very useful
• Many companies have graph data, they just do not know it!
o Customers call each other  they form a social graph!
• Social Networks can be extremely useful predictors of demography
o Friends / family members / co-workers resemble each other in some respect
?
?, 30 Male, ?
Female, 40SMS SMS
Call
Likely to be of the same age as
other community members:
35-40

6. Lynx Analytics © Confidential 6
We developed a scalable comprehensive graph platform: LynxKiteTM
• Telcos can have a huge customer base, more than a 100mil users in the USA or Asia
• We had to scale up to hundreds of millions of nodes and billions of connections
• Lynx Analytics developed its own scalable graph analytics platform called LynxKite using
Hadoop / Spark

7. Lynx Analytics © Confidential 7
Lynx Analytics © Confidential
Case Study
Predict age from registration / usage data

8. Lynx Analytics © Confidential 8
Customer age was predicted from registration data + CDR
Task: Estimate age of 100 million customers
• Registration data available for some but unreliable
• Call Data Records (CDR) available
• Small, but accurate survey data can be used as ground truth
Messy
Registration Data
Cleansed
Registration Data
Model EvaluationSocial Graph + Demography Data
as attributes

9. Lynx Analytics © Confidential 9
Registration data had to be cleansed
• Registration data can be unreliable due to fake information, missing fields or default values
• Select reliable records using simple predictive models and text mining heuristics
Name Address ID number Date of Birth
John Doe New York, … 342-69-7465 1983.03.11
asdasd qweqwe 123 1980.11.11
Jane Doe Boston,… 167-52-9274 1962.06.21
John Smith Austin,… 926-25-6284
Jane Smith San Jose,… 382-34-7622 1970.01.01
Missing
Data
Fake
Data
Default
Value
Reliable
Data
Reliable
Data

10. Lynx Analytics © Confidential 10
Age was stamped for 30M people from registrations with 82% accuracy
An age prediction is “correct” if the prediction is within a 5 years, otherwise incorrect
82%
accuracy
Out of the 100mill customers, 30mill is pre-stamped from registration

11. Lynx Analytics © Confidential 11
We created Social Graph from calls, SMS and location
• Every call and text message infers a social relationship
o Frequency, duration can be used as weights
• Approximate location (cell tower) is often recorded for calls, texting or browsing
o People who often appear at the same location around the same time probably know each other
Calls at
2016/06/21, 12:00
Texts at
2016/06/02, 07:00
Tower 946
2016/06/15, 15:00
Co-location
SMSCall

12. Lynx Analytics © Confidential 12
Age can be predicted from graph neighborhood
Original age: 40 Original age: 35 Original age: 45
Predicted age: 40
Small coverage in
Iteration 1: skip
Original age: 40
Original age: 35 Original age: 45
Predicted age
(Iteration 1): 40
Predicted age
(Iteration 2): 40
Iterative Neighborhood Methods
Only predict where we have enough values +
small deviation and iterate
Simple Neighborhood Methods
Predict missing values from neighborhood
median / average

13. Lynx Analytics © Confidential 13
Communities can be detected from Social Graph structure
Maximal Cliques
Overlapping
Cliques can be merged if they overlap enough,
based on INSEAD Research Paper  Infocom
communities
Modular Clustering
Non overlapping, only one community per person

14. Lynx Analytics © Confidential 14
Infocom Communities resemble real life the most
Infocom communities are merged maximal cliques
• Maximal cliques are too rigid
(not all relationships are captured in CDR)
• We have multiple communities in real life
(friends, family, classmates, co-workers)
 Real communities overlap too

15. Lynx Analytics © Confidential 15
Age predicted from Community Average / Median
• Take communities with enough defined ages
and small deviation
• If multiple available, pick the one with the
lowest standard deviation
o “Classmates” works better for age than “family”
• Predict from community average / median
• Iterate
Original age: 45
Original age: 40
Original age: 55
Predicted age: 42 Original age: 60
Original age: 20
Smaller Deviation
 Better Predictor

16. Lynx Analytics © Confidential 16
Geo-partition of the graph improved community algorithm
• Finding maximal cliques / communities is
computationally hard if the graph is too large
• Graph partitioning can help, but it is a difficult
problem in itself
o Heuristic: Partition using geolocation info,
most of people’s friends are physically close too
• For example a geo induced partition containing 20mil
nodes preserved 85% of the edges associated with
those nodes
• Much better than random partitioning of same sizes
(20% of edges preserved)

17. Lynx Analytics © Confidential 17
A combination of methods resulted in an overall 70% accuracy
<<
Neighborhood methods Community methods
Best community method: ~72% accuracy,
but not available for all customers
• Some people have too few connections, or their
“friends” may not know each other
Best simple neighborhood method: ~62% accuracy
Best iterative neighborhood method: ~67% accuracy

18. Lynx Analytics © Confidential 18
Further Research
Predict demography from
Social Graph variables
Integrate Graph Analytics &
Deep Learning methods
GraphAI

19. Lynx Analytics © Confidential 19
Social Graphs can help predicting marital status, level of education or occupation
Members of communities with a high ratio of university graduates are likely to be university
graduates themselves, the same applies to single people
Real estate agents tend to have low clustering coefficient, their “friends” (business connections), do
not necessarily know each other
Entrepreneurs have larger than average PageRank (often used to measure influence) or centrality

20. Lynx Analytics © Confidential 20
GraphAI
Integrating Graphs with Deep Learning
• Neural networks are graphs themselves
• Lynx’s R&D lab is currently working on a cutting edge technology integrating graph analytics and deep
learning methods to predict missing variables from existing ones / graph structure
Age: 35
Gender: Female
Edu: MSc
Age: ?
Gender: Male
Edu: BA
Age: ?
Gender: Female
Edu: ?

21. Lynx Analytics © Confidential 21
Lynx Analytics © Confidential
gergely.svigruha@lynxanalytics.com
https://www.lynxanalytics.com/
Thank you