11/06/2014
1
Using Passive Mobile Positioning
Data for Generating Statistics:
Estonian Experiences
Seminar. Statistics Finland
02.06.2014 Helsinki
Prof. Rein Ahas (University of Tartu)
http://mobilitylab.ut.ee/eng/
Objectives:
• BIG data as source for statistics?
• Use of Mobile Phone data for
statistical purposes
11/06/2014
2
Feasibility Study on the Use of
Mobile Positioning Data for Tourism
Statistics
Eurostat contract no. 30501.2012.001-2012.452
BIG DATA
11/06/2014
3
Do we need new data?
Can BIG data replace existing statistics?
Can we trust secondary BIG data?
Privacy…
ICT revolution - fastest change in
human behaviour
11/06/2014
4
ICT is changing society
(Sheller & Urry 2006):
• More communication = more travel
• More information = more spatial mobility
It is not possible to understand and
govern contemporary society
without digital information layers
- Quantitative
- Qualitative
11/06/2014
5
The Global Database of Events,
Language, and Tone (GDELT)
Georgetown University, Washington DC
http://gdeltproject.
org/
Do we think like:
„data managers“ – is there need to replace
traditional data with new BIG sources?
„end-users“ - what kind of data is needed
for managing this „new“ society?
11/06/2014
6
F1 success in1990:
budget, car, driver…
F1 success in 2014:
budget, sensors, …
11/06/2014
7
Paradigm shift:
• Intelligent transportation systems
• Smart City
• Monitoring systems
Scheveningen Memorandum „Big
Data and Official Statistics“ DGINS
1. Acknowledge that Big Data represent new
opportunities and challenges for Official
Statistics,
and therefore encourage the European Statistical
System and its partners to effectively examine
the potential of Big Data sources in that regard.
• EUROSTAT Task Force ‘Big Data and Official
Statistics’
Director Generals of the National Statistical Institutes (DGINS)
11/06/2014
8
Mobile phone data
11/06/2014
9
I Active Positioning
Locating phone with special Query:
„find“ „ask“ „record“…
Requires approval from the phone owner
Smartphone based questionnaires
• Tracking locations
• Recording sensor data
• Movement
• Phone use
• Noise
• …
• Asking questions in
phone
11/06/2014
10
II Passive mobile positioning
Memory files of Mobile Network Operator (MNO)
Call Detail Record (CDR), Data Detail Record
(DDR)…
Passive Positioning
Subscriber Activity Time Cell
3725264020 SMS 07.04.2014 12:15:00 43879
244121965188 Call 07.04.2014 12:15:01 43879
206201963365 SMS 07.04.2014 12:15:01 44866
244121965188 Data 07.04.2014 12:15:04 43879
244121965188 Call 07.04.2014 12:15:04 43879
244211964246 Data 07.04.2014 12:15:05 43877
244121965188 Call 07.04.2014 12:15:07 43879
24405239944 SMS 07.04.2014 12:15:08 48512
244211548784 Call 07.04.2014 12:15:11 48987
244121964444 Call 07.04.2014 12:15:14 45559
244051604891 Data 07.04.2014 12:15:15 45601
24201725641 SMS 07.04.2014 12:15:15 45463
244051965315 Data 07.04.2014 12:15:17 48987
244211963912 Call 07.04.2014 12:15:20 43570
244051605773 Data 07.04.2014 12:15:20 35550
244211914278 Data 07.04.2014 12:15:23 48987
24421417297 Call 07.04.2014 12:15:26 48987
24421838967 Data 07.04.2014 12:15:28 43951
244051965316 SMS 07.04.2014 12:15:29 43909
Antenna ID
Subscriber ID
11/06/2014
11
ONE MONTH OF DATA
150M records / month
Transportation studies
11/06/2014
12
Studying individual mobility:
Movement of University professor in Estonia
2007-2013
 Second home
439 days
 Work 1000
Movement of professor in world 2007-2013
694 days
34 states
11/06/2014
13
Generating transportation data
from Call Detail Records
Passive mobile
positiong data
Transportation
zones
Movement
vectors
Anchor points
model
Characterised
movements
Reference data
Penetration
model
Corrected
movements
OD-matricies and
temporal & social
coeficents
Modelling traffic
flows
30.11.2009 26 Erki Saluveer
11/06/2014
14
OD-Matrices -> transportation model
Publications in transportation
studies
Järv, O., Ahas, R. and Witlox, F. 2014. Understanding monthly variability in human
activity spaces: a twelve-month study using mobile phone call detail records.
Transportation Research Part C: Emerging Technologies 38 (1): 122–135.
Saluveer E, Ahas, R. 2014. Using Call Detail Records of Mobile Network Operators
for transportation studies, In Timmermans H. & Rasouli S. (eds.) Mobile
Technologies for Activity-Travel Data Collection & Analysis, IGI Global.
Jarv. O., Ahas, Saluveer, E., Derudder, B., Witlox, F. 2012. Mobile Phones in a
Traffic Flow: A Geographical Perspective to Evening Rush Hour Traffic Analysis
Using Call Detail Records, PLoS ONE 7(11),
http://dx.plos.org/10.1371/journal.pone.0049171
Ahas, R., Silm, S., Järv, O., Saluveer E., Tiru, M. 2010. Using Mobile Positioning
Data to Model Locations Meaningful to Users of Mobile Phones , Journal of Urban
Technology, 17(1): 3-27.
Ahas, R. Aasa, A., Silm, S., Tiru, M. 2010. Daily rhythms of suburban commuters’
movements in the Tallinn metropolitan area: case study with mobile positioning
data. Transportation Research C, 18: 45–54.
11/06/2014
15
Urban studies
Ethnic segregation studies:
 Russian-speaking people visit a smaller number of
districts than Estonians when travelling in Tallinn, in
Estonia and abroad.
Tallinn Estonia
(excluding
Tallinn)
Foreign
countries
Estonians 16.7 19.3 2.04
Russians 16.6 10.6 1.68
Difference with
language only (ref.
Estonian)
-0.189** -8.707*** -0.362***
Difference with other
characteristics (ref .
Estonian)
0.021 -8.157*** -0.117**
11/06/2014
16
Temporal segregation in City:
 Ethnic groups are more unevenly distributed in the evenings.
 Probability of interethnic contacts are higher on working hours (10-16).
 Ethnic groups are more unevenly distributed on residential areas than
on working hours.
Segregation in social networks:
11/06/2014
17
Publications in Urban Studies
Silm, S. & Ahas, R. 2014.The temporal variation of ethnic segregation in a city:
evidence from a mobile phone use dataset, Social Science Research 47: 30-43.
http://dx.doi.org/10.1016/j.ssresearch.2014.03.011
Silm, S. & Ahas, R. 2014. Ethnic differences activity spaces: The study of out-of-home
non-employment activities with mobile phone data, Annals of Association of American
Geographers 104(5): 542-559.
http://dx.doi.org/10.1080/00045608.2014.892362
Novak, J., Ahas, R., Aasa, A., Silm, S. 2013. Application of mobile phone location data
in mapping of commuting patterns and functional regionalization: a pilot study of
Estonia, Journal of Maps 9(1): 10-15.,
http://dx.doi.org/10.1080/17445647.2012.762331
Silm, S., Ahas, R., Nuga, M. 2013. Gender differences in space-time mobility patterns
in a post-communist city: a case study based on mobile positioning in the suburbs of
Tallinn. Environment and Planning B: Planning and Design 40(5) 814 – 828.
Silm,S., Ahas, R., 2010. 'The seasonal variability of population in Estonian
municipalities, Environment and Planning A, 42(10) 2527-2546.
Tourism data
11/06/2014
18
Balance of Payments – Travel Item
Monthly international travel statistics for
Balance of Payment calculations
Country level
Inbound and outbound (to and from Estonia)
Data since 2009
Inbound Travel
Indicators:
• Number of visits
• Number of days spent
• Number of nights spent
Breakdown:
• Country of origin
• Estonia as transit / destination
• Same-day / overnight visit
• Tourist / long-term visitor (resident)
11/06/2014
19
Outbound Travel
Indicators:
• Number of trips / visits
• Number of days spent
• Number of nights spent
Breakdown:
• Total abroad / specific country
• Country as transit / destination
• Same-day / overnight visit
• Tourist / long-term visitors (non-residents)
11/06/2014
20
Data Transfer
Monthly transfer of CSV files with prepared
Excel pivot tables
11/06/2014
21
11/06/2014
22
11/06/2014
23
Permanent
Transit
Visitor
Permanent
Temporary
Foreigner
11/06/2014
24
Publications in Tourism Studies:
Nilbe, K., Ahas, R., Silm, S. 2014. Evaluating the Travel Distances of Events and Regular
Visitors using Mobile Positioning Data: The case of Estonia, Journal of Urban Technology
21(2):
Kuusik, A., Tiru, M., Varblane, U., Ahas, R. 2011. Process innovation in destination
marketing:
use of passive mobile positioning (PMP) for segmentation of repeat visitors in case of
Estonia, Baltic Journal of Management 6(3): 378 – 399.
Tiru, M., Kuusik, A., Lamp, M-L., Ahas, R. 2010. LBS in marketing and tourism
management: measuring destination loyalty with mobile positioning data. Journal of
Location Based Services, 4(2): 120-140.
Ahas, R. 2010. Mobile positioning data in geography and planning, Editorial. Journal of
Location Based Services, 4(2): 67-69.
Tiru, M., Saluveer E., Ahas, R., Aasa, A. 2010. Web-based monitoring tool for assessing
space-time mobility of tourists using mobile positioning data: Positium Barometer.
Journal of Urban Technology, 17(1): 71-89.
Ahas, R. Aasa, A., Roose, A., Mark, Ü., Silm, S. 2008. Evaluating passive mobile
positioning data for tourism surveys: An Estonian case study. Tourism Management
29(3): 469–486.
Ahas, R., Aasa, A., Mark, Ü., Pae, T., Kull, T. 2007. Seasonal tourism spaces in Estonia:
case study with mobile positioning data. Tourism Management 28(3): 898–910.
Conclusions
11/06/2014
25
Conclusions I:
• Timeliness – fast data collection, digital
processing, automatic
• Better spatial and temporal accuracy
• Longitudiness – covering longer time period
and area
• …
Conclusions II
• Access to data complicated, privacy…
• Missing information about users,
purpose of trips, expenditures
• Sampling issues
• …
11/06/2014
26
Conclusions III
• Replacing existing data with BIG data?
• Improving existing data with BIG data?
• Collecting data about new aspects of social life?
• NEW PRODUCTS and CONSUMER GROUPS,
monitoring, real-time…
Thank you!
rein.ahas@ut.ee
Silm, S. & Ahas, R. 2014.The temporal variation of ethnic segregation in a
city: evidence from a mobile phone use dataset, Social Science Research
47: 30-43. http://dx.doi.org/10.1016/j.ssresearch.2014.03.011
Silm, S. & Ahas, R. 2014. Ethnic differences activity spaces: The study of
out-of-home non-employment activities with mobile phone data, Annals of
Association of American Geographers 104(5): 542-559.
http://dx.doi.org/10.1080/00045608.2014.892362
11/06/2014
27
„BIG data as mirror“ – society is trying to
understand fast changes

Using Passive Mobile Positioning Data for Generating Statistics: Estonian Experiences, Rein Ahas

  • 1.
    11/06/2014 1 Using Passive MobilePositioning Data for Generating Statistics: Estonian Experiences Seminar. Statistics Finland 02.06.2014 Helsinki Prof. Rein Ahas (University of Tartu) http://mobilitylab.ut.ee/eng/ Objectives: • BIG data as source for statistics? • Use of Mobile Phone data for statistical purposes
  • 2.
    11/06/2014 2 Feasibility Study onthe Use of Mobile Positioning Data for Tourism Statistics Eurostat contract no. 30501.2012.001-2012.452 BIG DATA
  • 3.
    11/06/2014 3 Do we neednew data? Can BIG data replace existing statistics? Can we trust secondary BIG data? Privacy… ICT revolution - fastest change in human behaviour
  • 4.
    11/06/2014 4 ICT is changingsociety (Sheller & Urry 2006): • More communication = more travel • More information = more spatial mobility It is not possible to understand and govern contemporary society without digital information layers - Quantitative - Qualitative
  • 5.
    11/06/2014 5 The Global Databaseof Events, Language, and Tone (GDELT) Georgetown University, Washington DC http://gdeltproject. org/ Do we think like: „data managers“ – is there need to replace traditional data with new BIG sources? „end-users“ - what kind of data is needed for managing this „new“ society?
  • 6.
    11/06/2014 6 F1 success in1990: budget,car, driver… F1 success in 2014: budget, sensors, …
  • 7.
    11/06/2014 7 Paradigm shift: • Intelligenttransportation systems • Smart City • Monitoring systems Scheveningen Memorandum „Big Data and Official Statistics“ DGINS 1. Acknowledge that Big Data represent new opportunities and challenges for Official Statistics, and therefore encourage the European Statistical System and its partners to effectively examine the potential of Big Data sources in that regard. • EUROSTAT Task Force ‘Big Data and Official Statistics’ Director Generals of the National Statistical Institutes (DGINS)
  • 8.
  • 9.
    11/06/2014 9 I Active Positioning Locatingphone with special Query: „find“ „ask“ „record“… Requires approval from the phone owner Smartphone based questionnaires • Tracking locations • Recording sensor data • Movement • Phone use • Noise • … • Asking questions in phone
  • 10.
    11/06/2014 10 II Passive mobilepositioning Memory files of Mobile Network Operator (MNO) Call Detail Record (CDR), Data Detail Record (DDR)… Passive Positioning Subscriber Activity Time Cell 3725264020 SMS 07.04.2014 12:15:00 43879 244121965188 Call 07.04.2014 12:15:01 43879 206201963365 SMS 07.04.2014 12:15:01 44866 244121965188 Data 07.04.2014 12:15:04 43879 244121965188 Call 07.04.2014 12:15:04 43879 244211964246 Data 07.04.2014 12:15:05 43877 244121965188 Call 07.04.2014 12:15:07 43879 24405239944 SMS 07.04.2014 12:15:08 48512 244211548784 Call 07.04.2014 12:15:11 48987 244121964444 Call 07.04.2014 12:15:14 45559 244051604891 Data 07.04.2014 12:15:15 45601 24201725641 SMS 07.04.2014 12:15:15 45463 244051965315 Data 07.04.2014 12:15:17 48987 244211963912 Call 07.04.2014 12:15:20 43570 244051605773 Data 07.04.2014 12:15:20 35550 244211914278 Data 07.04.2014 12:15:23 48987 24421417297 Call 07.04.2014 12:15:26 48987 24421838967 Data 07.04.2014 12:15:28 43951 244051965316 SMS 07.04.2014 12:15:29 43909 Antenna ID Subscriber ID
  • 11.
    11/06/2014 11 ONE MONTH OFDATA 150M records / month Transportation studies
  • 12.
    11/06/2014 12 Studying individual mobility: Movementof University professor in Estonia 2007-2013  Second home 439 days  Work 1000 Movement of professor in world 2007-2013 694 days 34 states
  • 13.
    11/06/2014 13 Generating transportation data fromCall Detail Records Passive mobile positiong data Transportation zones Movement vectors Anchor points model Characterised movements Reference data Penetration model Corrected movements OD-matricies and temporal & social coeficents Modelling traffic flows 30.11.2009 26 Erki Saluveer
  • 14.
    11/06/2014 14 OD-Matrices -> transportationmodel Publications in transportation studies Järv, O., Ahas, R. and Witlox, F. 2014. Understanding monthly variability in human activity spaces: a twelve-month study using mobile phone call detail records. Transportation Research Part C: Emerging Technologies 38 (1): 122–135. Saluveer E, Ahas, R. 2014. Using Call Detail Records of Mobile Network Operators for transportation studies, In Timmermans H. & Rasouli S. (eds.) Mobile Technologies for Activity-Travel Data Collection & Analysis, IGI Global. Jarv. O., Ahas, Saluveer, E., Derudder, B., Witlox, F. 2012. Mobile Phones in a Traffic Flow: A Geographical Perspective to Evening Rush Hour Traffic Analysis Using Call Detail Records, PLoS ONE 7(11), http://dx.plos.org/10.1371/journal.pone.0049171 Ahas, R., Silm, S., Järv, O., Saluveer E., Tiru, M. 2010. Using Mobile Positioning Data to Model Locations Meaningful to Users of Mobile Phones , Journal of Urban Technology, 17(1): 3-27. Ahas, R. Aasa, A., Silm, S., Tiru, M. 2010. Daily rhythms of suburban commuters’ movements in the Tallinn metropolitan area: case study with mobile positioning data. Transportation Research C, 18: 45–54.
  • 15.
    11/06/2014 15 Urban studies Ethnic segregationstudies:  Russian-speaking people visit a smaller number of districts than Estonians when travelling in Tallinn, in Estonia and abroad. Tallinn Estonia (excluding Tallinn) Foreign countries Estonians 16.7 19.3 2.04 Russians 16.6 10.6 1.68 Difference with language only (ref. Estonian) -0.189** -8.707*** -0.362*** Difference with other characteristics (ref . Estonian) 0.021 -8.157*** -0.117**
  • 16.
    11/06/2014 16 Temporal segregation inCity:  Ethnic groups are more unevenly distributed in the evenings.  Probability of interethnic contacts are higher on working hours (10-16).  Ethnic groups are more unevenly distributed on residential areas than on working hours. Segregation in social networks:
  • 17.
    11/06/2014 17 Publications in UrbanStudies Silm, S. & Ahas, R. 2014.The temporal variation of ethnic segregation in a city: evidence from a mobile phone use dataset, Social Science Research 47: 30-43. http://dx.doi.org/10.1016/j.ssresearch.2014.03.011 Silm, S. & Ahas, R. 2014. Ethnic differences activity spaces: The study of out-of-home non-employment activities with mobile phone data, Annals of Association of American Geographers 104(5): 542-559. http://dx.doi.org/10.1080/00045608.2014.892362 Novak, J., Ahas, R., Aasa, A., Silm, S. 2013. Application of mobile phone location data in mapping of commuting patterns and functional regionalization: a pilot study of Estonia, Journal of Maps 9(1): 10-15., http://dx.doi.org/10.1080/17445647.2012.762331 Silm, S., Ahas, R., Nuga, M. 2013. Gender differences in space-time mobility patterns in a post-communist city: a case study based on mobile positioning in the suburbs of Tallinn. Environment and Planning B: Planning and Design 40(5) 814 – 828. Silm,S., Ahas, R., 2010. 'The seasonal variability of population in Estonian municipalities, Environment and Planning A, 42(10) 2527-2546. Tourism data
  • 18.
    11/06/2014 18 Balance of Payments– Travel Item Monthly international travel statistics for Balance of Payment calculations Country level Inbound and outbound (to and from Estonia) Data since 2009 Inbound Travel Indicators: • Number of visits • Number of days spent • Number of nights spent Breakdown: • Country of origin • Estonia as transit / destination • Same-day / overnight visit • Tourist / long-term visitor (resident)
  • 19.
    11/06/2014 19 Outbound Travel Indicators: • Numberof trips / visits • Number of days spent • Number of nights spent Breakdown: • Total abroad / specific country • Country as transit / destination • Same-day / overnight visit • Tourist / long-term visitors (non-residents)
  • 20.
    11/06/2014 20 Data Transfer Monthly transferof CSV files with prepared Excel pivot tables
  • 21.
  • 22.
  • 23.
  • 24.
    11/06/2014 24 Publications in TourismStudies: Nilbe, K., Ahas, R., Silm, S. 2014. Evaluating the Travel Distances of Events and Regular Visitors using Mobile Positioning Data: The case of Estonia, Journal of Urban Technology 21(2): Kuusik, A., Tiru, M., Varblane, U., Ahas, R. 2011. Process innovation in destination marketing: use of passive mobile positioning (PMP) for segmentation of repeat visitors in case of Estonia, Baltic Journal of Management 6(3): 378 – 399. Tiru, M., Kuusik, A., Lamp, M-L., Ahas, R. 2010. LBS in marketing and tourism management: measuring destination loyalty with mobile positioning data. Journal of Location Based Services, 4(2): 120-140. Ahas, R. 2010. Mobile positioning data in geography and planning, Editorial. Journal of Location Based Services, 4(2): 67-69. Tiru, M., Saluveer E., Ahas, R., Aasa, A. 2010. Web-based monitoring tool for assessing space-time mobility of tourists using mobile positioning data: Positium Barometer. Journal of Urban Technology, 17(1): 71-89. Ahas, R. Aasa, A., Roose, A., Mark, Ü., Silm, S. 2008. Evaluating passive mobile positioning data for tourism surveys: An Estonian case study. Tourism Management 29(3): 469–486. Ahas, R., Aasa, A., Mark, Ü., Pae, T., Kull, T. 2007. Seasonal tourism spaces in Estonia: case study with mobile positioning data. Tourism Management 28(3): 898–910. Conclusions
  • 25.
    11/06/2014 25 Conclusions I: • Timeliness– fast data collection, digital processing, automatic • Better spatial and temporal accuracy • Longitudiness – covering longer time period and area • … Conclusions II • Access to data complicated, privacy… • Missing information about users, purpose of trips, expenditures • Sampling issues • …
  • 26.
    11/06/2014 26 Conclusions III • Replacingexisting data with BIG data? • Improving existing data with BIG data? • Collecting data about new aspects of social life? • NEW PRODUCTS and CONSUMER GROUPS, monitoring, real-time… Thank you! rein.ahas@ut.ee Silm, S. & Ahas, R. 2014.The temporal variation of ethnic segregation in a city: evidence from a mobile phone use dataset, Social Science Research 47: 30-43. http://dx.doi.org/10.1016/j.ssresearch.2014.03.011 Silm, S. & Ahas, R. 2014. Ethnic differences activity spaces: The study of out-of-home non-employment activities with mobile phone data, Annals of Association of American Geographers 104(5): 542-559. http://dx.doi.org/10.1080/00045608.2014.892362
  • 27.
    11/06/2014 27 „BIG data asmirror“ – society is trying to understand fast changes