Presentation for the 2011 New Techniques and Technologies for Statistics conference, Brussels, Belgium.

1. New data sources for
statistics
Experiences at Statistics Netherlands
Piet Daas, Marko Roos, Chris de Blois,
Rutger Hoekstra, Olav ten Bosch, and Yinyi Ma
NTTS 2011

2. Why new data sources?
• Many NSI’s traditionally use:
• Surveys
• Administrative data (registers)
• But there are other sources of information out
there (especially the electronic ones)
• Are they really useful?
• Investigate it! (studies are supported by DG of Stat. Neth.)
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
1

3. Examples of new data sources
Three will be discussed:
1. Product prices on the internet
2. Mobile phone location data
3. Twitter text messages
4. Global Positioning System (GPS) data
(and traffic loop information)
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
2

4. 1) Product prices on the internet
• Ubiquitous on the World Wide Web
• Stat. Neth. already uses web price data for the
Consumer Price Index:
• E.g. prices of airline tickets, books, CD’s, DVD’s
• But this data is manually collected
• Why not automatically (and more often)?
• With a web robot (web spider)
• Which is a script or (commercial) tool
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
3

5. Product prices on the internet (2)
Collected data:
• Daily (over a 10 month period)
• 6 websites: 4 airlines, 1 housing, and 1 (unstaffed) petrol station
site
Works well but:
• Some websites are very complicated
• Especially dynamic websites, if possible directly tap into database
• Sometimes websites change lay-out
• 3 of the 4 airline websites did this (in test period)
• Affects the cost efficiency (redesigning scripts takes a lot of time)
• Manual data collection is much easier and cheaper!
• But: Automatic data collection has its own merits
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
4

6. Product prices on the internet (3)
Example: Airline ticket prices (shown over 116 day period)
Manual
collected:
1 day before
departure date
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
5

7. 2) Mobile phone location data
• Almost everybody has a mobile phone
• in the Netherlands ~92%
• People use it a lot
• For every outgoing and incoming call the phone
connects to a nearby telephone mast (‘cell’)
• Source of location information!
• Every mobile phone and masts have an unique ID
• This data is logged by mobile phone companies
• Used for billing purposes & network maintenance
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
6

8. Mobile phone location data (2)
• Could be an interesting source of information
• Let's study a dataset
• Obtained data from 1 large Dutch mobile phone company
• over 5 million different phones active on their own network
• Dataset covered a 14 day period
• Contained 550 million records (call events)
• Every record contains an unique phone ID, date-time stamp, and
mast (cell) connection ID (= location info)
• Phone ID’s were scrambled to avoid identification
• Scrambled ID’s were stable over 14 day period
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
7

9. Mobile phone location data (3)
Typical day in the Netherlands (call activity)
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
8

10. Movie of call activity during the day
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
9

11. Mobile phone location data (4)
This type of location information could possibly
be used to study:
• Overall day time movement of people
• Perhaps also: movement of individual mobile phones during the
day
• Distinguish regions of different economic activity
• Different behaviour during the week and on ‘specific’ days
• For tourism
• Roaming info: Activity of non-‘Dutch’ phones in the Netherlands
• BUT!
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
10

12. Mobile phone location data (5)
However: methodological issues
1. Translation of call intensity per cell to call intensity
per region (cell coverage area)
2. Representativity
• Phone-ID’s vs. Dutch population
• Only used data of one mobile phone company
• Some people are more active callers then others
3. How does the number of calls relate to the number of
people present at the location?
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
11

13. 3) Twitter text messages
• Social media is used more and more
intensively in the Netherlands & Europe
• Potential source of personal information,
opinions, and sentiments
• But what type of information is actually
exchanged?
• Investigated Twitter (as an example)
• Easily accessible (text)data and used a lot in
the Netherlands
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
12

14. Twitter text messages (2)
• Twitter is a micro blogging service
• Text messages of 140 characters max
• Called ‘tweets’
• Posted to the public or to friends only
• Hashsign (#) is used to highlight ‘keywords’
• Example: #Eurostat, #NTTS
• A few examples
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
13

15. Twitter text messages (3)
• Identify topics discussed on Twitter in the
Netherlands by collecting ‘tweets’
• Use this information to decide if Twitter (and
perhaps social media in general) is of interest for
Stat. Neth.
• Collect tweets from ‘all’ Dutch Twitter users!
• People located in the Netherlands
• Use location info provided by users
• Try to get a complete overview
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
14

16. Twitter text messages (4)
• Collect tweets
• First studies by using Twitter search option
• Radius of 200km from Utrecht + Dutch language filter
• BUT: Twitter appears to apply a ‘quality filter’ so data was incomplete
• Best alternative: First ‘crawl’ for users, then collect tweets
• Search through users tree: select users with large number of
followers (‘friends’), select these and expand search
• User is Dutch if location includes ‘Netherlands’ or the name of a
Dutch municipality
• Collected 380,415 unique usernames
• For ever user collect up to 200 tweets; obtained ~12 million tweets
• Identify topics discussed (first approach: used hashtags, manually)
• ~1,8 million tweets contained a hashtag
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
15

17. Twitter text messages (5)
All #hashtags
News Events Products
Companies
Top 500 #hashtags Locations
Radio 1% 1% 1% 1% 1%
Emotions
Table 1. Classification of Twitter messages collected according to hashtags used
• Results of #hashtag classification 500# Top 500# no
Applications
Top TV
2%
3%
Politics All# no
Category Description
Politics 7% Examples only (%) 3%Other (%) Other (%)
Twitter Sports
Twitter/internet specific language & slang #durftevragen, #fail, #twexit 12 3%
19 Applications
19
7%
Sports Sports, clubs, and sports events
9% #WK2010, #ajax, #oranje 9 14 14
TV Twitter specific programs
Applications #nowplaying, #lastfm, #in 8 13
3% 12
Politics 6%Political debates, leaders, and parties #tk2010, #NOSdebat, #formatie 7 11 Sports 11
TV Dutch TV-programs (no political & no news) #dwdd, #ohohcherso, #tvoh 6 10
4% 11
Emotions Sentiment and feelings
Twitter
Emotions #moe, #LOL, #zucht, #heerlijk 6 10 10
12%
Locations 6%References to a location or municipality #amsterdam, #utrecht 3 5 Twitter 5
5%
Products
Locations 3% Referring to products #iPhone, #iPad, #android 3 4 4
Events Non-sport and non-political happenings #twibbon, #LL10, #lowlands 3 4 4
Products 3% Referring to news programs
News #nos, #pownews, #Nujij 2 4 4
Companies Referring to companies #ns, #google, #tmobile, #KPN 2 4 4
3%
Events
Radio Dutch radio programs #3fm, #53j8, #radio1 1 2 2
Other 2% Rest group, mostly unrelated tags #koffie, #goedemorgen Other 38 - -
News
72%
2% Other
Companies
1%
Radio 38%
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
16

18. Twitter text messages (6)
• First conclusions (based on topics identified by
using hashtags)
• Potential interesting for politics and events
• Overall study suggests ~5% in our total dataset
• Around 600,000 tweets
• Twitter could probably also be used:
• for info on social and cultural participation and on social cohesion
• Need to further refine our studies
• More in depth studies of all tweets collected (also without #)
• Use (more advanced) text mining techniques for classification
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
17

19. Overall remarks
• Many interesting new data sources out there
• But its not easy to determine their usefulness
• Automatically collect product prices from the web
• Only in addition to the traditional manual process
• To obtain more & more frequent data
• Mobile phone & Twitter data
• Representativity of the data is a key issue
• Not all (Dutch) people are observed
• Hardly any background information available
• Is a major topic in future research
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
18

20. Thank you for your attention!
• #Questions?
NTTS2011 New data sources for statistics: Exp. Stat. Neth.
19