This document discusses INEGI's use of Twitter as a source of big data. It outlines INEGI's process for collecting over 260 million geo-tagged tweets from Twitter's API and analyzing them using Apache Spark. The tweets are analyzed to extract sentiment indicators and examine mobility patterns. INEGI has also integrated tweets with other data sources and is exploring various applications of the Twitter data like tracking tourism, migration, and subjective wellbeing.

1. Twitter: A source of Big Data in a
NSO.
Experimental Indicators in Sentiment and Mobility
The ESS Big Data Workshop 2016

2. Objective
To share the experience of INEGI in the use of Twitter as a
big data source.

3. Types of Big Data Sources
• Meters and smart sensors. Traffic cameras, GPS devices, power
consume meters, IoT, smartwatches, smartphones, etc.
• Social interactions. Conversations and publications on social
networks like Twitter, Facebook, FourSquare, etc.
• Business transactions. Credit cards movements, scanned data,
cell phone records, etc.
• Electronic files. Documents which are available in electronic
formats such as PDF files, websites, videos, audio, images, photos,
etc.
• Broadcast media. Digital video and audio streamed on real-time

4. Types of Big Data Sources
• Meters and smart sensors. Traffic cameras, GPS devices,
power consume meters, IoT, smartwatches, smartphones, etc.
• Social interactions. Conversations and publications on social
networks like Twitter, Facebook, FourSquare, etc.
• Business transactions. Credit cards movements, electronic
cash registers, cell phone records, etc.
• Electronic files. Documents which are available in electronic
formats such as PDF files, websites, videos, audio, digital media
broadcasting
• Broadcast media. Digital video and audio streamed on real-time

5. Process Followed by INEGI (until now)

6. Production process

7. Study Case
• Initial objective of INEGI’s Big Data Project: To generate
experimental indicators using Big Data techniques with
social media data, to complement statistical information
obtained from traditional methods and sources.
• Initial Goal: To obtain indicators of subjective wellbeing
from social media data sources.

8. Why did we choose Twitter as a data
source?
• It’s a widely adopted social network where you can find content
written by common people
• Tweets are public, so we can use them without concerns about
privacy
• There is a free API which allows to get up to 1% of the tweets
that are being produced on real time
• ( https://dev.twitter.com/streaming )

9. In the beginning…
February 2014 – October 2015
24/7

10. Collection / Analysis infrastructure
Since October 2015 - …
(EMC / VMWare)
LOGSTASH
Location Query
Free Access
https://twittercommunity.com/t/stream-filter-sample-size/30865
Apache Spark
Clean & Sentiment
Analysis
Tweets
Daily
Processing
(4 a.m.)
300 K
Geo-Tweets
Minimal
Representation
~260 Millions
of Geo-Tweets
~130 Millions inside Mexico
~ 2 Years and 8 Months ~ 24/7

11. Multivariate Stratification
of blocks in Apache Spark
%Acceso a Internet, %Pc, %Telefono Celular, %Automovil

12. Software Stack (2016)

13. Tweet Structure

14. Tweets Map

15. Tweets Map

16. We found that
• The JSON structure is easy to process (APACHE SPARK)
• The content is text that we can examine to make the
sentiment analysis
• Geographical coordinates can be used to filter the tweets
and obtain only those of interest (warning: not all the
tweets are geo referenced)
• We can make mobility analysis, based in Tweets’ location
and time (a serendipity)

17. Exploring Big Data Base

18. Supervised Training
Manual Tagging
5000 people (TecMilenio students),
100 tweets tagged by each one,
each tweet was tagged nine times,
about 40,000 different tagged tweets,
interpreted accordingly to regional
idioms
http://cienciadedatos.inegi.org.mx/animotuitero/
They’re not enchiladas!

19. Training and modeling Collaborative
Research
Python
Implementation

20. Sentiment Visualization (2015)
(Monthly Indicator)
http://www.inegi.org.mx/inegi/contenidos/investigacion/experimentales/animotuitero/default.aspx
{JSON:File}

21. Sentiment Visualization (Late 2016)
(Daily Indicator)
C#
{RESTful:API}
{NoSQL}

22. Mobility (Late 2016)

23. Integration of other sources
Big Spatial Join
(5 M Economic Units with +60 M Tweets)
https://github.com/syoummer/SpatialSpark

24. Some applications
• Tourism
• Migration
• Use of roads
• Regional influence of big cities
• Mobility patterns
• Business activity patterns
• Subjective wellbeing
• Inequities impact
• Impact analysis of relevant
news
• Mental health
• Misogynist/discriminatory
language use
• SDG indicators?

25. Collaboration
• International
– UNECE
• ICHEC
– UNSD
– LAMBDoop
– University of Pensylvania
• National
– KioNetworks
– Dattlas
– TecMilenio
– INFOTEC
– Centro Geo
– CIDE
– CIMAT
– Sectur
• Internal
– INEGI General Directorates

26. Questions?
Abel.Coronado@inegi.org.mx
M.Sc. Abel Coronado
@abxda

27. Conociendo México
01 800 111 46 34
www.inegi.org.mx
atencion.usuarios@inegi.org.mx
@inegi_informa INEGI Informa