The document describes methods for using survey sampling techniques to estimate statistics about rare events on the Twitter social graph. It discusses using stratified Bernoulli sampling and Horvitz-Thompson estimators to sample from the Twitter population and estimate values like the number of tweets mentioning "Star Wars". It then explores extensions like snowball sampling, where the sample is expanded by including neighbors of initially sampled nodes. The goal is to enable design-based statistical inference on large network datasets.

1. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Using sampling methods to estimate rare stats on
Twitter’s graph
Antoine Rebecq
INSEE - Universit´e Paris X
12/14/15
Antoine Rebecq Sampling the Twitter graph

2. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Sommaire
1 Stats on social networks / Twitter
Motivation
Towards design-based estimation
2 Survey sampling
Estimates
Sampling design
3 Extending the sampling design
Snowball sampling
Adaptive sampling
4 Results and future work
Results
Sample size
Future work
Antoine Rebecq Sampling the Twitter graph

3. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Motivation
Towards design-based estimation
Section 1
Stats on social networks / Twitter
Antoine Rebecq Sampling the Twitter graph

4. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Motivation
Towards design-based estimation
Subsection 1
Motivation
Antoine Rebecq Sampling the Twitter graph

5. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Motivation
Towards design-based estimation
Big data begets big graph
Twitter in 2013
Image from [2]
Antoine Rebecq Sampling the Twitter graph

6. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Motivation
Towards design-based estimation
Studies - Twitter
A large range of studies used Twitter data (Computer Science,
Sociology, Psychology, etc.)
Data on Twitter can be collected via :
The REST API (limited number of queries - queries can be on
anything)
The Streaming API (Only 1% of tweets matching some
criteria)
The Firehose (Unlimited access. Expensive)
Antoine Rebecq Sampling the Twitter graph

7. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Motivation
Towards design-based estimation
The Twitter graph
The Twitter graph ([7]) :
Is undirected
Degree distribution is heavy-tailed
Antoine Rebecq Sampling the Twitter graph

8. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Motivation
Towards design-based estimation
The Twitter graph
Has small path lengths
Antoine Rebecq Sampling the Twitter graph

9. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Motivation
Towards design-based estimation
Subsection 2
Towards design-based estimation
Antoine Rebecq Sampling the Twitter graph

10. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Motivation
Towards design-based estimation
Towards design-based estimation
Model-based estimation :
Scale-free networks, Barab´asi-Albert ([1])
Small-world networks, Watts-Strogatz ([13])
Antoine Rebecq Sampling the Twitter graph

11. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Motivation
Towards design-based estimation
Towards design-based estimation
Very little exists about design-based statistical inference on
networks (Kolaczyk 2009 , [6])
We try survey sampling methods used in official Statistics
Institutes to make design-based inference about “big graphs”
Antoine Rebecq Sampling the Twitter graph

12. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Motivation
Towards design-based estimation
Example : Star Wars : The Force Awakens
Star Wars : The Force Awakens
Antoine Rebecq Sampling the Twitter graph

13. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Motivation
Towards design-based estimation
Example : “Star Wars, The Force Awakens”
Let’s write :
yk = Number of tweets @starwars by user k
between 10/29/15, 7 :48 - 10 :48 PM EST
zk = 1{yk ≥ 1}
Goal : estimate NC = T(Z)
Additionally, we write : nC =
k∈s
zk
Antoine Rebecq Sampling the Twitter graph

14. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Estimates
Sampling design
Section 2
Survey sampling
Antoine Rebecq Sampling the Twitter graph

15. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Estimates
Sampling design
Subsection 1
Estimates
Antoine Rebecq Sampling the Twitter graph

16. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Estimates
Sampling design
Horvitz-Thompson estimator
Population U : vertices of the Twitter graph.
Assign all k ∈ U an inclusion probability P(k ∈ s) = πk
Antoine Rebecq Sampling the Twitter graph

17. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Estimates
Sampling design
Horvitz-Thompson estimator
Classic unbiased estimator for totals and means :
Horvitz-Thompson
ˆT(Y )HT =
k∈s
yk
πk
ˆ¯y =
1
N
k∈s
yk
πk
Antoine Rebecq Sampling the Twitter graph

18. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Estimates
Sampling design
Horvitz-Thompson estimator
Variance of the Horvitz-Thompson estimator depends on the first
and second-order inclusion probabilities :
πk = P(k ∈ s)
πkl = P(k, l ∈ s)
V( ˆT(Y )HT ) =
k∈U l∈U
(πkl − πkπl )
yk
πk
yl
πl
Antoine Rebecq Sampling the Twitter graph

19. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Estimates
Sampling design
Calibrated estimator
Deville-Sarndal, 1992 ([3]). Modification of the Horvitz-Thompson
estimator to take auxiliary information into account. For example :
T(Y ) = Number of tweets @StarWars
N = Number of users in scope
Structure of number of followers
Number of verified users
. . .
Very similar to empirical likelihood methods ([9]).
Antoine Rebecq Sampling the Twitter graph

20. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Estimates
Sampling design
Subsection 2
Sampling design
Antoine Rebecq Sampling the Twitter graph

21. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Estimates
Sampling design
Sampling frame
Each Twitter user is assigned a unique id. When a new user is
created, the id that is assigned to it is greater than the last
previous id.
But, not all ids match an existing user (≈ 3.1 · 109 ids as of
October 2015), which means our frame over-covers the
population. Over-coverage can be corrected either by using a
Horvitz-Thompson or Hajek estimator (see [10]).
Antoine Rebecq Sampling the Twitter graph

22. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Estimates
Sampling design
Sampling design : Bernoulli
Poisson sampling : For each k ∈ U , run a πk-Bernoulli experiment
to decide whether to include unit k in the sample.
Bernoulli sampling : ∀k, πk = p
Sampling design of non-fixed sample size. We set the expected
sample size to 20000.
Antoine Rebecq Sampling the Twitter graph

23. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Estimates
Sampling design
Sampling design : Stratified Bernoulli
We write : U = U1 U2 (h = 1, 2 being called “strata”) and
draw two independant Bernoulli samples in U1 and U2.
Here :
U1 = Followers of official @starwars account
U2 = Rest of Twitter users
Antoine Rebecq Sampling the Twitter graph

24. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Estimates
Sampling design
Sampling design : Neyman allocation
Optimal variance of the Horvitz-Thompson estimator is obtained
for (Neyman, [8]) :
nh =
NhS2
h
h
NhS2
h
Given the expected values, we set :
n1 = 9700
n2 = 10300
Antoine Rebecq Sampling the Twitter graph

25. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Estimates
Sampling design
Sampling design : Stratified Bernoulli
Estimators for the two “simple” designs :
ˆNC1 =
nC
p
ˆNC2 =
N1
n1
nC1 +
N − N1
n2
nC2
Antoine Rebecq Sampling the Twitter graph

26. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Estimates
Sampling design
Variance estimators
ˆV( ˆT(Y ))1 =
k∈s
(1 − p)yk
p2
Antoine Rebecq Sampling the Twitter graph

27. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Snowball sampling
Adaptive sampling
Section 3
Extending the sampling design
Antoine Rebecq Sampling the Twitter graph

28. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Snowball sampling
Adaptive sampling
Snowball sampling
From now on, our sampling designs will include extensions :
s = s0 ∪ sext
s0 is still selected using stratified Bernoulli, but with expected
sample size of 1000, so that the expected sample size of s is more
or less 20000.
Antoine Rebecq Sampling the Twitter graph

29. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Snowball sampling
Adaptive sampling
Subsection 1
Snowball sampling
Antoine Rebecq Sampling the Twitter graph

30. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Snowball sampling
Adaptive sampling
Snowball sampling
Population U
Antoine Rebecq Sampling the Twitter graph

31. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Snowball sampling
Adaptive sampling
Snowball sampling
Initial sample s0
Antoine Rebecq Sampling the Twitter graph

32. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Snowball sampling
Adaptive sampling
Snowball sampling
One stage snowball extension s = A(s0)
Antoine Rebecq Sampling the Twitter graph

33. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Snowball sampling
Adaptive sampling
Snowball sampling
Formally, we write :
Bi = {i} ∪ {j ∈ V , Eji = ∅}
Ai = {i} ∪ {j ∈ V , Eij = ∅}
s = A(s0)
Antoine Rebecq Sampling the Twitter graph

34. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Snowball sampling
Adaptive sampling
Snowball sampling
ˆNC3 =
k∈s
zi
1 − ¯π(Bi )
where :
¯π(Bi ) = P(Bi ⊂ ¯s)
=
k∈Bi
(1 − P(k ∈ s))
= q
#(Bi ∩U1)
S1 · q
#(Bi ∩U2)
S2
Antoine Rebecq Sampling the Twitter graph

35. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Snowball sampling
Adaptive sampling
Snowball sampling
ˆV( ˆNC3) =
i∈s j∈s
zi zj
¯π(Bi ∪ Bj )
γij
where :
γij =
¯π(Bi ∪ Bj ) − ¯π(Bi )¯π(Bj )
[1 − ¯π(Bi )][1 − ¯π(Bj )]
Antoine Rebecq Sampling the Twitter graph

36. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Snowball sampling
Adaptive sampling
Subsection 2
Adaptive sampling
Antoine Rebecq Sampling the Twitter graph

37. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Snowball sampling
Adaptive sampling
Adaptive sampling
In adaptive sampling, when (Thompson, [11])
Used in official statistics to measure number of drugs users or
HIV-positive people
Sampling design often compared to the video game
“minesweeper”
Antoine Rebecq Sampling the Twitter graph

38. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Snowball sampling
Adaptive sampling
Adaptive sampling
Image from [12]
Antoine Rebecq Sampling the Twitter graph

39. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Snowball sampling
Adaptive sampling
Adaptive sampling
Once a unit bearing the characteristic of interest (i.e. a user who
tweeted about the Star Wars trailer) is found, all its network (i.e.
its friends and friends of friends, etc. who have tweeted about Star
Wars) is included in the sample.
Antoine Rebecq Sampling the Twitter graph

40. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Snowball sampling
Adaptive sampling
Adaptive sampling
Estimator :
ˆNC4 =
K
k=1
n∗
CkJk
πgk
where :
K = number of networks
y∗
k = total of Y in the network k
n∗
Ck
= Number of people with yk ≥ 1in the network k
Jk = 1{k ∈ C}
πgk = probability that the initial sample intersects k
Antoine Rebecq Sampling the Twitter graph

41. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Snowball sampling
Adaptive sampling
Adaptive sampling
When using an adaptive design, it is often better to use the
Rao-Blackwell of the previous estimate. It has a very simple closed
form in the case of the adaptive stratified.
ˆNC5 = n0
+
K
k=1
nr
1 − (1 − p)nr
where : n0 = #s0 and s0 = ∪r {k ∈ s, δ(k, C) = 1} is the union of
the sides of C.
Antoine Rebecq Sampling the Twitter graph

42. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Results
Sample size
Future work
Section 4
Results and future work
Antoine Rebecq Sampling the Twitter graph

43. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Results
Sample size
Future work
Subsection 1
Results
Antoine Rebecq Sampling the Twitter graph

44. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Results
Sample size
Future work
Results
Design n nscope n0
ˆNC
ˆCV ˆDeff
Bernoulli 20013 3946 354121 0.231 1.04
Stratified 20094 9832 316889 0.097 0.68
1-snowball 159957 73570 1000 331097 0.031 0.60
Antoine Rebecq Sampling the Twitter graph

45. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Results
Sample size
Future work
Results
Mean number of tweets @StarWars per user : 1.18 ± 0.07
Suggests that bots are not responsible for this very large number of
tweets (see [5], [4]) !
Antoine Rebecq Sampling the Twitter graph

46. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Results
Sample size
Future work
Subsection 2
Sample size
Antoine Rebecq Sampling the Twitter graph

47. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Results
Sample size
Future work
Snowball sampling - sample size
Expected sample size ≈ 20000.
Actual sample size : > 150000 !
Antoine Rebecq Sampling the Twitter graph

48. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Results
Sample size
Future work
Adaptive sampling
With our test subject (tweets @AmericanIdol), average network
size was no greater than a few units (≈ 10000 tweets in the scope)
With Star Wars (≈ 300000 tweets in the scope, with much less
tweets per people), we couldn’t get to the end of every network !
Antoine Rebecq Sampling the Twitter graph

49. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Results
Sample size
Future work
Subsection 3
Future work
Antoine Rebecq Sampling the Twitter graph

50. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Results
Sample size
Future work
Future work
Control sample size
Estimates and calibration on graph totals (centrality,
clustering coefficients, path length, etc.)
Antoine Rebecq Sampling the Twitter graph

51. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Results
Sample size
Future work
Conclusion
Thank you !
http://nc233.com/cmstatistics2015
@nc233
Antoine Rebecq Sampling the Twitter graph

52. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Results
Sample size
Future work
Albert-L´aszl´o Barab´asi and R´eka Albert.
Emergence of scaling in random networks.
science, 286(5439) :509–512, 1999.
Paul Burkhardt and Chris Waring.
An nsa big graph experiment.
In presentation at the Carnegie Mellon University SDI/ISTC
Seminar, Pittsburgh, Pa, 2013.
Jean-Claude Deville and Carl-Erik S¨arndal.
Calibration estimators in survey sampling.
Journal of the American statistical Association,
87(418) :376–382, 1992.
Emilio Ferrara.
”manipulation and abuse on social media” by emilio ferrara
with ching-man au yeung as coordinator.
SIGWEB Newsl., (Spring) :4 :1–4 :9, April 2015.
Antoine Rebecq Sampling the Twitter graph

53. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Results
Sample size
Future work
Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer,
and Alessandro Flammini.
The rise of social bots.
arXiv preprint arXiv :1407.5225, 2014.
Eric D Kolaczyk.
Statistical analysis of network data.
Springer, 2009.
Seth A Myers, Aneesh Sharma, Pankaj Gupta, and Jimmy Lin.
Information network or social network ? : the structure of the
twitter follow graph.
In Proceedings of the companion publication of the 23rd
international conference on World wide web companion, pages
493–498. International World Wide Web Conferences Steering
Committee, 2014.
Jerzy Neyman.
Antoine Rebecq Sampling the Twitter graph

54. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Results
Sample size
Future work
On the two different aspects of the representative method :
the method of stratified sampling and the method of purposive
selection.
Journal of the Royal Statistical Society, pages 558–625, 1934.
Art B. Owen.
Empirical likelihood.
CRC press, 2010.
Olivier Sautory.
Les enjeux m´ethodologiques li´es `a l’usage de bases de sondage
imparfaites.
Steven K Thompson.
Adaptive cluster sampling.
Journal of the American Statistical Association,
85(412) :1050–1059, 1990.
Steven K Thompson.
Antoine Rebecq Sampling the Twitter graph

55. Stats on social networks / Twitter
Survey sampling
Extending the sampling design
Results and future work
Results
Sample size
Future work
Stratified adaptive cluster sampling.
Biometrika, pages 389–397, 1991.
Duncan J Watts and Steven H Strogatz.
Collective dynamics of ‘small-world’networks.
nature, 393(6684) :440–442, 1998.
Antoine Rebecq Sampling the Twitter graph