Big Data: Mapping Twitter Communities

1,150 views
1,093 views

Published on

A deck presented at the MRS 'Maximising the Value of Big Data' conference in London, January 2013.

Presents my view of big data and the potential it gives us for mapping the systems that we deal with on a day-to-day basis. Big data holds the promise of providing us with a meta-view of the systems that we all think we are so familiar with. I think we will find that the woods look nothing like the trees.

Published in: Technology

Big Data: Mapping Twitter Communities

  1. 1. Mapping Twitter Communities Presented at the MRS ‘Maximizing the Value of Big Data’ conference, London, January 2013 Kyle Findlay (TNS) Paul Oosterveld (TNS) Timothy Wilson (Independent)
  2. 2. My understanding of ‘big data’ Lots of Resource- Resource- intensive to intensive to data collect process (often unstructured) Meta view of A new world of …through insights, technique old, familiar s, understanding… primary systems research???
  3. 3. The tools Machine learning Integration of Text and artificial Next-gen multiple data analysis intelligence sources databases Etc . etc. etc.
  4. 4. What we are really …is mapping complexdoing here though… systems
  5. 5. What’s the similarity between… This? http://vimeo.com/2543720
  6. 6. What’s the similarity between… And this? http://www.youtube.com/watch?v=YadP3w7vkJA
  7. 7. AndWhat’s the similarity between… this? http://www.youtube.com/watch?v=2guKJfvq4uI
  8. 8. …? Networks
  9. 9. Why am I tellingyou all this? Let’s look at some of our primary research… …a case study
  10. 10. We mapped the South African Political Twittersphere
  11. 11. Who are the popular Who is influencing What is the nature What sub- and/or influential whom within a of their influence i.e. communities exist individuals in my specific topic of how are they within my online network? interest to me? influencing people? community? What topics are What are people saying What do these sub- Who is sharing about my communities believe people talking brand, political and how do they relate my content about? party, charity, music, et to my brand/ online? c.? organisation/etc? What are the Are they saying How far is word of How is my emerging topics mostly positive or my brand or messaging being and trends? negative things? negative publicity shared online? being spread?Relevance?
  12. 12. A paradigm shift… Participants not respondents Market (CASRO) researchers “Private gardens” vs. public spaces misunderstand social media “Representative sample” what? research…Source: Barber, Michael, 2011
  13. 13. Actual tweetsWe collected two types of dataTwitter user data
  14. 14. Tweets data summary Number of tweets collected: @helenzille = 28,500 @PresidencyZA = 3,500 @SAPresident = 2,900 Time period: Sept 2011 – March 2012
  15. 15. Target nodes Source nodes Collected 64,357 Cleaned 58,349 Distance <= 4 37,695 2.5 million Distance <= 3 20,460 Distance <= 2 12,753 Distance <= 1 4,479 Distance = 0 3 Time period: April 2012Twitter user data summary
  16. 16. 3.0 2.0 1.0 Ego
  17. 17. Topic modelling HashtagsTypes of analyses Structural network Betweeness centrality Authority (HITS) Community detection Influence networks Image sources: Knowledge Matters. http://www.durantlaw.info/category/miscellaneous/strategy Gutiérrez-Pérez, JA, et al. 2011. Application of graph-spectral methods in the vulnerability assessment of water supply networks
  18. 18. Actual tweetsWe collected two types of data
  19. 19. @SAPresident + @PresidencyZA
  20. 20. @helenzille
  21. 21. We collected two types of dataTwitter user data
  22. 22. The South African politicalTwittersphere
  23. 23. Structural network by follower count
  24. 24. Top 10 byfollower count
  25. 25. Structural network bybetweeness centrality
  26. 26. Top 10 bybetweeness centrality
  27. 27. Structural network by Authority
  28. 28. Top 10 byAuthority
  29. 29. Black influentials News mediaTechnology Proudly SA & DA Sportsmen & celebrities Community detection Louvain
  30. 30. Community detectionLouvain
  31. 31. Top 10 sportsmen & celebritiesFollower countHelen Zille Bryan Habana John Smit Albie Morkel Mark Boucher157,642 100,141 84,529 94,593 84,564Cricket South South African Trevor Immelman Steve Hofmeyr 94.7 HighveldAfrica 61,536 Rugby 60,280 53,777 39,351 Radio 37,554
  32. 32. Top 10 sportsmen & celebritiesBetweeness centralityRob van Vuuren Arno Carstens Cape Talk 567 Alex van Tonder Sam Wilson16,610 11,562 11,685 10,065 6,756Karen Zoid Lead SA Jeannie D Creative Cape The Foodie (David)14,410 18,786 38,858 Town 6,259 5,219
  33. 33. Top 10 news media Follower countYaseen Theba Mail & Guardian South African BBC Africa Redi Tlhabi77,191 57,320 Presidency 53,413 49,425 45,866SA Breaking News City Press allAfrica.com Mandy Wiener Evita Bezuidenhout38,487 36,376 32,788 31,296 30,139
  34. 34. Top 10 news media Betweeness centralityMandy Wiener Sam Mkokeli Ferial Haffajee Mail & Guardian Robyn Clark31,296 3,033 29,530 57,320 1,598Sipho Hlongwane US South African Verashni Pillay The Big Issue Stephen Grootes9,093 Embassy 17,803 8,304 4,363 19,708
  35. 35. Top 10 technologyFollower countAki Anastasiou Jaco van Wyk Simon Dingle Dave Duarte Toby Shapshak28,063 20,087 15,386 9,563 9,073Finance24 Mxit Matthew Buckland Pieter Uys Duncan McLeod8,184 7,651 7,033 6,654 6,194
  36. 36. Top 10 technologyBetweeness centralityMatthew Buckland Dave Duarte Aki Anastasiou Raoul de Jongh Toby Shapshak7,033 9,563 28,063 4,133 9,073Saul Kropman Mike Sharman Uno de Waal Pete Flynn Cathryn Reece2,530 4,680 2,469 1,008 2,302
  37. 37. Top 10 black influentials Follower countBonang Matheba Julius Malema Dineo Ranaka Jacob Zuma Black Coffee231,696 (unconfirmed) 192,482 142,745 136,403 97,302David Kau Terry Pheto Pabi Moloi Claire Mawisa Kuli Roberts83,169 64,117 66,841 54,562 57,352
  38. 38. Top 10 black influentials Betweeness centralityKhaya Dlanga Simphiwe Dana TimesLIVE Julius Malema Xolisa Dyeshana46,101 33,986 52,706 (unconfirmed) 192,482 4,747Jason Von Berg David Kau Zama Ndlovu Bonang Matheba Mvelase Peppetta4,445 83,169 6,718 231,696 2,804
  39. 39. Top 10 proudly SA & DAFollower countLindiwe Mazibuko SA The Good News Brand South Africa SouthAfrica.info City of Cape Town30,460 12,483 8,549 8,575 7,725Ryan Coetzee Mmusi Maimane Tim Harris DA Youth Andrew Boraine3,528 4,917 2,792 3,239 2,214
  40. 40. Top 10 proudly SA & DABetweeness centralityLindiwe Mazibuko Ryan Coetzee Climate Smart Tim Harris Mmusi Maimane30,460 3,528 Cape Town 867 2,792 4,917Solly Malatsi Gareth van Mbali Ntuli Cape Town Phumzile Van853 Onselen 2,085 1,903 Green Map 1,408 Damme 1,588
  41. 41. Influence is contextual
  42. 42. Actual tweetsWe collected two types of data
  43. 43. #ANCYL#ANCYL #malema #ANCYLmarch | 143 tweets
  44. 44. #POIB#POIB #POSIB #stopthesecrecybill #secrecybill#BlackTuesday | 654 tweets Interesting: @PatriciaDeLille: • High presence (mentions) • Low influence (interactions)
  45. 45. @hopeleighm @PresidencyZA @SAPresident@helenzille and membersof the Democratic Alliance
  46. 46. Issues Scalability Privacy Data sourcing Resource intensive Paradigm shift Rise of walled gardens New skills Reseller limitations
  47. 47. Next steps +
  48. 48. A new kind of research Complex systems Influence and dynamics Public data MR = computer science? CONCLUSIONS Primary researchParadigm shift Representative what!? Participants, not respondents 3rd party disruptive innovation
  49. 49. Thank you@socialphysicist
  50. 50. Appendices
  51. 51. Top 10 sportsmen & celebritiesAuthorityHelen Zille Rob van Vuuren John Smit Arno Carstens Cape Talk 567Bryan Habana Alex van Tonder Sam Wilson Karen Zoid Vanessa Raphaely
  52. 52. Top 10 news media Authority South AfricanMail & Guardian Ferial Haffajee Mandy Wiener Stephen Grootes PresidencyEvita Bezuidenhout Redi Tlhabi Zapiro Max du Preez Philip de Wet
  53. 53. Top 10 technologyAuthorityAki Anastasiou Matthew Buckland Toby Shapshak Dave Duarte Simon DingleDuncan McLeod Memeburn Cherryflava Media Pieter Uys Mike Sharman
  54. 54. Top 10 black influentials Authority Julius MalemaJacob Zuma TimesLIVE Khaya Dlanga Leanne Manas (unconfirmed)Riaad Moosa Zakes Mda David Kau Simphiwe Dana Pabi Moloi
  55. 55. Top 10 proudly SA & DAAuthorityLindiwe Mazibuko Ryan Coetzee SA The Good News City of Cape Town Mmusi Maimane Gareth vanTim Harris David Maynier Andrew Boraine Gareth Morgan Onselen
  56. 56. Topicmodelling

×