How can we mine, analyse and visualise the Social Web? In this lecture, you will learn about mining social web data for analysis. Data preparation and gathering basic statistics on your data.

1. Social Web
Lecture 4
How can we MINE, ANALYSE and VISUALISE
the Social Web? (1)
Marieke van Erp
The Network Institute
VU University Amsterdam

2. Why?
• UCG provides an enormous wealth of data
• insights in users’ daily lives
• insights in communities
• insights in trends

3. To whom it may
concern
• Politicians
• Companies
• Governmental institutions
• You?

5. The Age of Big Data
• 25 billion tweets on Twitter in 2010, by 175
million users
• 360 billion pieces of contents on Facebook
in 2010, by 600 million different users
• 35 hours of videos uploaded to YouTube
every minute
• 130 million photos uploaded to flickr per
month

6. Questions to Ask
• Who uploads/talks? (age, gender,
nationality, community)
• What are the trending topics?
• What else do these users like?
• Who are the most/least active users?
• etc.

7. What do you prefer?
Image: http://www.co.olmsted.mn.us/prl/propertyrecords/RecordingDocuments/PublishingImages/forms.jpg

9. The Rise of the Data
Scientist
http://radar.oreilly.com/2010/06/what-is-data-science.html

10. The Rise of the Data
Scientist
• Data Science enables the creation of data
products
• Data products are applications that acquire
their value from the data, and create more
data as a result.
• Users are in a feedback loop: they constantly
provide information about the products they
use, which gets used in the data product.

11. Popular Data Products

12. Data Mining 101
Data mining is the exploration and analysis of large quantities of
data in order to discover valid, novel, potentially useful, and
ultimately understandable patterns in data.
(Inspired by George Tziralis’ FOSS Conf’09, John Elder IV’s
Salford Systems Data Mining Conf. and Toon Calders’ slides)
http://www.freefoto.com/images/33/12/33_12_7---Pebbles_web.j

13. Data Mining 101
Databases Statistics
Artificial
Intelligence

14. Steps
• Data input & exploration
• Preprocessing
• Data mining algorithms
• Evaluation & Interpretation

15. Data Input &
Exploration
• What data do I need to answer question
X?
• What variables are in the data?
• Basic stats of my data?

17. Input & Exploration in
‘LikeMiner’

18. Preprocessing
• Cleanup!
• Choose a suitable data model
• What happens if you integrate data
from multiple sources?
• Reformat your data

19. Preprocessing in
‘LikeMiner’

20. Data mining algorithms
• Classification: Generalising a known
structure & apply to new data
• Association: Finding relationships between
variables
• Clustering: Discovering groups and
structures in data

21. Mining in ‘LikeMiner’
• Filter users by interests
• Construct user graphs
• PageRank on graphs to mine
representativeness
• Result: set of influential users
• Compare page topics to
user interests to find pages
most representative for
topics

22. Interpreting your
results

24. Data Mining is not easy

27. Mining Social Web Data
source: http://kunau.us/wp-content/uploads/
2011/02/Screen-shot-2011-02-09-
at-9.03.46-PM-w600-h900.png

28. Single Person
Source: http://infosthetics.com/archives/2011/12/
all_the_information_facebook_knows_about_you.html
See also: http://www.youtube.com/watch?feature=player_embedded&v=kJvAUqs3Ofg

29. Populations
http://www.brandrants.com/brandrants/obama/

30. Brand Sentiment via
Twitter
http://flowingdata.com/2011/07/25/brand-sentiment-showdown/

31. Recommended Reading
http://www.cs.cornell.edu/home/kleinber/
networks-book/networks-book.pdf

32. Final Assignment:Your SocWeb App
• Create a Social Web app with
your group
• Use structured data,
relationships between entities,
data analysis, visualisation
• Write individual research report
on one of the main aspects of
your app
Image Source: http://blog.compete.com/wp-content/uploads/2012/03/Like.jpg

33. Hands-on Teaser
• Build your own recommender
system 101
• Recommend pages on
del.icio.us
• Recommend pages to your
Facebook friends
image source: http://www.flickr.com/photos/bionicteaching/1375254387/