1.
CSE509: Introduction to Web Science and Technology<br />Lecture 5: Social Network Analysis<br />ArjumandYounus<br />Web Science Research Group<br />Institute of Business Administration (IBA)<br />

2.
Last Time…<br />Web Data Explosion<br />Part I<br />MapReduce Basics<br />MapReduce Example and Details<br />MapReduce Case-Study: Web Crawler based on MapReduce Architecture<br />Part II<br />Large-Scale File Systems<br />Google File System Case-Study<br />August 06, 2011<br />

3.
Today<br />Transition from Web 1.0 to Web 2.0<br />Social Media Characteristics<br />Part I: Theoretical Aspects<br />Social Networks as a Graph<br />Properties of Social Networks<br />Part II: Getting Hands-On Experience on Social Media Analytics<br />Twitter Data Hacks<br />Part III: Example Researches<br />August 06, 2011<br />

4.
Quick Survey<br />Do you have a Facebook, MySpace, Twitter, or LinkedIn account?<br />Do you own a blog?<br />Do you read blogs?<br />Have you ever searched for something on Wikipedia?<br />Have you ever submitted content to a social network?<br />August 06, 2011<br />

5.
Web 1.0 vs. Web 2.0<br />August 06, 2011<br />Borrowed from SIGKDD 2008 tutorial slides of Professor Huan Liu and Professor Nitin Agarwal with permission <br />

6.
What is so Different about Web 2.0?<br />User Generated Content<br />Collaborative Environment: Participatory Web, Citizen Journalism<br />User is the Driving Factor<br />August 06, 2011<br />A Paradigm Shift rather than a Technology Shift<br />

7.
Top 20 Most Visited Web Sites<br />Internet traffic report by Alexa on July 29th 2008<br />August 06, 2011<br />Borrowed from SIGKDD 2008 tutorial slides of Professor Huan Liu and Professor Nitin Agarwal with permission <br />

8.
Various forms of Social Media<br />Blog: Wordpress, blogspot, LiveJournal<br />Forum: Yahoo! Answers, Epinions<br />Media Sharing: Flickr, YouTube, Scribd<br />Microblogging: Twitter, FourSquare<br />Social Networking: Facebook, LinkedIn, Orkut<br />Social Bookmarking: Del.icio.us, Diigo<br />Wikis: Wikipedia, scholarpedia, AskDrWiki<br />August 06, 2011<br />

9.
Characteristics of Social Media<br />“Consumers” become “Producers”<br />Rich User Interaction<br />User-Generated Contents<br />Collaborative environment<br />Collective Wisdom<br />Long Tail<br />Broadcast Media<br />Filter, then Publish<br />Social Media<br />Publish, then Filter<br />August 06, 2011<br />

10.
August 06, 2011<br />

11.
PART I: Theoretical Aspects<br />August 06, 2011<br />

12.
Networks and Representation<br />Social Network: A social structure made of nodes (individuals or organizations) and edges that connect nodes in various relationships like friendship, kinship etc. <br />August 06, 2011<br /><ul><li>Graph Representation

13.
Matrix Representation</li></li></ul><li>Properties of Large-Scale Networks<br />Networks in social media are typically huge, involving millions of actors and connections<br />Large-scale networks in real world demonstrate similar patterns<br />Scale-free Distributions<br />Small-world Effect<br />Strong Community Structure<br />August 06, 2011<br />

14.
Scale-Free Distributions<br />Degree distribution in large-scale networks often follows a power law. <br />A.k.a. long tail distribution, scale-free distribution<br />August 06, 2011<br />Degrees<br /> Nodes<br />

15.
Small-World Effect<br />“Six Degrees of Separation”<br />A famous experiment conducted by Travers and Milgram (1969)<br />Subjects were asked to send a chain letter to his acquaintance in order to reach a target person <br />The average path length is around 5.5<br />Verified on a planetary-scale IM network of 180 million users (Leskovec and Horvitz 2008) <br />The average path length is 6.6<br />August 06, 2011<br />

16.
Small World Facebook Experiment by Yahoo! Labs<br />Anyone in the world can get a message to anyone else in just "six degrees of separation" by passing it from friend to friend. Sociologists have tried to prove (or disprove) this claim for decades, but it is still unresolved.<br />http://smallworld.sandbox.yahoo.com/<br />August 06, 2011<br />

17.
Community Structure<br />Community: People in a group interact with each other more frequently than those outside the group<br />ki = number of edges among node Ni’s neighbors<br />Friends of a friend are likely to be friends as well<br />Measured by clustering coefficient: <br />Density of connections among one’s friends<br />August 06, 2011<br />

18.
Clustering Coefficient<br />August 06, 2011<br /><ul><li>d6=4, N6= {4, 5, 7,8}

19.
k6=4 as e(4,5), e(5,7), e(5,8), e(7,8)

20.
C6 = 4/(4*3/2) = 2/3

21.
Average clustering coefficient</li></ul>C = (C1 + C2 + … + Cn)/n<br /><ul><li>C = 0.61 for the left network

22.
In a random graph, the expected coefficient is 14/(9*8/2) = 0.19.</li></li></ul><li>Challenges<br />Scalability<br />Social networks are often in a scale of millions of nodes and connections<br />Traditional network analysis often deals with at most hundreds of subjects<br />Heterogeneity<br />Various types of entities and interactions are involved<br />Evolution<br />Timelines are emphasized in social media<br />Collective Intelligence<br />How to utilize wisdom of crowds in forms of tags, wikis, reviews<br />Evaluation<br />Lack of ground truth, and complete information due to privacy<br />August 06, 2011<br />

23.
Social Computing Tasks<br />Social Computing: a young and vibrant field<br />Conferences: KDD, WSDM, WWW, ICML, AAAI/IJCAI, SocialCom, etc.<br />Tasks<br />Centrality Analysis and Influence Modeling<br />Community Detection<br />Classification and Recommendation<br />Privacy, Spam and Security<br />August 06, 2011<br />

24.
Centrality Analysis and Influence Modeling<br />Centrality Analysis: <br />Identify the most important actors or edges<br />E.g. PageRank in Google<br />Various other criteria<br />Influence modeling: <br />How is information diffused? <br />How does one influence each other? <br />Related Problems<br />Viral marketing: word-of-mouth effect<br />Influence maximization<br />August 06, 2011<br />

25.
Community Detection<br />A community is a set of nodes between which the interactions are (relatively) frequent<br />A.k.a., group, cluster, cohesive subgroups, modules <br />Applications: Recommendation based communities, Network Compression, Visualization of a huge network <br />New lines of research in social media<br />Community Detection in Heterogeneous Networks<br />Community Evolution in Dynamic Networks<br />Scalable Community Detection in Large-Scale Networks<br />August 06, 2011<br />

26.
Classification and Recommendation<br />Common in social media applications<br />Tag suggestion, Product/Friend/Group Recommendation<br />August 06, 2011<br />Link prediction<br />Network-Based Classification<br />

27.
Privacy, Spam and Security<br />Privacy is a big concern in social media<br />Facebook, Google buzz often appear in debates about privacy<br />NetFlix Prize Sequel cancelled due to privacy concern<br />Simple anonymization does not necessarily protect privacy<br />Spam blog (splog), spam comments, fake identity, etc., all requires new techniques<br />As private information is involved, a secure and trustable system is critical <br />Need to achieve a balance between sharing and privacy<br />August 06, 2011<br />

28.
PART II: Practical SNA with Twittersphere Mining<br />August 06, 2011<br />

29.
Pre-Requisites<br />Expectation that Python is installed and you have some hands-on experience with it<br />Dependencies<br />easy_install<br />networkx<br />twitter (Twitter API for Python)<br />For Windows users<br />Install ActivePython: comes bundled with easy_install<br />easy_installnetworkx<br />easy_install twitter<br />For Linux users<br />sh setuptools-0.6c11-py2.6.egg<br />sudoeasy_installnetworkx<br />sudoeasy_install twitter<br />August 06, 2011<br />

30.
Getting Tweets from Twitter Search API<br />import twitter<br />import json<br />twitter_search=twitter.Twitter(domain="search.twitter.com")<br />search_results=[]<br />for page in range(1,6):<br />search_results.append(twitter_search.search(q="pakistan",rpp=100,page=page))<br />print json.dumps(search_results, sort_keys=True, indent=1)<br />tweets=[r['text'] for result in search_results for r in result['results']]<br />print tweets<br />August 06, 2011<br />

31.
Lexical Diversity for Tweets<br />words=[]<br />for t in tweets:<br /> words+= [w for w in t.split()]<br />lexical_diversity=1.0*len(set(words))/len(words)<br />August 06, 2011<br />

32.
What People are Tweeting: Frequency Analysis<br />freq_dist=nltk.FreqDist(words)<br />freq_dist.keys()[:50]<br />freq_dist.keys()[-50:]<br />August 06, 2011<br />

33.
Extracting Relationships from Tweets (1/3)<br />Step 1: Extracting Graph Data<br />import networkx as nx<br />import re<br />g=nx.DiGraph()<br />twitter_search=twitter.Twitter(domain="search.twitter.com")<br />search_results=[]<br />for page in range(1,6):<br />search_results.append(twitter_search.search(q="pakistan",rpp=100,page=page))<br />all_tweets=[tweet for page in search_results for tweet in page["results"]]<br />def get_rt_sources(tweet):<br />rt_patterns=re.compile(r"(RT|via)((?:*@+)+)",re.IGNORECASE)<br />return [source.strip() for tuple in rt_patterns.findall(tweet) for source in tuple if source not in ("RT", "via")]<br />for tweet in all_tweets:<br />rt_sources=get_rt_sources(tweet["text"])<br />if not rt_sources:<br />continue<br />for rt_source in rt_sources:<br />g.add_edge(rt_source,tweet["from_user"],{"tweet_id":tweet["id"]})<br />August 06, 2011<br />

34.
Extracting Relationships from Tweets (2/3)<br />Step 2: Generating DOT File<br />OUT = "pakistan_search_results.dot“<br />dot=['"%s" -> "%s" [tweet_id=%s]' % (n1.encode('utf-8'), n2.encode('utf-8'), g[n1][n2]['tweet_id']) for n1, n2 in g.edges()]<br />f=open(OUT, 'w')<br />f.write('strict digraph {%s}' % (';'.join(dot),))<br />f.close()<br />August 06, 2011<br />

35.
Extracting Relationships from Tweets (3/3)<br />Step 3: Visualizing the Retweet Data in Graphical Form<br />For Windows users<br />For Linux users<br />circo -Tpng -Osnl_search_results pakistan_search_results.dot<br />August 06, 2011<br />

36.
PART III: Example Researches<br />August 06, 2011<br />

37.
Million Follower Fallacy (New York Times)<br />August 06, 2011<br />

38.
Twitter: More a News Medium than a Social Network (PC World)<br />August 06, 2011<br />

39.
Twitter for World Peace (Business Week)<br />August 06, 2011<br />

40.
SocialFlow: Social Media Optimization<br />Social Media Optimization Platform<br />Works in Domains of Viral and Word-of-Mouth Marketing<br />Provides Services to Major Media Outlets <br />Recent study<br />How different audiences consumed and rebroadcast messages news organizations were sending out: AlJazeera English, BBC News, CNN, The Economist, Fox News and New York Times<br />August 06, 2011<br />

41.
August 06, 2011<br />Twitter as a Real-Time News Analysis Service<br />

42.
Studying Ins and Outs of News<br />Using Twitter to study hot news items people are heavily tweeting about<br />August 06, 2011<br />

43.
Algorithm for Identification of Popular News<br />August 06, 2011<br />

44.
Application Prototype<br />August 06, 2011<br />

45.
Observations (1/3)<br />August 06, 2011<br />Percentage of news in tweets per day greater than 50% for all days except one day<br />

46.
Observations (2/3)<br />August 06, 2011<br />Highest Number of Recorded Tweets per Day<br />

47.
Observations (3/3)<br />August 06, 2011<br />