Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Digital Trails Dave King 1 5 10 Part 2 D3


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Digital Trails Dave King 1 5 10 Part 2 D3

  1. 1. Collective Intelligence (CI): Defined <ul><li>The intelligence that’s extracted out from the collective set of interactions and contributions made by your users. </li></ul><ul><li>The use of this intelligence to act as a filter for what’s valuable in your application for a user </li></ul>Source: Alag, S. Collective Intelligence in Action . Manning Press (2009)
  2. 2. Collective Intelligence: Explicit Resources
  3. 3. Ways to Harness CI Source: Alag, S. Collective Intelligence in Action . Manning Press (2009)
  4. 4. CI Requirements <ul><li>You need to: </li></ul><ul><li>Allow users to interact with your site and with each other, learning about each user through their interactions and contributions. </li></ul><ul><li>Aggregate what you learn about your users and their contributions using some useful models. </li></ul><ul><li>Leverage those models to recommend relevant content to a user (yielding higher retention & completion rates) </li></ul>Source: Alag, S. Collective Intelligence in Action . Manning Press (2009)
  5. 5. Forms of CI Data <ul><li>Data comes in two forms: structured data and unstructured data. </li></ul><ul><ul><li>Structured data has a well defined form, something that makes it easily stored and queried on.( e.g. user ratings, content articles viewed, and items purchased …) </li></ul></ul><ul><ul><li>Unstructured data is typically in the form of raw text (e.g. reviews, discussion forum posts, blog entries, and chat sessions …) </li></ul></ul><ul><li>Most applications transform unstructured data into structured data </li></ul>Source: Alag, S. Collective Intelligence in Action . Manning Press (2009)
  6. 6. CI Data Model Most applications generally consist of users and items. An item is any entity of interest in your application. If your application is a social-networking application, or you’re looking to connect one user with another, then a user is also a type of item. Source: Alag, S. Collective Intelligence in Action . Manning Press (2009) Users Metadata Items
  7. 7. Classification of Recommender Engines
  8. 8. Non-Personalized Collaboration <ul><li>Non-personalized recommendations are identical for each user. The recommendations are either manually selected (e.g. editor choices) or based on the popularity of items (e.g. average ratings, sales data). </li></ul>
  9. 9. Non-Personalized: Example
  10. 10. Non-Personalized: Example
  11. 11. Demographic Recommendation <ul><li>The users are categorized based on the attributes of their demographic profiles in order to find users with similar features. The engine then suggests or recommends (explicitly or implicitly) items that are preferred by these similar users. </li></ul>
  12. 12. Demographic Recommendation: Example
  13. 13. Demographic Recommendation: Example
  14. 14. Demographic Recommendation: Mystery Movie
  15. 15. Demographic Recommendation: Guilt by Association <ul><li>Advantages </li></ul><ul><ul><li>New users can get recommendations before they have rated any item. </li></ul></ul><ul><ul><li>Technique is domain independent </li></ul></ul><ul><li>Limitations </li></ul><ul><ul><li>Gathering the required demographic data leads to privacy issues. </li></ul></ul><ul><ul><li>Demographic classification is too crude for highly personalized recommendations. </li></ul></ul><ul><ul><li>Users with an unusual taste may not get good recommendations (“gray sheep” problem). </li></ul></ul><ul><ul><li>Once established user preferences do not change easily (stability vs. plasticity problem). </li></ul></ul>
  16. 16. Collaborative Filtering <ul><li>Employs user-item ratings (or votes) as their information source. The concept is to make correlations between users or between items. </li></ul><ul><li>The correlations are used to predict user behavior and make recommendations. </li></ul><ul><li>Widely implemented and the most mature recommendation technique. </li></ul>
  17. 17. Collaborative Filtering: Example
  18. 18. Collaborative Filtering: Example
  19. 19. Collaborative Filtering: Example
  20. 20. Collaborative Filtering: Example
  21. 21. Collaborative Filtering: Example
  22. 22. Collaborative Filtering: Main Approaches <ul><li>User-based </li></ul><ul><li>Item-based </li></ul><ul><li>Model-based </li></ul>
  23. 23. Collaborative Filtering: User-Based <ul><li>Assumption that users that rated the same items similarly probably have the same taste. </li></ul><ul><li>It make user-to-user correlations by using the rating profiles of different users to find highly correlated users. </li></ul><ul><li>These users form like-minded neighborhoods based on their shared item preferences. </li></ul><ul><li>The engine then can recommend the items preferred by the other users in the neighborhood. </li></ul>j 1 j 2 j 3 j 4 j 5 i u 1 2 3 4 5 ? v 1 1 2 3 4 5 5 v 2 5 4 3 2 1 1
  24. 24. Collaborative Filtering: Similarity? <ul><li>Mathematical concept analogous to the notion of Euclidean Distance </li></ul>1 2 3 0 1 2 3 A B C 1 2.24
  25. 25. Collaborative Filtering: Similarity? <ul><li>Cosine </li></ul><ul><li>Correlation </li></ul><ul><li>Adjusted Cosine </li></ul>
  26. 26. Collaborative Filtering: Cosine Similarity (an Example) Step 1: Find SQRT of Sum of Squares Each Row of Scores Step 2: Divide each Scores In row by SQRT of Sum of SQs Step3: Calculate Cosine Similarity Between Users by Summing X-Products of their normalized Scores (from Step 2)
  27. 27. Collaborative Filtering: User-Based Predictions and Recommendations
  28. 28. Collaborative Filtering: User-Based Disadvantages <ul><li>Cold Start: What do you do with users who have no or few ratings? </li></ul><ul><li>Sparcity: What do you do if there is little overlap in user ratings across users in the data set? </li></ul><ul><li>Scale: What if there are millions of users? Does this scale well as the number of comparisons increases? </li></ul><ul><li>Real-time: How do you do these calculations in real-time. </li></ul>
  29. 29. Collaborative Filtering: Item-Based Example has more than 29 million customers and several million catalog items. Other major retailers have comparably large data sources. While all this data offers opportunity, it’s also a curse, breaking the backs of algorithms designed for data sets three orders of magnitude smaller. Almost all existing algorithms were evaluated over small data sets.
  30. 30. Collaborative Filtering: Item-Based <ul><li>Assumes that items rated similarly are probably similar. </li></ul><ul><li>Compares items based on the shared appreciation of users, in order to create neighborhoods of similar items. </li></ul><ul><li>The engine then recommends the neighboring items of the user’s known preferred ones. </li></ul>Item-based: i similar to j 5 more than other items Predict ? = 5 j 1 j 2 j 3 j 4 j 5 i u 1 2 3 4 5 ? v 1 1 2 3 4 5 5 v 2 5 4 3 2 1 1
  31. 31. Collaborative Filtering: Item-Based Example
  32. 32. Collaborative Filtering: Item-Based Advantages <ul><li>Scalable: More scalable than the user-based approach because correlations are drawn among a limited number of products, instead of a potentially very large number of users. </li></ul><ul><li>Sparcity: Because the number of items is naturally smaller than the number of users, the item-based approach has a reduced sparsity problem in comparison to the user-based approach. </li></ul>
  33. 33. Collaborative Filtering: There's Money in CF – The Netflix Prize
  34. 34. Collaborative Filtering: Netflix Prize
  35. 35. Collaborative Filtering: Group Lens Rating Data Sets for Testing <ul><li>MovieLens , Wikilens (Beers), Book-Crossing, Jester Joke, HP EachMovie </li></ul>
  36. 36. Collaborative Filtering: Other Applications Anything that can be represented in matrix form where n is a number representing a nominal (e.g. 0,1 for present, absent), ordinal, interval or ratio value
  37. 37. CI from Content: Text Mining Defined <ul><li>Text mining (also known as text data mining or knowledge discovery in textual databases) is the semi-automated process of extracting patterns (useful information and knowledge) from large amounts of unstructured data sources. </li></ul><ul><ul><li>Information extraction. Identification of key phrases and relationships within text by looking for predefined sequences in text via pattern matching. </li></ul></ul><ul><ul><li>Topic tracking. Based on a user profile and documents that a user views, text mining can predict other documents of interest to the user. </li></ul></ul><ul><ul><li>Summarization. Summarizing a document to save time on the part of the reader. </li></ul></ul><ul><ul><li>Categorization. Identifying the main themes of a document and then placing the document into a predefined set of categories based on those themes. </li></ul></ul><ul><ul><li>Clustering. Grouping similar documents without having a predefined set of categories. </li></ul></ul><ul><ul><li>Concept linking. Connects related documents by identifying their shared concepts and, by doing so, helps users find information that they perhaps would not have found using traditional search methods. </li></ul></ul><ul><ul><li>Question answering . Finding the best answer to a given question through knowledge-driven pattern matching. </li></ul></ul>
  38. 38. CI from Content: Resources
  39. 39. CI from Content: Resources <ul><li>Ronen Feldman, “Information Extraction: Theory and Practice,” Bar-Ilan University, ISRAEL, </li></ul><ul><li>Seth Grimes, “Text Analytics for BI/DW Practitioners.” </li></ul><ul><li>Bing Liu, “Opinion Mining & Summarization – Sentiment Analysis.” April 2008. </li></ul>
  40. 40. CI from Content: Some Interesting Data Sets for Research and Training <ul><li>Natural Language Toolkit (NLTK): </li></ul><ul><ul><li> Diverse set of “corpora,” used in conjunction with Natural Language Processing with Python . </li></ul></ul><ul><ul><li>Data set description: </li></ul></ul><ul><li>Enron Email </li></ul><ul><ul><li>500K Enron emails sent primarily by sr. managers over a 3.5 year period covering the height of the scandal. There are multiple versions of the set including database versions. </li></ul></ul><ul><ul><li> </li></ul></ul><ul><li>9/11 Pager Messages </li></ul><ul><ul><li>Approximately 500K messages sent in and around WTC area before, during, and after the attacks </li></ul></ul><ul><ul><li> </li></ul></ul><ul><li>Web Site APIs </li></ul><ul><ul><li> </li></ul></ul><ul><ul><li>Technorati </li></ul></ul><ul><ul><li>Twitter </li></ul></ul><ul><li>Web Sites Devoted to Data Sets </li></ul><ul><ul><li> </li></ul></ul><ul><ul><li> </li></ul></ul>
  41. 41. CI from Content: Text Mining Process
  42. 42. CI from Content: Preparing Data for Term Document Matrix <ul><li>Tokenization —Parse the text to generate terms. Sophisticated analyzers can also extract phrases from the text. </li></ul><ul><li>Normalize — Convert them to lowercase. </li></ul><ul><li>Eliminate stop words — Eliminate terms that appear very often (e.g. the). </li></ul><ul><li>Stemming — Convert the terms into their stemmed form—remove plurals. </li></ul>
  43. 43. CI for Unstructured Contents: Analyzing Blogs
  44. 44. CI for Unstructured Contents: Analyzing Blogs (RSS Feed)
  45. 45. CI for Unstructured Contents: Analyzing Blogs (Source of RSS Feed)
  46. 46. CI for Unstructured Contents: Structure of an “Atom” Feed
  47. 47. CI for Unstructured Contents: Preparing Blog RSS Feed for Analysis … <ul><li>Access & Parse Feed </li></ul><ul><li>Retrieve contents of each entry </li></ul>Collection of Entry Contents (HTML) for each Blog <ul><li>Normalize </li></ul><ul><li>Remove stop words </li></ul><ul><li>Stem </li></ul>List of word stems for each entry <ul><li>Compute the word/stem counts for each word in each collection </li></ul><ul><li>Select words for analysis based on word counts </li></ul>Matrix of Word counts by Blog Subset of Words for Analysis Collection of Blogs RSS Feed Entry1 Entry2 <ul><li>Compute word counts for each Blog by summing word counts across entries </li></ul>Matrix of Word counts for each Blog
  48. 48. CI for Unstructured Contents: Word Counts for Collection of Blogs
  49. 49. CI from Content: Data Mining applied to Prepared Text Data <ul><li> </li></ul>
  50. 50. CI for Unstructured Contents: Blog Dendogram
  51. 51. CI for Unstructured Contents: Blog Results for K-Means Clustering
  52. 52. CI from Content: Simple Example “We Feel Fine” <ul><li>Scours the Internet every ten minutes, harvesting human feelings from a large number of blogs (generally identifying and saving between 15,000 and 20,000 feelings per day. ). </li></ul><ul><li>Scans blog posts for sentences with the phrases &quot;I feel&quot; and &quot;I am feeling“, extracts the sentence, and look to see if it includes one of about 5,000 pre-identified &quot;feelings&quot;. If a valid feeling is found, the sentence is said to represent one person who feels that way. </li></ul><ul><li>URL format of many blog posts can be used to extract the username of the post's author which is used to extract the age, gender, country, state, and city of the blog's owner. </li></ul><ul><li>Given the country, state, and city, we can then retrieve the local weather conditions for that city at the time the post was written. We extract and save as much of this information as we can, along with the post. </li></ul>
  53. 53. CI from Content: Simple Example “We Feel Fine” Visualizations Madness Murmerings Montage Mounds Metrics Mobs
  54. 54. CI from Content: 9/11 Pager Data 2001-09-11 08:52:46 Skytel [002386438] B ALPHA||Reports of a plane crash near World Trade Center - no more details at this point. WNBC's LIVE pix - Network working on coverage.
  55. 55. CI from Content: 9/11 Pager Data
  56. 56. Dataveillance: Roger Clarke <ul><li>The systematic use of personal data systems in the investigation or monitoring of the actions or communications of one or more persons. </li></ul><ul><li>The terms personal surveillance and mass surveillance are commonly used, but seldom defined </li></ul><ul><ul><li>Personal surveillance is the surveillance of an identified person. In general, a specific reason exists for the investigation or monitoring. </li></ul></ul><ul><ul><li>Mass surveillance is the surveillance of groups of people, usually large groups. In general, the reason for investigation or monitoring is to identify individuals who belong to some particular class of interest to the surveillance organization. </li></ul></ul>
  57. 57. Dataveillance: Resources
  58. 58. Dataveillance Data Mining & Social Network Analysis ChoicePoint (17B records) Acxiom Equifax (400M credit holders) Experian … Internet & Other Communication Data Sources
  59. 59. Issues with Privacy and Dataveillance <ul><li>Worldwide Privacy Protection </li></ul><ul><ul><li>There is a tangled matrix of laws and regulations world-wide governing the privacy and protection of this data </li></ul></ul><ul><ul><li>Anytime we interaction on the Web were likely to cross a number of jurisdictions </li></ul></ul><ul><li>Widely held belief that our data produced from our activities is protected. </li></ul><ul><li>Widely held belief among internet users that it’s hard to identify or link specific traces & trails with specific </li></ul><ul><li>Law enforcement and intelligence agencies (worldwide) are persistent in the requests for internet and communications data </li></ul>
  60. 60. Re-Identifiability of Information <ul><li>Deals with the linkage of datasets without explicit identifiers such as name and address. </li></ul><ul><li>Examples of Re-identification </li></ul><ul><ul><li>Large portion of the US can be re-identified using a combination of 5-digit ZIP code, gender and date of birth. </li></ul></ul><ul><ul><li>AOL case 4417749 (2006 release of 20 million search queries of over 650,000 users </li></ul></ul><ul><ul><li>CMU study of predicting SSNs -- it is possible to guess many -- if not all -- of the nine digits in an individual's Social Security number using publicly available information (about location and birth date) </li></ul></ul>