Harnessing the Social Web: The Science of Identity Disambiguation<br />Matthew Rowe and Fabio Ciravegna<br />Organisations...
Outline<br />Problem<br />Dissemination of personal information across the Web<br />Motivation<br />Need for automation<br...
Problem<br />Large amount of information now residing on the World Wide Web is personal information<br />Disseminated volu...
Motivation<br />To avoid such practices, web users must manually collect web resources which may cite them and then decide...
Harnessing the Social Web<br />Overcome the problem of producing seed data manually by harnessing the Social Web<br />Soci...
Collecting Seed Data from the Social Web<br />
Collecting Seed Data from the Social Web<br /><http://www.dcs.shef.ac.uk/~mrowe/foaf.rdf#me><br />rdf:typefoaf:Person ;<br...
Identity Disambiguation: Inference Rules<br />Rules provides a means to logically infer conclusions based on the presence ...
Identity Disambiguation: Inference Rules<br />
Identity Disambiguation: Self-training<br />Self-training provides a semi-supervised approach to disambiguation:<br />Seed...
Identity Disambiguation: Self-training<br />Intuition is that as the classifier learns from unlabelled data it will learn ...
Evaluation<br />Dataset<br />50 members of the Semantic Web and Web 2.0 communities<br />Collected seed data from Facebook...
Evaluation: Inference Rules<br />Yields high levels of precision, but poor recall scores<br />Specific nature of rules lea...
Evaluation: Self-training<br />Perceptron and SVM are combined with Entailment outperform humans<br />Due to the large lev...
Conclusions<br />Social Web platforms provide a useful source for identity information<br />Significant similarity between...
Twitter: @mattroweshow<br />Web: http://www.dcs.shef.ac.uk/~mrowe<br />Email: m.rowe@dcs.shef.ac.uk<br />Questions?<br />(...
Upcoming SlideShare
Loading in …5
×

Harnessing the Social Web: The Science of Identity Disambiguation

896 views
825 views

Published on

Slides from Web Science Conference 2010

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
896
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Harnessing the Social Web: The Science of Identity Disambiguation

  1. 1. Harnessing the Social Web: The Science of Identity Disambiguation<br />Matthew Rowe and Fabio Ciravegna<br />Organisations, Information and Knowledge Group<br />University of Sheffield, UK<br />Web Science 2010<br />
  2. 2. Outline<br />Problem<br />Dissemination of personal information across the Web<br />Motivation<br />Need for automation<br />Harnessing the Social Web<br />Identity Disambiguation<br />Inference Rules<br />Self-training<br />Evaluation<br />Results<br />Conclusions<br />
  3. 3. Problem<br />Large amount of information now residing on the World Wide Web is personal information<br />Disseminated voluntarily: homepages, profiles pages<br />Or involuntarily: telephone directories, electoral registers<br />Sensitive nature of this information has lead to:<br />Identity Theft: act of stealing a person’s identity and reusing it<br />Currently costs UK economy £1.2 billion<br />http://www.identitytheft.org.uk/faqs.asp<br />Lateral Surveillance: act of watching someone without their knowledge<br />Often performed by employers vetting potential employees<br />And by socialities vetting prospective dates<br />Could affect reputation if detrimental content exists<br />
  4. 4. Motivation<br />To avoid such practices, web users must manually collect web resources which may cite them and then decide which do<br />The latter stage of this process is referred to as disambiguation<br />Decides which web resources are references and produces a unary set of identity web references for a given person<br />However, this practice is <br />Time consuming<br />Expensive<br />Must be repeated often as the more and more data is published on the Web<br />Automated disambiguation techniques can replace this manual processing<br />To function effectively however, seed data (background knowledge about a person) is required:<br />Expensive to produce (e.g. filling in an extensive form)<br />Must contain sufficient features describing a person’s identity<br />
  5. 5. Harnessing the Social Web<br />Overcome the problem of producing seed data manually by harnessing the Social Web<br />Social Web platforms such as Facebook, Twitter and MySpace allow web users to build an online persona/identity visible to others<br />Sociological studies have argued of the similarity between online and offline identities<br />(Hart et al, 2008) states that online social networks are merely extensions of offline lives<br />(Ellison et al, 2007) states that Social Web platforms are used to reinforce established offline relationships<br />A user study was conducted to assess the relationship between digital identities constructed on Social Web platforms and their real worldequivalent using 50 participants from the University of Sheffield (25 male, 25 female) with a wide age range (18 – 45)<br />Study consisted of three stages<br />1. Participants listed their real world social network<br />2. Digital social network was extracted from Facebook for each participant<br />3. Digital and real world networks were compared<br />Relevance: proportion of digital social network containing strong-tied relationships<br />Coverage: extent of to which the real world network is replicated online<br />Results from the user study show<br />Coverage range of 0.5 to 1 with an average of 0.77<br />Indicating that, on average, 77% of a person’s real world social network is replicated online<br />Average relevance of 0.23<br />Indicating that, on average, 23% of a person’s digital social network contains strong tied relationships<br />
  6. 6. Collecting Seed Data from the Social Web<br />
  7. 7. Collecting Seed Data from the Social Web<br /><http://www.dcs.shef.ac.uk/~mrowe/foaf.rdf#me><br />rdf:typefoaf:Person ;<br />foaf:name "Matthew Rowe" ;<br />foaf:homepage <www.dcs.shef.ac.uk/~mrowe> ;<br />foaf:mbox <m.rowe@dcs.shef.ac.uk> ;<br />foaf:based_near <http://www.geonames.org/2638077> ;<br />foaf:knows <http://www.dcs.shef.ac.uk/~mrowe/foaf.rdf#fabio> ;<br />foaf:knows <http://www.dcs.shef.ac.uk/~mrowe/foaf.rdf#sam> .<br /><http://www.dcs.shef.ac.uk/~mrowe/foaf.rdf#fabio><br />rdf:typefoaf:Person ;<br />foaf:name "Fabio Ciravegna";<br />foaf:mbox <fabio@dcs.shef.ac.uk>;<br />foaf:homepage <http://www.dcs.shef.ac.uk/~fabio> .<br /><http://www.dcs.shef.ac.uk/~mrowe/foaf.rdf#sam><br />rdf:typefoaf:Person ;<br />foaf:name "Sam Chapman" ;<br />foaf:mbox <sam@dcs.shef.ac.uk> ;<br />foaf:homepage <http://www.dcs.shef.ac.uk/~sam> .<br /><http://www.geonames.org/2638077><br />rdf:typegeo:Feature ;<br />geo:name “Sheffield” ;<br />geo:inCountry “UK” .<br />
  8. 8. Identity Disambiguation: Inference Rules<br />Rules provides a means to logically infer conclusions based on the presence of information<br />In the context of identity disambiguation, rules replicate the cognitive process by which a human decides if a web resource refers to a given entity<br />Using background knowledge known about the entity<br />Uses a supervised approach by only using the provided seed data to make decisions<br />Rules are built from the seed data as follows:<br />RDF instances are extracted from the seed data (e.g. an instance of a person or location)<br />A rule is constructed from the information in each instance description<br />Rules are then added to the rule base which are then applied to a collected set of web resources to disambiguate identity web references<br />If the triple pattern in the antecedent (the if part) of the rule matches the knowledge structure of a web resource then a web reference is inferred<br />
  9. 9. Identity Disambiguation: Inference Rules<br />
  10. 10. Identity Disambiguation: Self-training<br />Self-training provides a semi-supervised approach to disambiguation:<br />Seed data collected from the Social Web provides the positive training data<br />Possible web citations provide the unlabelled data<br />Negative training data is generated using Rocchio classification over the unlabelled data<br />Positive and negative training data is then used to train an initial classifier<br />Classifier is applied to the unlabelled data and labels each example<br />Training sets (positive and negative) are enlarged with the examples from the unlabelled data which exhibit the strongest classification confidences<br />Examples are removed from the unlabelled data, reducing its size<br />Steps 4-7 are repeated until all unlabelled data has been classified<br />Tested 3 different machine learning classifiers: Perceptron, Support Vector Machines and Naïve Bayes<br />RDF models (for both the seed data and the web resources) are converted into machine learning instances<br />RDF instances from the models are used as featuresfor the machine learninginstances<br />This permits the variation of distinct feature similarity measures between 3 different RDF graph matching techniques: <br />RDF Entailment: does one graph subsume that of another?<br />Inverse Functional Property Matching: do property values match in distinct graphs where the property is inverse functional?<br />Jaccard similarity (strict graph equivalence): are the graphs identical?<br />
  11. 11. Identity Disambiguation: Self-training<br />Intuition is that as the classifier learns from unlabelled data it will learn from previously unknown features<br />Seed data only covers a portion of a person’s identity<br />Will lead to the detection of more web references<br />This is similar to the cognitive process by which humans identify web citations <br />Only a portion of background knowledge is known at the start<br />As more web references are found, the knowledge of the person is expanded<br />
  12. 12. Evaluation<br />Dataset<br />50 members of the Semantic Web and Web 2.0 communities<br />Collected seed data from Facebook and Twitter<br />Collected possible web citations from searching WWW and the Semantic Web for each participant<br />Converted each returned resource into an RDF model representation<br />~346 web resources to be analysed for each participant<br />Evaluation Measures<br />Information retrieval metrics: precision, recall and f-measure<br />Web presence level: proportion of web resources that refer to each participant (e.g. 50 of 350 web resources refer to a given person, then web presence is 14%)<br />Baseline Measure: Human Processing<br />Group of 12 raters manually processed a portion of the dataset for each participant<br />3 raters performed disambiguation for each participant, then used interrater agreement (Hripcsak & Rothschild, 2005) to calculate IR metrics<br />
  13. 13. Evaluation: Inference Rules<br />Yields high levels of precision, but poor recall scores<br />Specific nature of rules leads to poor application to new instances <br />Consistently outperforms humans in terms of precision for all web presence levels<br />At low levels of web presence, where web references are sparse, humans perform poorly<br />This is characterised by a “Needle in a Haystack” problem<br />Inhibited by the lack web references to learn from<br />
  14. 14. Evaluation: Self-training<br />Perceptron and SVM are combined with Entailment outperform humans<br />Due to the large levels of recall achieved by these permutations<br />Entailment leads to a reduction in overfitting to the training data<br />Precision is lowered, but recall is improved significantly<br />Performance also remains consistent for all web presence levels<br />Jaccard uses strict matching between RDF instances, leading to high precision levels<br />Poor recall levels due to overfitting to training data <br />unable to generalise to new instances <br />
  15. 15. Conclusions<br />Social Web platforms provide a useful source for identity information<br />Significant similarity between real world and digital social networks<br />This can in turn be used to support automated disambiguation techniques<br />Inference Rules, using a supervised strategy, yields high precision levels yet fails to detect a large portion of identity web references<br />Self-training overcomes the limitations of supervised techniques by learning from disambiguation decisions<br />High recall levels demonstrate the effectiveness of such methods to detect a large portion of web references<br />Future work will look to combine these two methods together<br />Enlarging the positive training data using Inference Rules – given their high precision levels<br />Then applying Self-training to increase recall levels<br />
  16. 16. Twitter: @mattroweshow<br />Web: http://www.dcs.shef.ac.uk/~mrowe<br />Email: m.rowe@dcs.shef.ac.uk<br />Questions?<br />(Hart et al, 2008) - J. Hart, C. Ridley, F. Taher, C. Sas, and A. Dix. Exploring the facebook experience: a new approach to usability. In NordiCHI ’08: Proceedings of the 5th Nordic conference on Human-computer interaction, pages 471–474, New York, NY, USA, 2008. ACM<br />(Ellison et al, 2007) - N. B. Ellison, C. Steinfield, and C. Lampe. Thebenefits of facebook friends: Social capital and college students’ use of online social network sites. Journal of Computer Mediated Communication, 12:1143–1168, 2007.<br />(Hripcsak & Rothschild, 2005) – G. Hripcsak and A. S. Rothschild. Agreement, the f-measure, and reliability in information retrieval. Journal of American Medical Informatics Association, 12(3):296–298, 2005. <br />

×