Linking Really Big Data Andrew Borthwick, Ph.D. Principal Scientist and Director of Data Research Intelius, Inc. February ...
About Intelius
“ Vulcan” Record Linkage Engine
How Vulcan matches records Machine Learning Decider
Societal Implications <ul><li>Privacy </li></ul><ul><li>Benefits and Hazards </li></ul><ul><li>Tolerance for error </li></ul>
Privacy by Obscurity <ul><li>Pre-Intelius, data used to require a private eye and a trip to the court house </li></ul><ul>...
Benefits and Hazards <ul><li>Benefits </li></ul><ul><ul><li>Know if your date is a criminal or if a car pool parent has DU...
Mitigating the bad stuff <ul><li>Search engines can work with organizations representing people who don’ t want to be foun...
Basic Accuracy Measures <ul><li>Precision  </li></ul><ul><ul><li>True Positive/(True Positive + False Positive) </li></ul>...
Error Tolerance (1) <ul><li>Do you care more about precision or recall error? </li></ul><ul><ul><li>For a search engine, a...
Error Tolerance (2) <ul><li>Decisions are made based on probabilities.  Can’ t be 100% certain </li></ul><ul><ul><li>Possi...
Questions? <ul><li>Andrew Borthwick, Ph.D. </li></ul><ul><li>Principal Scientist and Director of Data Research </li></ul><...
Upcoming SlideShare
Loading in...5
×

I conference big data_andrew borthowick

550

Published on

This panel will address issues encountered when working with large data sets to bring about information, understanding, meaning and value. It will touch upon databases, sensors, networks, security, privacy, visualization, analytics, search, perceptual & cognitive psychology, business strategy and more.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
550
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Demonstrated Vulcan’s viability A/B test will show how we compare to Accurint
  • Had a friend whose wife cheated on him. She moved away. Sent a letter with no return address.
  • Also true for an immunization registry
  • I conference big data_andrew borthowick

    1. 1. Linking Really Big Data Andrew Borthwick, Ph.D. Principal Scientist and Director of Data Research Intelius, Inc. February 9, 2011
    2. 2. About Intelius
    3. 3. “ Vulcan” Record Linkage Engine
    4. 4. How Vulcan matches records Machine Learning Decider
    5. 5. Societal Implications <ul><li>Privacy </li></ul><ul><li>Benefits and Hazards </li></ul><ul><li>Tolerance for error </li></ul>
    6. 6. Privacy by Obscurity <ul><li>Pre-Intelius, data used to require a private eye and a trip to the court house </li></ul><ul><ul><li>Marriage </li></ul></ul><ul><ul><li>Civil Suits </li></ul></ul><ul><ul><li>Property ownership and property tax </li></ul></ul><ul><ul><li>Criminal records </li></ul></ul><ul><li>Difficult to track people down who don’ t want to be found </li></ul>
    7. 7. Benefits and Hazards <ul><li>Benefits </li></ul><ul><ul><li>Know if your date is a criminal or if a car pool parent has DUI’s </li></ul></ul><ul><ul><li>Women know where abusive ex-husband is living </li></ul></ul><ul><li>Hazards </li></ul><ul><ul><li>Abusive ex-husband knows where ex-wife is living </li></ul></ul><ul><ul><li>Dogged for life by youthful indiscretions </li></ul></ul>
    8. 8. Mitigating the bad stuff <ul><li>Search engines can work with organizations representing people who don’ t want to be found </li></ul><ul><ul><li>Battered women’ s shelters </li></ul></ul><ul><ul><li>Police officers unions </li></ul></ul><ul><li>People can correct their data or opt-out of Intelius </li></ul>
    9. 9. Basic Accuracy Measures <ul><li>Precision </li></ul><ul><ul><li>True Positive/(True Positive + False Positive) </li></ul></ul><ul><li>Recall </li></ul><ul><ul><li>True Positive/(True Positive + False Negative) </li></ul></ul>Machine Match Machine No-Match Human Match True Positive False Negative Human No-match False Positive True Negative
    10. 10. Error Tolerance (1) <ul><li>Do you care more about precision or recall error? </li></ul><ul><ul><li>For a search engine, a precision error is worse than a recall error </li></ul></ul><ul><ul><ul><li>Don’ t falsely mix two people’s profiles </li></ul></ul></ul><ul><ul><li>For an airline, a recall error is worse </li></ul></ul><ul><ul><ul><li>Need to identify all possible terrorists who are getting on the plane </li></ul></ul></ul><ul><li>There is always a tradeoff between precision and recall </li></ul>
    11. 11. Error Tolerance (2) <ul><li>Decisions are made based on probabilities. Can’ t be 100% certain </li></ul><ul><ul><li>Possible that there could be two people named “Andrew Borthwick” in Kirkland, WA </li></ul></ul>
    12. 12. Questions? <ul><li>Andrew Borthwick, Ph.D. </li></ul><ul><li>Principal Scientist and Director of Data Research </li></ul><ul><li>Intelius, Inc. </li></ul><ul><li>[email_address] </li></ul>

    ×