This is a presentation given by Ana Martinez

1. Ana Martinez
Kin Lane
February 2012 M.C. Escher

4. The problem

5. Big Bottleneck!

6. Single POF!

8. Places Processing

9. Places Processing
Source 2
• Name
• Address
• Phone
• reviews
Source 1 Source 3
• Name • Name
• Address • Address
• Phone • Phone
• Images • menu
CityGrid
Place

10. Why is it hard?
Book is to ISBN what Product is to UPC and what Place is to ______
No centrally regulated unique id (tax id is, but not public). Now what?
Spago
176 Canon Dr
Beverly Hills, CA 90210
310-944-3924
R. French Ac & Heating Inc Ray French Air Conditioning & Heating
Service
2211 martin luther king blvd 2211 MLK boulevard #104
los angeles, CA, 90069 west Hollywood, CA, 90069
310-358-5903 866-465-5303

11. Problem Definition
• Medium size data set
– 21mill rows, 120 cols
• Time to process: Daily
• Hybrid environment
• Not all data is from same source

12. Solution
Normalizer Matcher Merger

13. Normalizer
Soundex Metaphone NYSIIS
Matching
Rating Coverphone
Approach

14. Know Your Data
Stop Words
• The Viper Room Viper Room
Stemming
• av aven avenu
• avenue avn avnue
Compression
• county line county rd county road
Trunction
• apt unit #

15. Normalizer
123 Martin Luther King.n
123 MartinLutherKing.
123 martinlutherking.
Martin Luther King | martinlutherking
canon column
the | n | ave | (tokens)

16. Matching Strategy
Do what you can on automated fashion and
complement with manual steps.

17. Matching Strategy
Exact matching
Set similarity joins
Custom fuzzy matching

18. Matching Strategy
• C - Support Vector Machine
• Threashold: 0.996
– Precision: 98.1%
– Recall: 97.5%
84% + manual -> % Match Rate

19. Merger
Rules:
Provider truthworthiness
Voting rules
New data vs Old data
Super providers
History:
Accepted
Rejected

20. Example
123 M L K Road Ste 45 123 Martin Luther King Rd 123 Martin L King Drive #45
123 m l k road ste 45 123 martinluther king rd 123 martin l king drive #45
(123) (m) (l) (k) (road) (123) (martin) (luther) (123) (martin) (l) (king)
(ste) (45) (king) (rd) (drive) (#) (45)
123 mlk road ste 45 123 martinlutherkingrd 123 martinlking drive # 45
123 mlkrdste 45 123 mlkrd 123 mlkdr #45
123 mlkrd 123 mlkrd 123 mlkdr
123 mlk 123 mlk 123 mlk
MATCH! MATCH! MATCH!

21. Findings & Tips
• Domain Knowledge
• Automation
• Mechanical Turk
• Machine Learning
Run every 2hrs -> Match Rate of %

24. Solution for Search APIs

26. Solution for Places API

29. Performance Results

30. Updates
• Hours
• Real Time

32. Places Detail – Demo Time!
• Details by ID
– http://api.citygridmedia.com/content/places/v2/detail?listing_i
d=11280452&client_ip=123.4.56.78&publisher=test
– http://api.citygridmedia.com/content/places/v2/detail?public_i
d=pinks-hot-dogs-los-angeles-
2&client_ip=123.4.56.78&publisher=test

33. Improvements
• Shard Listing and Content Data
• Integrate Mongo across all APIs

34. APIs
Now we have rich Places API
How do we make developers aware they exist?
How do we get them to successfully integrate?

35. APIs – Supporting Developer Area
Common Building Blocks
• Getting Started
•Terms of Use
Publisher Overview
• Documentation
• FAQ
• Terms of Use

36. APIs – Supporting Developer Area
Developers Tools
• Code Samples
•Terms of Use
Libraries
• Mobile SDKs
• Starter Kits
• Hackathon Toolkits
• Partner APIs

37. APIs – Evangelism - Online
• Blogging
• Twitter
• LinkedIn
• Facebook of Use
Terms
• Github
• Stack Overflow
• Quora
• Hacker News
• StumbleUpon
• Reddit

38. APIs – Evangelism - Offline
• Conferences
• Hackathons
Terms of Use
• Meetups
• Workshops

39. APIs – Easy Start + Engage Immediately
• Testable APIs
• Self-Service
Terms of Use
• Email After Registration
• Follow on Twitter
• Follow on LinkedIn

40. APIs – Feedback Loop + Voice
• Email Support
• Forum(s) of Use
Terms
• Twitter
• LinkedIn

41. APIs – Monetization = Sustainability
• Local Web Advertising
• Local Mobile Advertising
Terms of Use
• Local Custom Ads
• Places that Pay

42. APIs – Evangelize Internally
• Developer Feedback
• Roadmap Suggestions
Terms of Use
• Landscape Analysis
• Technology Awareness
• Trends
• Internal Hackathons

43. APIs – Measure & Repeat
Terms of Use