Presenter: Ken Krugler, President of Scale Unlimited Early Warning has information on hundreds of millions of people and companies. When a person wants to open a new bank account, they need to be able to accurately find similar entities in this large dataset, to provide a risk assessment. Using the combination of Cassandra & Solr via DSE, they can quickly find and evaluate all reasonable candidates.

1. Fuzzy Entity Matching
Ken Krugler | President, Scale Unlimited

2. whoami
•Ken Krugler, Scale Unlimited - Nevada City, CA
•Consulting on big data (workflows, search, etc)
•Training for Hadoop, Cascading, Solr & Cassandra

3. The Problem

4. Should I Trust You?
•When opening a bank account...
•...what is the applicant's risk?
!
•Key is matching person...
•...to other account info

5. Matching people
•I have some information you've provided
•I need to match against ALL bank data
•But banks won't exchange their customer info
•So what can we do?

6. Early Warning Services
•Owned by the top 5 US banks
•Gets data from 800+ financial institutions
•So they have details on most US bank accounts

7. Fuzzy Matching

8. What's a fuzzy match?
•Match everything that's equivalent
!
≅
!
•Match nothing that's different
≇

9. Why is it hard?
•Lots of gray areas in fuzzy matching
≟
!
•Can't use exact key join
•So no easy lookup using C* row key
•Often computationally intensive

10. Matching People
•I've got information on lots of people
•I'm being asked about a specific person
•How to quickly find all good matches?
•Not doing batch matching ≟

11. What's a Good Match?
•Comparing field values between records
•Are these two people the same?
Name Bob Bogus Robert Bogus
Address 220 3rd Ave 220 3rd Avenue
City Seattle Seattle
State WA WA
ZIP 98104-2608 98104

12. What about now?
•Normalization becomes critical
•How to focus on the important features?
Name Bob Bogus Robert H. Bogus
Address Apt 102, 220 3rd Ave 3220 3rd Avenue South
City Seattle Seattle
State Washington WA
ZIP 98104

13. How do you calc similarity?
•Calculate degree of similarity for each field (0 -> 1.0)
•Give each field a weight (these sum to 1.0)
•Score is sum(fieldN sim * fieldN weight)
•So score is 0 (nothing in common) to 1.0 (exact dup)

14. Does that scale?
•For a given person being matched...
•You need to compare to every other person
•Works for a few thousand people
•Doesn't scale for 100s of millions of people

15. Search to the Rescue

16. Search is (fast) similarity
•Find N most similar docs to this doc (my query)
•Each doc has multi-dimensional feature vector
•Each feature (dimension) is a unique word
•Feature weight is TF * IDF

17. Cosine Similarity
•Each document has a term vector
•E.g. three unique words x, y, z
•Weight is TF*IDF of each word
•Calc cosine of angle between 2 vectors
•That is the similarity score

18. Cosine sim ≢ match sim
•Doesn't have same level of sophistication
•So throw a bigger net to find candidates
•e.g. get top N*X, assuming at most X matches
•Then do match similarity calc on this (small) set

19. So two-step process
Match
0.90
0.50
0.10
0.85
...
Query: name=“Bob Bogus”^3
and ssn=“222447777”^10
and dob=“19600723”^5
Solr
Index
Name SSN DOB
Bob Bogus 222447777 19610603
Robert Bogus 193618919 19600723
Bob Smith 479385821 19600723
Sam Stealthy 222447777 19930523
Name SSN DOB
Bob Bogus 222447777 19600723
... ... ...

20. How do you pick N?
•Can be small, if match sim ≈ search sim
•If N is too big, it's inefficient
•If N is too small, you miss matches
•Tune search to mimic match sim
•Right tradeoff depends on use case

21. What is Solr?
•Enterprise search system, build on top of Lucene
•Open source project at Apache Software Foundation
•Scales to billions of documents
•Highly configurable & customizable
•Integrated with Cassandra in DSE

22. Solr Schema
•Defines set of fields in a document
•Direct one-to-one mapping with Cassandra columns
•Fields can be defined with synonyms, etc., etc.
<fields>
<field name="key" type="string" indexed="true" stored="true" />
<field name="name" type="text" indexed="true" stored="true" />
</fields>

23. DSE Search with Solr

24. What is DSE with Solr?
•DSE-specific enhancement to Cassandra
•Keeps a Solr index in sync with a C* table
•Indexes distributed to all nodes C* &
Solr
C* &
Solr
C* &
Solr
C* Table
S* Index
C* Table
S* Index
C* Table
S* Index

25. Handy replication & failover
•Implementation leverages C* replication
•So you get load balancing, reliability, scalability
•You can replicate from a regular C* DC to Solr DC
C* &
Solr
C* &
Solr
Solr DC C* DC
C* &
Solr
C* C*
C*

26. Who builds the index?
•In background
•Much slower than
C* updates
•Uses existing
secondary index
hook
Secondary
Index Hook
Distribute to
indexing queues
Logical Rows
Indexing
Queue
Read C* storage row
back_pressure_threshold_per_core
max_solr_concurrency_per_core
Create one Solr doc
per entry
Apply
FieldInputTransformer Update Solr

27. How fast is it?
•Writing 170M records ≈ 2.5 hours
•8 node DSE 4.0 cluster, 8 1TB SSDs on each
•This is indexing during writes
•About 15% of index available when writes finish
•Complete index takes another 12 hours

28. System Overview

29. ETL Hadoop Workflow
•Extract, transform, load
•Built using Cascading API
•Parse data, simple normalization
•Other transformations happen in Solr

30. Cassandra ingress
•Reduce tasks in Hadoop talk to C* cluster
•Using DataStax Java driver for Cassandra
•Bottleneck is Solr indexing
•Inserts get throttled when this falls behind
•But total time less than with deferred indexing

31. Architectural Diagram
C* +
Solr
C* +
Solr
C* +
Solr
Hadoop
Cluster
Entity
Matcher API

32. Ingest performance
•For max performance, write without reads
•But how to avoid creating duplicate entries?
•Set the row key to the hash of searchable fields
•Accept "near duplicates" in search results
•Possible to push some Solr load into workflow

33. Summary

34. Key points to remember
•This is for ad hoc requests, not batch deduplication
•Use search to reduce candidate set, then match
•Pain is in normalization, matching logic
•DSE + Solr simplifies architecture & adds goodness

35. More questions?
•Feel free to contact me
•http://www.scaleunlimited.com/contact/
•Get training on DSE with Solr
•http://www.datastax.com/what-we-offer/products-services/
training