Approximate string comparators measure the similarity between two strings when an exact match is insufficient. Levenshtein distance measures the minimum number of edit operations (insert, remove, substitute characters) required to change one string into another. Jaro-Winkler distance compares characters and transpositions within a threshold and is commonly used for name comparisons. Soundex and Metaphone produce phonetic codes for strings to match similar-sounding names irrespective of spelling variations. There are many string similarity measures for different use cases.

1. Approximate string comparators
TvungenOne, 2012-06-15
Lars Marius Garshol, <larsga@bouvet.no>
http://twitter.com/larsga
1

2. Approximate string comparators?
• Basically, measures of the similarity between
two strings
• Useful in situations where exact match is
insufficient
– record linkage
– search
– ...
• Many of these are slow: O(n2)
2

3. Levenshtein
• Also known as edit distance
• Measures the number of edit operations
necessary to turn s1 into s2
• Edit operations are
– insert a character
– remove a character
– substitute a character
3

4. Levenshtein example
• Levenshtein -> Löwenstein
– Levenstein (remove „h‟)
– Lövenstein (substitute „ö‟)
– Löwenstein (substitute „w‟)
• Edit distance = 3
4

5. Weighted Levenshtein
• Not all edit operations are equal
• Substituting “i” for “e” is a smaller edit than
substituting “o” for “k”
• Weighted Levenshtein evaluates each edit
operation as a number 0.0-1.0
• Difficult to implement
– weights are also language-dependent
5

6. Jaro-Winkler
• Developed at the US Bureau of the Census
• For name comparisons
– not well suited to long strings
– best if given name/surname are separated
• Exists in a few variants
– originally proposed by Winkler
– then modified by Jaro
– a few different versions of modifications etc
6

7. Jaro-Winkler definition
• Formula:
– m = number of matching characters
– t = number of transposed characters
• A character from string s1 matches s2 if the
same character is found in s2 less then half the
length of the string away
• Levenshtein ~ Löwenstein = 0.8
• Axel ~ Aksel = 0.783
7

8. Jaro-Winkler variant
8

9. Soundex
• A coarse schema for matching names by sound
– produces a key from the name
– names match if key is the same
• In common use in many places
– Nav‟s person register uses it for search
– built-in in many databases
– ...
9

10. Soundex definition
10

11. Examples
• soundex(“Axel”) = „A240‟
• soundex(“Aksel”) = „A240‟
• soundex(“Levenshtein”) = „L523‟
• soundex(“Löwenstein”) = „L152‟
11

12. Metaphone
• Developed by Lawrence Philips
• Similar to Soundex, but much more complex
– both more accurate and more sensitive
• Developed further into Double Metaphone
• Metaphone 3.0 also exists, but only available
commercially
12

13. Metaphone examples
• metaphone(“Axel”) = „AKSL‟
• metaphone(“Aksel”) = „AKSL‟
• metaphone(“Levenshtein”) = „LFNX‟
• metaphone(“Löwenstein”) = „LWNS‟
13

14. Dice coefficient
• A similarity measure for sets
– set can be tokens in a string
– or characters in a string
• Formula:
14

15. TFIDF
• Compares strings as sets of tokens
– a la Dice coefficient
• However, takes frequency of tokens in corpus
into account
– this matches how we evaluate matches mentally
• Has done well in evaluations
– however, can be difficult to evaluate
– results will change as corpus changes
15

16. More comparators
• Smith-Waterman
– originated in DNA sequencing
• Q-grams distance
– breaks string into sets of pieces of q characters
– then does set similarity comparison
• Monge-Elkan
– similar to Smith-Waterman, but with affine gap distances
– has done very well in evaluations
– costly to evaluate
• Many, many more
– ...
16