Fuzzy Search in Ruby by Paweł Strzałkowski. How to find and replace when you don't really know what to look for.

1. Fuzzy Text Search in Ruby
How to find and replace when you
don't really know what to look for
Paweł Strzałkowski

2. Task: Replace company name with "The Company" in a job offer description
Take
- Job offer description
- Company name

3. Task: Replace company name with "The Company" in a job offer description
Take
- Job offer description
- Company name
Simple?
> description.gsub(company_name, 'The Company')

4. Task: Replace company name with "The Company" in a job offer description
Thank you for your attention
Take
- Job offer description
- Company name
Simple?
> description.gsub(company_name, 'The Company')

5. Task: Replace company name with "The Company" in a job offer description
Take
- Job offer description
- Company name
Simple?
Not so much (misspells, custom usages of the company name)
> description.gsub(company_name, 'The Company')

6. Fuzzy Text Comparison: Definition
Fuzzy - difficult to perceive clearly or understand and explain precisely; indistinct or
vague
Fuzzy text comparison doesn't answer whether phrases match
but defines their similarity
Example: Jaro–Winkler similarity level of:
- 1 means phrases are equal
- 0 means phrases are completely different
- Something in-between means some level of difference

7. Fuzzy Text Comparison: Examples
> require 'fuzzy-string-match'
> # (... create a wrapper as fuzzy_similarity method ...)
> fuzzy_similarity "Lorem Ipsum Dorem", "a"
=> 0.0
> fuzzy_similarity "Lorem Ipsum Dorem", "Visuality Test"
=> 0.4393162393162393
> fuzzy_similarity "Visuality", "Visuality"
=> 1.0

8. How to find and replace fuzzy matched phrases in a longer text?
Library?
- Not found...
Best practice?
- Not found...
Custom solution?
- Let's do it!

9. Fuzzy Text Comparison: Assumption #1
- Assumed some level of similarity to be considered "equal"
- Didn't know whether it should be 0.7 or 0.99 so went with something in-
between and tweaked
- It ended up being 0.9
- Whenever it is said that "phrases are equal in fuzzy-matching context", it
means that their Jaro-Winkler similarity is more than 0.9

10. Fuzzy Text Comparison: Assumption #2
- Searched phrase may be longer or shorter than the given term
- 1 word longer
- 2 words longer
- 3 words longer
> fuzzy_similarity "Lorem Ipsum Dorem", "Lorem Ipsum Dorem Carem"
=> 0.9791666666666666
> fuzzy_similarity "Lorem Ipsum Dorem", "Lorem Ipsum Dorem Carem Visuarem"
=> 0.9281462585034014
> fuzzy_similarity "Lorem Ipsum Dorem",
"Lorem Ipsum Dorem Carem Visuarem Odolanem"
=> 0.8865740740740742

11. Fuzzy Text Comparison: Assumption Summary
- Phrases are equal if their similarity level is more than 0.9
- Matching phrase might be up to 2 words longer than the searched one

12. Algorithm (at last)
Assignment: Take a story, find the name "Mary Joan" and replace it
with "The Girl" to make it anonymous. We'll start (and finish) with the story title.
Title: Mary S Jane of house Visuality
Problem: Somebody has
- taken a wrong Mary
- used an additional middle initial
We now have "Mary S Jane" instead of "Mary Joan" in the story.

13. Algorithm: Step 1 - Break Into Pieces
- Searched phrase (Mary Joan) is 2 words long
- Maximal matching phrase length is 4
"Mary S Jane of house Visuality".break_into_pieces_of(4) =>
(N == 1) => ["Mary", "S", "Jane", "of", "house", "Visuality"]
(N == 2) => ["Mary S", "S Jane", "Jane of", "of house", "house Visuality"]
(N == 3) => ["Mary S Jane", "S Jane of", "of house Visuality"]
(N == 4) => ["Mary S Jane of", "S Jane of house", "Jane of house Visuality"]

14. Algorithm: Step 1 - Break Into Pieces (Excercise)
- Take some of the created phrases
- Check how similar they are to the searched term
> fuzzy_similarity "Mary Joan" , "of house Visuality"
=> 0.4583333333333333
> fuzzy_similarity "Mary Joan" , "Mary S"
=> 0.86
> fuzzy_similarity "Mary Joan" , "Mary S Jane of"
=> 0.9156336088154271
> fuzzy_similarity "Mary Joan" , "Mary S Jane"
=> 0.9305555555555555
<< Equal
<< Equal

15. Algorithm: Step 2 - Choose best results
Mary S Jane of
S Jane of
0.92
Mary S Jane 0.510.93
S JaneMary S 0.380.86

16. Algorithm: Ta - Daaaaa
> FuzzyText.new(
"Mary S Jane of house Visuality", "Mary Joan"
).matches
=> ["Mary S Jane"]

17. Algorithm: Tweaks and Gotchas
- Skip characters which make the fuzzy comparison blurry
- Punctuation
- White characters
- Saxon genitives - 's
- Common words (when applicable, ie. "is", "are", "was", "were" etc.)
- Compare only once
- Use a hash table containing the results of every comparison
- Cache the results
- TDD

18. English is too easy, let's take a bit more challenging language
> hero = "Kapitan Ameryka"
> description = "Wbrew powszechnej opinii o Kapitanie z Ameryki nie był on
pierwszym superbohaterem wykorzystującym tematykę flagi Stanów
Zjednoczonych. Poprzedzał go superbohater o pseudonimie Shield czternaście
miesięcy wcześniej niż Kapitan Ameryka. Shield nosił na sobie pancerny
kostium z kuloodporną tarczą na piersi (zwraca się uwagę na uderzające
podobieństwo do wczesnej tarczy Kapitana Ameryki, która w późniejszych
wydaniach była już okrągła). Na dodatek obie postacie łączyła podobna
geneza: Shield tak jak bohater Marvela zyskał nadludzkie moce po zażyciu
serum superżołnierza, którego twórca został zabity przez hitlerowców.
Postać Kapitana Ameryki pojawiała się w wielu różnych adaptacjach."

19. English is too easy, let's take a bit more challenging language
> hero = "Kapitan Ameryka"
> description = "Wbrew powszechnej opinii o Kapitanie z Ameryki nie był on
pierwszym superbohaterem wykorzystującym tematykę flagi Stanów
Zjednoczonych. Poprzedzał go superbohater o pseudonimie Shield czternaście
miesięcy wcześniej niż Kapitan Ameryka. Shield nosił na sobie pancerny
kostium z kuloodporną tarczą na piersi (zwraca się uwagę na uderzające
podobieństwo do wczesnej tarczy Kapitana Ameryki, która w późniejszych
wydaniach była już okrągła). Na dodatek obie postacie łączyła podobna
geneza: Shield tak jak bohater Marvela zyskał nadludzkie moce po zażyciu
serum superżołnierza, którego twórca został zabity przez hitlerowców.
Postać Kapitana Ameryki pojawiała się w wielu różnych adaptacjach."
> FuzzyText.new(description, hero).matches
=> ["Kapitanie z Ameryki", "Kapitan Ameryka", "Kapitana Ameryki",
"Kapitana", "Kapitanie", "Kapitan"]

20. Thank you for your attention
Questions?