Download to read offline
Uploaded byBasis Technology
Understanding Names with Neural Networks - May 2020
The document discusses name matching techniques using neural networks. It describes how earlier techniques like Hidden Markov Models (HMMs) had limitations in capturing context around character sequences in names. The researchers at Basis Technology developed a sequence-to-sequence model using long short-term memory (LSTM) neural networks to transliterate names between languages. While more accurate, the LSTM model was slower than HMMs. To address this, they explored using a convolutional neural network which provided speed improvements while maintaining accuracy gains over HMMs. The researchers concluded that name matching remains an open problem but data-driven neural approaches hold promise for continued advances.
More Related Content
Similar to Understanding Names with Neural Networks - May 2020
Understanding Names with Neural Networks - May 2020
- 1. 12 May 2020 KfirBar, Chief Scientist Philip Blair, Senior Research Engineer Carmel Eliav, Research Engineer Understanding Names with Neural Networks
- 2. BASIS TECHNOLOGY ABOUT BASISTECHNOLOGY We engineer a safer & more productive world by building proven AI solutions for analyzing text, connecting data silos, & discovering digital evidence. EVERY DAY WE ENABLE 140MWATCHLIST CHECKS EVERY MONTH WE SUPPLY TOOLS TO 40,000DIGITAL INVESTIGATORS WE HAVE 25+YEARS EXPERIENCE
- 3.
- 4.
- 5. BASIS TECHNOLOGY OUR SPEAKERS KfirBar Philip Blair Carmel Eliav
- 6. BASIS TECHNOLOGY Names areextremely complex!
- 7. BASIS TECHNOLOGY Names area Challenge ● 4.4M unique names in the world (according to Alexa) ● 150,000 unique names in the US (according to some study on the US census) ● Names have a ton of variety Names are vitally important data points in financial compliance, anti-fraud, government intelligence, law enforcement, and identity verification.
- 8. BASIS TECHNOLOGY Task: NameMatching
- 9. BASIS TECHNOLOGY Names: BroadRange of Match “Phenomena”
- 10. BASIS TECHNOLOGY Searching Namesin Watch Lists Rosette Name Indexer
- 11. BASIS TECHNOLOGY Searching Namesin Watch Lists ● Two Pass Algorithm to find the best match: ○ First pass: Quickly generates a tractable set of candidate names for the second pass to consider ○ Second pass: compares name returned by the first pass against the query and returns a score between 0 and 1 inclusive. ● Supports matching for: ○ Names (Person, Organization, Location, etc) ○ Dates ○ Addresses 18 languages
- 12. BASIS TECHNOLOGY Name MatchingAlgorithms ● Edit Distance (Levenshtein distance) ● Rule Based Matching ● Phonetic Matching ○ Statistical Model - HMM ○ DNN Model - Sequence to Sequence encoder-decoder LSTM model ● Semantic Matching (Text Embeddings)
- 13. BASIS TECHNOLOGY Name MatchingAlgorithms ● Edit Distance (Levenshtein distance) ● Rule Based Matching ● Phonetic Matching ○ Statistical Model - HMM ○ DNN Model - Sequence to Sequence encoder-decoder LSTM model ● Semantic Matching (Text Embeddings)
- 14. BASIS TECHNOLOGY HMM-Based NameMatching
- 15. BASIS TECHNOLOGY Step One:Modeling Sequences of Characters J o h n e a r t
- 16. BASIS TECHNOLOGY Step One:Modeling Sequences of Characters J o h n
- 17. BASIS TECHNOLOGY Step Two:Modeling Transliterations ジ オ ョ ホ ン J o h n
- 18. BASIS TECHNOLOGY Step Two:Modeling Transliterations ジ オ ョ ホ ン J o h n Given a sequence of characters in the source language... ...what is the probability of the corresponding sequence of characters in the target language? This probability is our score!
- 19. BASIS TECHNOLOGY Issues withHMM-Based Name Matching ジ オ ョ J o English Character(s) Japanese Equivalent o オ yo ヨ ji ジ jo ジョ ...but this represents just "o", not "o following a 'j'"
- 20. BASIS TECHNOLOGY Issues withHMM-Based Name Matching ジ オ ョ J o ...but this represents just "o", not "o following a 'j'" Problems with HMMs: ● Multi-character equivalents ● Morphological effects on pronunciation ○ Arabic ○ Similar: "photograph" vs "photography" Common Thread: Missing Context!
- 21. BASIS TECHNOLOGY Let's trysomething different...
- 22. BASIS TECHNOLOGY How wouldyou transliterate a name?
- 23. BASIS TECHNOLOGY How WouldYou Transliterate a Name? John Titor ジョ ン ・ タイ ター
- 24. BASIS TECHNOLOGY What doesan HMM Actually Do? ジ オ ョ ホ ン J o h n For every input character... ...associate 0-1 output characters. Sequence Tagging
- 25. BASIS TECHNOLOGY How WouldYou Transliterate a Name? John Titor ジョ ン ・ タイ ター
- 26. BASIS TECHNOLOGY What technologycan we use to accomplish this?
- 27. BASIS TECHNOLOGY The Antidote:Sequence-to-Sequence (seq2seq) Source: https://d2l.ai/chapter_recurrent-modern/seq2seq.html
- 28. BASIS TECHNOLOGY Neural Networkof Choice: Long Short-Term Memory (LSTM) Cells Source: http://dprogrammer.org/rnn-lstm-gru
- 29. BASIS TECHNOLOGY Classification paradigmsin NLP T u p a c Classifier Spanish/American... T u p a c Tagger T u p a c Seq2seq
- 30. BASIS TECHNOLOGY Long Short-TermMemory (LSTM) Cells: Encoding Piece of input sequence (e.g. a character in a name) Understanding of the sequence so far
- 31. BASIS TECHNOLOGY Long Short-TermMemory (LSTM) Cells: Decoding Previously outputted character (e.g. a character in a name) Understanding of the remaining sequence Output Character
- 32. BASIS TECHNOLOGY Putting itall together
- 33. BASIS TECHNOLOGY Step One:Learning to Transliterate with seq2seq "Tupac" English Name Reader Japanese Name Generator "トゥパック"
- 34. BASIS TECHNOLOGY Step One:Learning to Transliterate with seq2seq T u p a c ト ゥ ー パ ッ ク First we "read" the English name... ...then we generate the translation
- 35. BASIS TECHNOLOGY Step Two:Running the Transliterator in Reverse to Score "Tupac" English Name Reader Japanese Name Generator "トゥパック" 0.790
- 36. BASIS TECHNOLOGY Step Two:Running the Transliterator in Reverse to Score T u p a c ト ゥ ー パ ッ ク First we "read" the English name... ...then we pass in the Japanese name... 0.790 ...to produce a score.
- 37. BASIS TECHNOLOGY How CanWe Produce a Score? ... ド ト プ カ ヨ ガ ゥ デ Our model's prediction All possible predictions
- 38. BASIS TECHNOLOGY How CanWe Produce a Score? ... ド ト プ カ ヨ ガ ゥ デ ト ゥ ... Perplexity: "How surprising is this name?" Higher 1/Perplexity = Better Match
- 39. 75% Error Reductionon Customer Data 10% Accuracy Boost for English/Katakana (Japanese) Name Pairs!
- 40.
- 41.
- 42. BASIS TECHNOLOGY Processing Timeon Name Pairs (seconds) # Pairs HMM LSTM (Neural) Slowdown 1,000 0.766 7.17 9.36x 10,000 3.591 64.3 17.91x 100,000 26.267 653.3 24.87x 500,000 135.17 3,569.6 26.41x
- 43. BASIS TECHNOLOGY All hopeis not lost...
- 44. BASIS TECHNOLOGY Faster seq2seqwith a Convolutional Neural Network (CNN) T u p a c ト ゥ ー パ ッ ク Attention Context vector <p> <p> T u p a c Embeddings Convolutions Gated Linear Units
- 45. BASIS TECHNOLOGY Convolutional NeuralNet (CNN) ● A CNN is capable of learning local features, which can replicated across the input ● CNN is popular in computer vision. It is typically structured in multiple layers and it is very good at detecting image features like lines and curves and then combine them into objects and faces https://www.aspexit.com/en/neural-network-lets-try-to-demystify-all-this-a-little-bit-3-application-to-images/
- 46. BASIS TECHNOLOGY CNN inNatural Language Processing https://arxiv.org/abs/1408.5882
- 47. Why CNN? ● Faster- CNN’s are able to process all characters in the input in parallel ● Context - While processing each character individually, we lose the context information Why didn’t we use it before? The compromise ● Positional encoding - A standard embedding layer where the input is not the character itself but the position of the character within the token
- 48.
- 49. BASIS TECHNOLOGY Results Experiment Accuracyimprovement Speed Lstm 14% X25 slower Cnn version 1 8.8% X1.6 slower Cnn version 2 9% X1.6 slower ● The speed was measured in seconds for 1,000 name pairs ● The improvement is relative to the HMM model
- 50. BASIS TECHNOLOGY What DoesThis Tell Us? ● Name matching is an open problem ○ Existing technology can always be improved ● Accuracy comes at a cost ○ Machine learning holds the answers, but not always in the way we expect ● Looking Forward: ○ Embracing data-driven approaches to NLP ○ Fully cross-lingual name matching?
- 51.
- 52.
- 54. Thank you kfir@basistech.com Kfir Bar pblair@basistech.com PhilipBlair carmel@basistech.com Carmel Eliav