Download as PDF, PPTX
Deep learning for NLP
- 1. Deep Learning for NaturalLanguage Processing Sihem Romdhani
- 2. Agenda What is deeplearning? Why is this important? How to apply deep learning to natural language processing?
- 3. About Me Sihem Romdhani Softwareengineer at Veeva MEng in Computer Engineering MSc in ECE at University of Waterloo - Deep learning - Speech/image recognition
- 4. What is machinelearning? Training Text DOCUMENTS, IMAGES, SOUNDS, ... Labels Features Vectors Machine Learning Algorithm
- 5. What is machinelearning? Training Text DOCUMENTS, IMAGES, SOUNDS, ... Labels Features Vectors Features Vector New Text DOCUMENTS, IMAGES, SOUNDS, ... Predictive Model Machine Learning Algorithm Expected Label
- 6. What is deeplearning? image (pixel) edges object parts object models Input Layer Output Layer SARA layers of abstraction DEEP NEURAL NETWORK (DNN)
- 7. Speech technology DeepMind beats thebest in GO Nvidia to drive car with Drive PX 2 super computer Apple’s Siri Amazon’s Alexa
- 8.
- 9. Which is human?Which is machine? I’m in a love affair I can’t share it ain’t fair Haha I’m just playin’ ladies you know I love you. I know my love is true and I know you love me too Girl I’m down for whatever cause my love is true SONG A For a chance at romance I would love to enhance But everything I love has turned to a tedious task One day we gonna have to leave our love in the past I love my fans but no one ever puts a grasp SONG B
- 10. DeepBeat.org Online tool forrap lyrics generator
- 11. Natural Language Processing(NLP) understanding and generating human language
- 12. Human language is difficult! variability kittyvs. cat ambiguity blue jays vs. Blue Jays vague and latent “Eating babies can be messy.”
- 13. How can werepresent words? AUDIO SPECTROGRAM IMAGE PIXELS VECTOR OF REAL NUMBERS Audio Image Word
- 14. ‘CAT’ 0 0.5 00.2 0.1 0 0.1 0 0.3 0 0.4 0.1 ‘KITTY’ Similarity? Embedding Embedding
- 15. The cat/kitty purrs Context defines similarity Infer the meaning of words from the company they keep The cat/kitty hunts mice
- 16. Word to Vector Thelittle cat hunts mice Embedding (‘cat’) Context Window Predict
- 17. Word2Vec (Google) forlearning embeddings 1. Collect large corpus of text 2. Randomly initialize embeddings Each word is mapped into a fixed-size embedding with random values 3. Set up prediction task For each word, pick a random word from the context and consider it as the target (label) INPUT WORD → TARGET WORD ‘Cat’ → ‘Hunt’ 4. Learn the embeddings Train a model (logistic classifier) to predict the target (neighbours of each word) Update the embedding
- 18.
- 19. ‘cat’ Embedding Predicted target PREDICTION Linear model wVcat+ b Word2Vec Process
- 20. 0 0 0 0 0 1 0 0 0 0 ‘cat’ Embedding Predicted target Real target ‘hunt’ PREDICTIONCOMPARE Linear model wVcat + b Volcabulary Word2Vec Process
- 21. Similar words arenearby Word Embedding ‘cat’ ‘tiger’ ‘home’ ‘kitten’ ‘dog’
- 22. Let’s discuss areal problem.
- 23. Pharma Company’s Market Expansion Newproduct: pancreatic cancer treatment! Who are the oncologists to inform about it?
- 24. Web scraping DIFFERENT DATA-EXTRACTIONSCRIPTS FOR DIFFERENT WEB PAGES
- 25. Consolidating into onerecord
- 26. Extracting addresses from webpages abbreviations Avenue- Ave - Ave. ambiguity in names main vs. Main Street missing elements mispellings misorder
- 27.
- 28. Solution architecture WEB PAGE (HTML) TOKENIZATION WORD REPRESENTATION/ WORDEMBEDDING DEEP NEURAL NETWORK CLASSIFIER POST PROCESSING OUTPUT EXTRACTED ADDRESS
- 29. Solution architecture WEB PAGE (HTML) TOKENIZATION WORD REPRESENTATION/ WORDEMBEDDING DEEP NEURAL NETWORK CLASSIFIER POST PROCESSING OUTPUT EXTRACTED ADDRESS
- 30. Solution architecture WEB PAGE (HTML) TOKENIZATION WORD REPRESENTATION/ WORDEMBEDDING DEEP NEURAL NETWORK CLASSIFIER POST PROCESSING OUTPUT EXTRACTED ADDRESS
- 31. Solution architecture WEB PAGE (HTML) TOKENIZATION WORD REPRESENTATION/ WORDEMBEDDING DEEP NEURAL NETWORK CLASSIFIER POST PROCESSING OUTPUT EXTRACTED ADDRESS
- 32. Solution architecture WEB PAGE (HTML) TOKENIZATION WORD REPRESENTATION/ WORDEMBEDDING DEEP NEURAL NETWORK CLASSIFIER POST PROCESSING OUTPUT EXTRACTED ADDRESS
- 33. TensorFlow for wordembedding
- 34. TensorFlow for wordembedding
- 35. TensorFlow for wordembedding
- 36. TensorFlow for wordembedding
- 40. Training a DNNclassifier 123 Main Street, Florida 32209 123 Street Florida 32209Main Context Words AO A: Address O: Other (non-address) Hidden Layer 3 Hidden Layer 2 Hidden Layer 1
- 41. Address extraction algorithm 1.Load the trained DNN model 2. for each new web page do 3. The web page is pre-processed to remove the HTML tags, then tokenized 4. Each token (word) is replaced by its embedding vector 5. Create the DNN input by concatenating the embedding of the context 6. The DNN labels the central word of the context as A (ADDRESS) or O (OTHER) 7.Extract the sequence of tokens with A label 8.if the total number of tokens is within range then 9. Output the token block as extracted address 10.End if 11.End For Input: Trained DNN model and unlabeled web page Output: Extracted address
- 42. Applications of NLP Informationextraction 文 A Social media analytics Question & Answer Natural language generation Speech recognition
- 43.
- 44.
- 45. Slides available onSlack channel #veeva Stop by our booth for questions and cool swag!