Deep Learning for Domain-Specific Entity Extraction from Unstructured Text with Zoran Dzunic and Mohamed AbdelHady
![[0.3, 0.2, 0.9 …] [0.8, 0.8, 0.1 …][0.5, 0.1, 0.5 …]Embedding:
Features: Word Embeddings
12#DL1SAIS
B-Chemical O O
dim small (e.g., 50, 200)](https://image.slidesharecdn.com/1zorandzunicmohamedabdelhady-180612205503/75/deep-learning-for-domainspecific-entity-extraction-from-unstructured-text-with-zoran-dzunic-and-mohamed-abdelhady-12-2048.jpg?cb=1693340271)
Deep Learning for Domain-Specific Entity Extraction from Unstructured Text with Zoran Dzunic and Mohamed AbdelHady
Entity extraction, also known as named-entity recognition (NER), entity chunking and entity identification, is a subtask of information extraction with the goal of detecting and classifying phrases in a text into predefined categories. While finding entities in an automated way is useful on its own, it often serves as a preprocessing step for more complex tasks, such as relationship extraction. For example, biomedical entity extraction is a critical step for understanding the interactions between different entity types, such as the drug-disease relationship or the gene-protein relationship. Feature generation for such tasks is often complex and time consuming. However, neural networks can obviate the need for feature engineering and use original data as input. We will demonstrate how to build a domain-specific entity extraction system from unstructured text using deep learning. In the model, domain-specific word embedding vectors are trained with word2vec learning algorithm on a Spark cluster using millions of Medline PubMed abstracts and then used as features to train an LSTM recurrent neural network for entity extraction, using Keras with TensorFlow or CNTK on a GPU-enabled Azure Data Science Virtual Machine (DSVM). Results show that training a domain-specific word embedding model boosts performance when compared to embeddings trained on generic data such as Google News. While we use biomedical data as an example, the pipeline is generic and can be applied to other domains.
More Related Content
What's hot
What's hot(20)
Similar to Deep Learning for Domain-Specific Entity Extraction from Unstructured Text with Zoran Dzunic and Mohamed AbdelHady
Similar to Deep Learning for Domain-Specific Entity Extraction from Unstructured Text with Zoran Dzunic and Mohamed AbdelHady(20)
More from Databricks
More from Databricks(20)
Recently uploaded
Recently uploaded(20)
![[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation](https://cdn.slidesharecdn.com/ss_thumbnails/spelapoklukarteabrasanacprez-231129000106-d6582d1b-thumbnail.jpg?width=768&fit=bounds)
![[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/dsceurope23zsoltfeleki-machinetranslationshouldwetrustit-231128230607-05e0d0cd-thumbnail.jpg?width=768&fit=bounds)
Deep Learning for Domain-Specific Entity Extraction from Unstructured Text with Zoran Dzunic and Mohamed AbdelHady
- 1. Mohamed AbdelHady, Microsoft AI Platform Zoran Dzunic, Microsoft AI Platform Deep Learning for Domain- Specific Entity Extraction from Unstructured Text #DL1SAIS
- 2. Goals • What is entity extraction? • When to train a custom entity extraction model? • What are word embeddings? • How to train a custom word embedding model on a Spark cluster? • How to train a custom Deep Neural Network for entity extraction? 2#DL1SAIS
- 3. Entity Extraction • Subtask of information extraction • Also known as Named-entity recognition (NER), entity chunking and entity identification • Find phrases in text that refer to a real-world entity of specific types Zoran and Mohamed are at Spark+AI Summit in San Francisco. Zoran : PERSON : LOC 3#DL1SAIS
- 4. Entity Extraction • Subtask of information extraction • Also known as Named-entity recognition (NER), entity chunking and entity identification • Find phrases in text that refer to a real-world entity of specific types Zoran and Mohamed are at Spark+AI Summit in San Francisco. Zoran : PERSON Mohamed : PERSON Spark+AI Summit : ORG San Francisco : LOC 4#DL1SAIS
- 5. Biomedical Entity Extraction • Entity types drug/chemical, disease, protein, DNA, etc. • Critical step for complex biomedical NLP tasks: – Extraction of diseases, symptoms from electronic medical or health records – Understanding the interactions between different entity types such as drug- drug interaction, drug-disease relationship and gene-protein relationship, e.g., • Drug A cures Disease B. • Drug A causes Disease B. Similar for other domains (e.g., legal, finance) 5#DL1SAIS
- 8. Approach 1. Feature Extraction Phase – Domain Specific Features Use a large amount of unlabeled domain-specific data corpus such as Medline PubMed abstracts to train a neural word embedding model. 2. Model Training Phase – Domain Specific Model The output embeddings are considered as automatically generated features to train a neural entity extractor using a small/reasonable amount of labeled data. 8#DL1SAIS
- 9. Word Embedding a semantic continuous representation of words 9#DL1SAIS
- 10. #DL1SAIS 10
- 11. Input: Words 11#DL1SAIS Naloxon e reverses the B-Chemical O O Words:
- 12. [0.3, 0.2, 0.9 …] [0.8, 0.8, 0.1 …][0.5, 0.1, 0.5 …]Embedding: Features: Word Embeddings 12#DL1SAIS B-Chemical O O dim small (e.g., 50, 200)
- 14. Custom Word Embeddings • Publicly available pre-trained models such as Google News • Can we do better on a specific domain? • We trained a word embedding model for biomedical domain on 27 million Pubmed abstracts (22GB) • Azure HDInsight Spark Cluster, 11 worker nodes • Spark MLlib Word2Vec • Trained in ~30min 14#DL1SAIS
- 16. #DL1SAIS 16
- 18. DNN Architecture 18#DL1SAIS • Keras with TensorFlow • GPU enabled Azure Data Science VM (DSVM) NC6 Standard (56 GB, K80 NVIDIA Tesla) or Deep Learning VM (DLVM) • Parameters – # recurrent units = 150 – droput rate = 0.2
- 20. Datasets • Proteins, Cell Line, Cell Type, DNA and RNA Detection Bio-Entity Recognition Task at BioNLP/NLPBA 2004 - http://www.nactem.ac.uk/tsujii/GENIA/ERtask/report.html • Chemicals and Diseases Detection BioCreative V CDR task corpus - http://www.biocreative.org/tasks/biocreative-v/track-3-cdr/ • Drugs Detection Semeval 2013 - Task 9.1 (Drug Recognition) - https://www.cs.york.ac.uk/semeval-2013/task9/ 20#DL1SAIS
- 21. Dataset Description http://www.biocreative.org/tasks/biocreative-v/track-3-cdr/ 21#DL1SAIS • - - -
- 22. Experimental Setup • Azure ML Python Package for Text Analytics. https://docs.microsoft.com/en-us/python/api/overview/azure-machine- learning/textanalytics https://aka.ms/aml-packages/text/download 22#DL1SAIS
- 23. CRFSuite: • Extract traditional features • Train CRF model Conditional Random Fields (CRF) #DL1SAIS 23
- 24. Results (exact match) Algorithm + Features Recall Precision F-score Dictionary Lookup 64% 74% 68% CRF: Traditional Features 61% 81% 70% CRF: Pubmed Embedding 40% 61% 48% CRF: Traditional + Pubmed Embed. 65% 80% 71% LSTM: Pubmed Embedding 76% 77% 76% LSTM: Generic Embeddings 74% 63% 67% #DL1SAIS 24
- 27. Takeaways • Recipe for building a custom entity extraction pipeline: – Get a large amount of in-domain unlabeled data – Train a word2vec model on unlabeled data on Spark – Get as much of labeled data as possible – Train an LSTM -based Neural Network on a GPU-enabled machine • Word embeddings are powerful features – Convey word semantics – Perform better than traditional features – No feature engineering • LSTM NN is more powerful model than traditional CRF #DL1SAIS 27