This document provides an overview of a course on trends and research applications in natural language processing (NLP). It begins with introducing the goals of the course, which are to understand interesting NLP tasks and novel projects through a research-oriented webinar. The document then covers various NLP topics like question answering, machine translation, sentiment analysis, natural language generation applications, and challenges in NLP like grounded language and embodied language. It also provides tips for aspiring NLP researchers.
An on-going project on Natural Language Processing (using Python and the NLTK toolkit), which focuses on the extraction of sentiment from a Question and its title on www.stackoverflow.com and determining the polarity.Based on the above findings, it is verified whether the rules and guidelines imposed by the SO community on the users are strictly followed or not.
Nautral Langauge Processing - Basics / Non Technical Dhruv Gohil
This document provides an overview of natural language processing (NLP) and discusses several NLP applications. It introduces NLP and how it helps computers understand human language through examples like Apple's Siri and Google Now. It then summarizes popular NLP toolkits and describes applications including text summarization, information extraction, sentiment analysis, and dialog systems. The document concludes by discussing NLP system development, testing, and evaluation.
This document summarizes a conference paper published at ICLR 2020 that proposes a method called Plug and Play Language Models (PPLM) for controlled text generation using pretrained language models. PPLM allows controlling attributes of generated text like topic or sentiment without retraining the language model by combining it with simple attribute classifiers that guide the text generation process. The paper presents PPLM as a simple alternative to retraining language models that is more efficient and practical for controlled text generation.
IRJET- Survey on Deep Learning Approaches for Phrase Structure Identification...IRJET Journal
This document discusses deep learning approaches for identifying phrase structures in sentences. It begins with an introduction to natural language processing and phrase structure grammar. Traditional n-gram and rule-based approaches to phrase structure identification are described. Recent deep learning methods for natural language tasks that have been applied to phrase structure identification are then summarized, including word embeddings, convolutional neural networks, recurrent neural networks and recursive neural networks. The document concludes that deep learning requires less manual feature engineering and has achieved good performance on many NLP tasks, but still has room for improvement, especially on tasks involving unlabeled data.
This document provides an overview of natural language processing (NLP) research trends presented at ACL 2020, including shifting away from large labeled datasets towards unsupervised and data augmentation techniques. It discusses the resurgence of retrieval models combined with language models, the focus on explainable NLP models, and reflections on current achievements and limitations in the field. Key papers on BERT and XLNet are summarized, outlining their main ideas and achievements in advancing the state-of-the-art on various NLP tasks.
1. BlenderBot summarizes several papers on chatbot models from Google, OpenAI, and FAIR to provide context on its contributions.
2. It describes its use of large pre-training datasets like Reddit comments, and fine-tuning on datasets for personality, empathy, knowledge, and blended skills.
3. The paper considers retrieval, generative, and retrieve-and-refine models, selecting the Poly-Encoder for retrieval and BART for generation due to their advantages, and exploring techniques like unlikelihood training and decoding strategies.
XAI LANGUAGE TUTOR - A XAI-BASED LANGUAGE LEARNING CHATBOT USING ONTOLOGY AND...ijnlc
In this paper, we proposed a XAI-based Language Learning Chatbot (namely XAI Language Tutor) by using ontology and transfer learning techniques. To facilitate three levels of language learning, XAI Language Tutor consists of three levels for systematically English learning, which includes: 1) phonetics level for speech recognition and pronunciation correction; 2) semantic level for specific domain conversation, and 3) simulation of “free-style conversation” in English - the highest level of language chatbot communication as “free-style conversation agent”. In terms of academic contribution, we implement the ontology graph to explain the performance of free-style conversation, following the concept of XAI (Explainable Artificial Intelligence) to visualize the connections of neural network in bionics, and explain the output sentence from language model. From implementation perspective, our XAI Language Tutor agent integrated the mini-program in WeChat as front-end, and fine-tuned GPT-2 model of transfer learning as back-end to interpret the responses by ontology graph.
All of our source codes have uploaded to GitHub: https://github.com/p930203110/EnglishLanguageRobot
This document provides an overview of a course on trends and research applications in natural language processing (NLP). It begins with introducing the goals of the course, which are to understand interesting NLP tasks and novel projects through a research-oriented webinar. The document then covers various NLP topics like question answering, machine translation, sentiment analysis, natural language generation applications, and challenges in NLP like grounded language and embodied language. It also provides tips for aspiring NLP researchers.
An on-going project on Natural Language Processing (using Python and the NLTK toolkit), which focuses on the extraction of sentiment from a Question and its title on www.stackoverflow.com and determining the polarity.Based on the above findings, it is verified whether the rules and guidelines imposed by the SO community on the users are strictly followed or not.
Nautral Langauge Processing - Basics / Non Technical Dhruv Gohil
This document provides an overview of natural language processing (NLP) and discusses several NLP applications. It introduces NLP and how it helps computers understand human language through examples like Apple's Siri and Google Now. It then summarizes popular NLP toolkits and describes applications including text summarization, information extraction, sentiment analysis, and dialog systems. The document concludes by discussing NLP system development, testing, and evaluation.
This document summarizes a conference paper published at ICLR 2020 that proposes a method called Plug and Play Language Models (PPLM) for controlled text generation using pretrained language models. PPLM allows controlling attributes of generated text like topic or sentiment without retraining the language model by combining it with simple attribute classifiers that guide the text generation process. The paper presents PPLM as a simple alternative to retraining language models that is more efficient and practical for controlled text generation.
IRJET- Survey on Deep Learning Approaches for Phrase Structure Identification...IRJET Journal
This document discusses deep learning approaches for identifying phrase structures in sentences. It begins with an introduction to natural language processing and phrase structure grammar. Traditional n-gram and rule-based approaches to phrase structure identification are described. Recent deep learning methods for natural language tasks that have been applied to phrase structure identification are then summarized, including word embeddings, convolutional neural networks, recurrent neural networks and recursive neural networks. The document concludes that deep learning requires less manual feature engineering and has achieved good performance on many NLP tasks, but still has room for improvement, especially on tasks involving unlabeled data.
This document provides an overview of natural language processing (NLP) research trends presented at ACL 2020, including shifting away from large labeled datasets towards unsupervised and data augmentation techniques. It discusses the resurgence of retrieval models combined with language models, the focus on explainable NLP models, and reflections on current achievements and limitations in the field. Key papers on BERT and XLNet are summarized, outlining their main ideas and achievements in advancing the state-of-the-art on various NLP tasks.
1. BlenderBot summarizes several papers on chatbot models from Google, OpenAI, and FAIR to provide context on its contributions.
2. It describes its use of large pre-training datasets like Reddit comments, and fine-tuning on datasets for personality, empathy, knowledge, and blended skills.
3. The paper considers retrieval, generative, and retrieve-and-refine models, selecting the Poly-Encoder for retrieval and BART for generation due to their advantages, and exploring techniques like unlikelihood training and decoding strategies.
XAI LANGUAGE TUTOR - A XAI-BASED LANGUAGE LEARNING CHATBOT USING ONTOLOGY AND...ijnlc
In this paper, we proposed a XAI-based Language Learning Chatbot (namely XAI Language Tutor) by using ontology and transfer learning techniques. To facilitate three levels of language learning, XAI Language Tutor consists of three levels for systematically English learning, which includes: 1) phonetics level for speech recognition and pronunciation correction; 2) semantic level for specific domain conversation, and 3) simulation of “free-style conversation” in English - the highest level of language chatbot communication as “free-style conversation agent”. In terms of academic contribution, we implement the ontology graph to explain the performance of free-style conversation, following the concept of XAI (Explainable Artificial Intelligence) to visualize the connections of neural network in bionics, and explain the output sentence from language model. From implementation perspective, our XAI Language Tutor agent integrated the mini-program in WeChat as front-end, and fine-tuned GPT-2 model of transfer learning as back-end to interpret the responses by ontology graph.
All of our source codes have uploaded to GitHub: https://github.com/p930203110/EnglishLanguageRobot
[Paper Reading] Supervised Learning of Universal Sentence Representations fro...Hiroki Shimanaka
This document summarizes the paper "Supervised Learning of Universal Sentence Representations from Natural Language Inference Data". It discusses how the researchers trained sentence embeddings using supervised data from the Stanford Natural Language Inference dataset. They tested several sentence encoder architectures and found that a BiLSTM network with max pooling produced the best performing universal sentence representations, outperforming prior unsupervised methods on 12 transfer tasks. The sentence representations learned from the natural language inference data consistently achieved state-of-the-art performance across multiple downstream tasks.
A prior case study of natural language processing on different domain IJECEIAES
This document summarizes a prior case study on natural language processing across different domains. It begins with an introduction to natural language processing, describing how it is a branch of artificial intelligence that allows computers to understand human language. It then reviews several existing studies that applied natural language processing techniques such as named entity recognition and text mining to tasks like identifying technical knowledge in resumes, enhancing reading skills for deaf students, and predicting student performance. The document concludes by highlighting some of the challenges in developing new natural language processing models.
Grammarly AI-NLP Club #2 - Recent advances in applied chatbot technology - Jo...Grammarly
Speaker: Jordi Carrera Ventura, Artificial Intelligence technologist at Telefónica R&D
Summary: Chatbots (aka conversational agents, spoken dialogue systems) allow users to interface with computers using natural language by simply asking questions or issuing commands.
Given a query, the chatbot builds a semantic representation of the input, transforms it into a logical statement, and performs all the necessary actions to fulfill the user's intent. Sometimes this simply means calculating an exact answer or retrieving a fact from a database, whereas other times it means building a contextual model and running a full-fledged conversation flow while keeping track of anaphoras and cross-references.
Besides the direct applications of chatbots in IoT (Amazon’s Alexa, Apple's Siri) and IT (the historical field of Information Retrieval as a whole can be seen as a sub-problem of spoken dialogue systems), chatbots' main appeal for technologists is their location at the intersection of all major Natural Language Processing technologies and many of the deepest questions in Cognitive Science today: semantic parsing, entity recognition, knowledge representation, and coreference resolution.
In this talk, I will explore those questions in the context of an applied industry setting, and I will introduce a framework suitable for addressing them, together with an overview of the state-of-the-art in chatbot technology and some original techniques.
The document discusses two neural network models for reading comprehension tasks: the Attentive Reader model proposed by Herman et al. in 2015 and the Stanford Reader model proposed by Chen et al. in 2016. The author implemented a two-layer attention model inspired by these previous models that achieves a 1.5% higher accuracy on reading comprehension tasks compared to the Stanford Reader.
1.0 Introduction
1.1 Objectives
1.2 Some Simple Definition of A.I.
1.3 Definition by Eliane Rich
1.4 Definition by Buchanin and Shortliffe
1.5 Another Definition by Elaine Rich
1.6 Definition by Barr and Feigenbaum
1.7 Definition by Shalkoff
1.8 Summary
1.9 Further Readings/References
Conversational AI Agents have become mainstream today due to significant advancements in the methods required to build accurate models, such as machine learning and deep learning, and, secondly, because they are seen as a natural fit in a wide range of domains, such as healthcare, e-commerce, customer service, tourism, and education, that rely heavily on natural language conversations in day-to-day operations. This rapid increase in demand has been matched by an equally rapid rate of research and development, with new products being introduced on a daily basis.
Learn More:https://bit.ly/3tBkT81
Contact Us:
Website: https://www.phdassistance.com/
UK: +44 7537144372
India No:+91-9176966446
Email: info@phdassistance.com
Artificial intelligence and knowledge representationSajan Sahu
The document discusses artificial intelligence and knowledge representation. It describes how computers can be made intelligent through speed of computation, filtering responses, using algorithms and neural networks. It also discusses knowledge representation techniques in AI like propositional logic, semantic networks, frames, predicate logic and nonmonotonic reasoning. The document provides examples and applications of AI like pattern recognition, robotics and natural language processing. It also discusses some fundamental problems of AI.
This document discusses definitions and concepts related to artificial intelligence (AI). It provides several definitions of AI from different experts that describe AI as studying how to make computers behave intelligently like humans, studying symbolic and non-algorithmic problem solving, and studying how to solve exponentially hard problems efficiently. The document also discusses key differences between conventional and intelligent computing, applications of AI, and proposes the Turing Test for evaluating machine intelligence.
EXTENDING OUTPUT ATTENTIONS IN RECURRENT NEURAL NETWORKS FOR DIALOG GENERATIONijaia
In natural language processing, attention mechanism in neural networks are widely utilized. In this paper, the research team explore a new mechanism of extending output attention in recurrent neural networks for dialog systems. The new attention method was compared with the current method in generating dialog sentence using a real dataset. Our architecture exhibits several attractive properties such as better handle long sequences and, it could generate more reasonable replies in many cases.
COLING 2014: Joint Opinion Relation Detection Using One-Class Deep Neural Net...Peinan ZHANG
The document proposes a novel joint opinion relation detection method using a one-class deep neural network (OCDNN). It addresses the problems of prior methods by simultaneously considering opinion words, targets, and their linking relations. The OCDNN consists of two levels: the lower level learns features using word embeddings and a recursive autoencoder; the higher level performs one-class classification. Experiments on customer review datasets show the proposed method outperforms baselines by up to 9% F-measure by verifying all three required conditions of opinion relations.
French machine reading for question answeringAli Kabbadj
This paper proposes to unlock the main barrier to machine reading and comprehension French natural language texts. This open the way to machine to find to a question a precise answer buried in the mass of unstructured French texts. Or to create a universal French chatbot. Deep learning has produced extremely promising results for various tasks in natural language understanding particularly topic classification, sentiment analysis, question answering, and language translation. But to be effective Deep Learning methods need very large training da-tasets. Until now these technics cannot be actually used for French texts Question Answering (Q&A) applications since there was not a large Q&A training dataset. We produced a large (100 000+) French training Dataset for Q&A by translating and adapting the English SQuAD v1.1 Dataset, a GloVe French word and character embed-ding vectors from Wikipedia French Dump. We trained and evaluated of three different Q&A neural network ar-chitectures in French and carried out a French Q&A models with F1 score around 70%.
THE EFFECTS OF THE LDA TOPIC MODEL ON SENTIMENT CLASSIFICATIONijscai
Online reviews are a feedback to the product and play a key role in improving the product to cater to consumers. Online reviews that rely heavily on manual categorization are time consuming and labor intensive.The recurrent neural network in deep learning can process time series data, while the long and short term memory network can process long time sequence data well. This has good experimental verification support in natural language processing, machine translation, speech recognition and language model.The merits of the extracted data features affect the classification results produced by the classification model. The LDA topic model adds a priori a posteriori knowledge to classify the data so that the characteristics of the data can be extracted efficiently.Applied to the classifier can improve accuracy and efficiency. Two-way long-term and short-term memory networks are variants and extensions of cyclic neural networks.The deep learning framework Keras uses Tensorflow as the backend to build a convenient two-way long-term and short-term memory network model, which provides a strong technical support for the experiment.Using the LDA topic model to extract the keywords needed to train the neural network and increase the internal relationship between words can improve the learning efficiency of the model. The experimental results in the same experimental environment are better than the traditional word frequency features.
This document discusses knowledge representation in artificial intelligence. It covers various techniques for knowledge representation including logical representation using propositional logic and first-order predicate logic, semantic network representation, frame representation, and production rules. It also discusses issues in knowledge representation such as representing important attributes, relationships, and granularity of knowledge. Propositional logic is introduced as the simplest form of logic where statements are represented by propositions that can be either true or false. The syntax and semantics of propositional logic are also covered.
NLP Bootcamp 2018 : Representation Learning of text for NLPAnuj Gupta
The document provides an outline for a workshop on representation learning of text for natural language processing (NLP). The workshop will be divided into 4 modules covering both foundational techniques like one-hot encoding and bag-of-words as well as state-of-the-art methods like word, sentence, and character vectors. The objective is for participants to gain a deeper understanding of the key ideas, math, and code behind text representation techniques in order to apply them to solve NLP problems and achieve higher accuracies and understanding.
This document provides an introduction to knowledge representation in artificial intelligence. It discusses how knowledge representation and reasoning forms the basis of intelligent behavior through computational means. The key types of knowledge that need to be represented are defined, including objects, events, facts, and meta-knowledge. Different types of knowledge such as declarative, procedural, structural and heuristic knowledge are explained. The importance of knowledge representation for modeling intelligent behavior in agents is highlighted. The requirements for effective knowledge representation including representational adequacy, inferential adequacy, inferential efficiency, and acquisitional efficiency are outlined. Propositional logic is introduced as the simplest form of logic using propositions.
Won Ik Cho presented on his research related to intention understanding in Korean natural language processing. He discussed developing annotation guidelines and corpora to classify Korean utterances by speech act, considering factors like intonation, context, and rhetoricalness. He proposed a method using text-based analysis combined with speech-aided disambiguation. Future work includes developing structured paraphrasing for argument extraction and an improved dialog manager.
This document provides an overview of natural language processing (NLP). It begins with examples of NLP applications like translation and question answering. It then discusses the backgrounds in artificial intelligence, linguistics, and the web. The document outlines several common NLP tasks like part-of-speech tagging, named-entity recognition, word sense disambiguation, and parsing. It also discusses challenges like ambiguity in natural language. The document concludes with a discussion of why NLP is difficult due to ambiguity at both the linguistic and acoustic levels.
Working in NLP in the Age of Large Language ModelsZachary S. Brown
The document provides an overview of recent advances in natural language processing (NLP) and large language models (LLMs). It discusses several key moments and technological developments that have contributed to progress in the field over the past decade, including the introduction of neural networks for language modeling in 2001, word embeddings in 2013, the attention mechanism in 2015, and the transformer architecture in 2017. Recent years have seen massive LLMs like GPT-3 achieve strong performance across many NLP tasks through techniques like self-supervised pre-training and scaling up model sizes. This has led to new tooling ecosystems and commercial applications of generative NLP, though discriminative tasks still rely on smaller, more efficient models when data is available.
SENTIMENT ANALYSIS IN MYANMAR LANGUAGE USING CONVOLUTIONAL LSTM NEURAL NETWORKijnlc
In recent years, there has been an increasing use of social media among people in Myanmar and writing review on social media pages about the product, movie, and trip are also popular among people. Moreover, most of the people are going to find the review pages about the product they want to buy before deciding whether they should buy it or not. Extracting and receiving useful reviews over interesting products is very important and time consuming for people. Sentiment analysis is one of the important processes for extracting useful reviews of the products. In this paper, the Convolutional LSTM neural network architecture is proposed to analyse the sentiment classification of cosmetic reviews written in Myanmar Language. The paper also intends to build the cosmetic reviews dataset for deep learning and sentiment lexicon in Myanmar Language.
Sentiment Analysis In Myanmar Language Using Convolutional Lstm Neural Networkkevig
In recent years, there has been an increasing use of social media among people in Myanmar and writing
review on social media pages about the product, movie, and trip are also popular among people. Moreover,
most of the people are going to find the review pages about the product they want to buy before deciding
whether they should buy it or not. Extracting and receiving useful reviews over interesting products is very
important and time consuming for people. Sentiment analysis is one of the important processes for extracting
useful reviews of the products. In this paper, the Convolutional LSTM neural network architecture is
proposed to analyse the sentiment classification of cosmetic reviews written in Myanmar Language. The
paper also intends to build the cosmetic reviews dataset for deep learning and sentiment lexicon in Myanmar
Language.
[Paper Reading] Supervised Learning of Universal Sentence Representations fro...Hiroki Shimanaka
This document summarizes the paper "Supervised Learning of Universal Sentence Representations from Natural Language Inference Data". It discusses how the researchers trained sentence embeddings using supervised data from the Stanford Natural Language Inference dataset. They tested several sentence encoder architectures and found that a BiLSTM network with max pooling produced the best performing universal sentence representations, outperforming prior unsupervised methods on 12 transfer tasks. The sentence representations learned from the natural language inference data consistently achieved state-of-the-art performance across multiple downstream tasks.
A prior case study of natural language processing on different domain IJECEIAES
This document summarizes a prior case study on natural language processing across different domains. It begins with an introduction to natural language processing, describing how it is a branch of artificial intelligence that allows computers to understand human language. It then reviews several existing studies that applied natural language processing techniques such as named entity recognition and text mining to tasks like identifying technical knowledge in resumes, enhancing reading skills for deaf students, and predicting student performance. The document concludes by highlighting some of the challenges in developing new natural language processing models.
Grammarly AI-NLP Club #2 - Recent advances in applied chatbot technology - Jo...Grammarly
Speaker: Jordi Carrera Ventura, Artificial Intelligence technologist at Telefónica R&D
Summary: Chatbots (aka conversational agents, spoken dialogue systems) allow users to interface with computers using natural language by simply asking questions or issuing commands.
Given a query, the chatbot builds a semantic representation of the input, transforms it into a logical statement, and performs all the necessary actions to fulfill the user's intent. Sometimes this simply means calculating an exact answer or retrieving a fact from a database, whereas other times it means building a contextual model and running a full-fledged conversation flow while keeping track of anaphoras and cross-references.
Besides the direct applications of chatbots in IoT (Amazon’s Alexa, Apple's Siri) and IT (the historical field of Information Retrieval as a whole can be seen as a sub-problem of spoken dialogue systems), chatbots' main appeal for technologists is their location at the intersection of all major Natural Language Processing technologies and many of the deepest questions in Cognitive Science today: semantic parsing, entity recognition, knowledge representation, and coreference resolution.
In this talk, I will explore those questions in the context of an applied industry setting, and I will introduce a framework suitable for addressing them, together with an overview of the state-of-the-art in chatbot technology and some original techniques.
The document discusses two neural network models for reading comprehension tasks: the Attentive Reader model proposed by Herman et al. in 2015 and the Stanford Reader model proposed by Chen et al. in 2016. The author implemented a two-layer attention model inspired by these previous models that achieves a 1.5% higher accuracy on reading comprehension tasks compared to the Stanford Reader.
1.0 Introduction
1.1 Objectives
1.2 Some Simple Definition of A.I.
1.3 Definition by Eliane Rich
1.4 Definition by Buchanin and Shortliffe
1.5 Another Definition by Elaine Rich
1.6 Definition by Barr and Feigenbaum
1.7 Definition by Shalkoff
1.8 Summary
1.9 Further Readings/References
Conversational AI Agents have become mainstream today due to significant advancements in the methods required to build accurate models, such as machine learning and deep learning, and, secondly, because they are seen as a natural fit in a wide range of domains, such as healthcare, e-commerce, customer service, tourism, and education, that rely heavily on natural language conversations in day-to-day operations. This rapid increase in demand has been matched by an equally rapid rate of research and development, with new products being introduced on a daily basis.
Learn More:https://bit.ly/3tBkT81
Contact Us:
Website: https://www.phdassistance.com/
UK: +44 7537144372
India No:+91-9176966446
Email: info@phdassistance.com
Artificial intelligence and knowledge representationSajan Sahu
The document discusses artificial intelligence and knowledge representation. It describes how computers can be made intelligent through speed of computation, filtering responses, using algorithms and neural networks. It also discusses knowledge representation techniques in AI like propositional logic, semantic networks, frames, predicate logic and nonmonotonic reasoning. The document provides examples and applications of AI like pattern recognition, robotics and natural language processing. It also discusses some fundamental problems of AI.
This document discusses definitions and concepts related to artificial intelligence (AI). It provides several definitions of AI from different experts that describe AI as studying how to make computers behave intelligently like humans, studying symbolic and non-algorithmic problem solving, and studying how to solve exponentially hard problems efficiently. The document also discusses key differences between conventional and intelligent computing, applications of AI, and proposes the Turing Test for evaluating machine intelligence.
EXTENDING OUTPUT ATTENTIONS IN RECURRENT NEURAL NETWORKS FOR DIALOG GENERATIONijaia
In natural language processing, attention mechanism in neural networks are widely utilized. In this paper, the research team explore a new mechanism of extending output attention in recurrent neural networks for dialog systems. The new attention method was compared with the current method in generating dialog sentence using a real dataset. Our architecture exhibits several attractive properties such as better handle long sequences and, it could generate more reasonable replies in many cases.
COLING 2014: Joint Opinion Relation Detection Using One-Class Deep Neural Net...Peinan ZHANG
The document proposes a novel joint opinion relation detection method using a one-class deep neural network (OCDNN). It addresses the problems of prior methods by simultaneously considering opinion words, targets, and their linking relations. The OCDNN consists of two levels: the lower level learns features using word embeddings and a recursive autoencoder; the higher level performs one-class classification. Experiments on customer review datasets show the proposed method outperforms baselines by up to 9% F-measure by verifying all three required conditions of opinion relations.
French machine reading for question answeringAli Kabbadj
This paper proposes to unlock the main barrier to machine reading and comprehension French natural language texts. This open the way to machine to find to a question a precise answer buried in the mass of unstructured French texts. Or to create a universal French chatbot. Deep learning has produced extremely promising results for various tasks in natural language understanding particularly topic classification, sentiment analysis, question answering, and language translation. But to be effective Deep Learning methods need very large training da-tasets. Until now these technics cannot be actually used for French texts Question Answering (Q&A) applications since there was not a large Q&A training dataset. We produced a large (100 000+) French training Dataset for Q&A by translating and adapting the English SQuAD v1.1 Dataset, a GloVe French word and character embed-ding vectors from Wikipedia French Dump. We trained and evaluated of three different Q&A neural network ar-chitectures in French and carried out a French Q&A models with F1 score around 70%.
THE EFFECTS OF THE LDA TOPIC MODEL ON SENTIMENT CLASSIFICATIONijscai
Online reviews are a feedback to the product and play a key role in improving the product to cater to consumers. Online reviews that rely heavily on manual categorization are time consuming and labor intensive.The recurrent neural network in deep learning can process time series data, while the long and short term memory network can process long time sequence data well. This has good experimental verification support in natural language processing, machine translation, speech recognition and language model.The merits of the extracted data features affect the classification results produced by the classification model. The LDA topic model adds a priori a posteriori knowledge to classify the data so that the characteristics of the data can be extracted efficiently.Applied to the classifier can improve accuracy and efficiency. Two-way long-term and short-term memory networks are variants and extensions of cyclic neural networks.The deep learning framework Keras uses Tensorflow as the backend to build a convenient two-way long-term and short-term memory network model, which provides a strong technical support for the experiment.Using the LDA topic model to extract the keywords needed to train the neural network and increase the internal relationship between words can improve the learning efficiency of the model. The experimental results in the same experimental environment are better than the traditional word frequency features.
This document discusses knowledge representation in artificial intelligence. It covers various techniques for knowledge representation including logical representation using propositional logic and first-order predicate logic, semantic network representation, frame representation, and production rules. It also discusses issues in knowledge representation such as representing important attributes, relationships, and granularity of knowledge. Propositional logic is introduced as the simplest form of logic where statements are represented by propositions that can be either true or false. The syntax and semantics of propositional logic are also covered.
NLP Bootcamp 2018 : Representation Learning of text for NLPAnuj Gupta
The document provides an outline for a workshop on representation learning of text for natural language processing (NLP). The workshop will be divided into 4 modules covering both foundational techniques like one-hot encoding and bag-of-words as well as state-of-the-art methods like word, sentence, and character vectors. The objective is for participants to gain a deeper understanding of the key ideas, math, and code behind text representation techniques in order to apply them to solve NLP problems and achieve higher accuracies and understanding.
This document provides an introduction to knowledge representation in artificial intelligence. It discusses how knowledge representation and reasoning forms the basis of intelligent behavior through computational means. The key types of knowledge that need to be represented are defined, including objects, events, facts, and meta-knowledge. Different types of knowledge such as declarative, procedural, structural and heuristic knowledge are explained. The importance of knowledge representation for modeling intelligent behavior in agents is highlighted. The requirements for effective knowledge representation including representational adequacy, inferential adequacy, inferential efficiency, and acquisitional efficiency are outlined. Propositional logic is introduced as the simplest form of logic using propositions.
Won Ik Cho presented on his research related to intention understanding in Korean natural language processing. He discussed developing annotation guidelines and corpora to classify Korean utterances by speech act, considering factors like intonation, context, and rhetoricalness. He proposed a method using text-based analysis combined with speech-aided disambiguation. Future work includes developing structured paraphrasing for argument extraction and an improved dialog manager.
This document provides an overview of natural language processing (NLP). It begins with examples of NLP applications like translation and question answering. It then discusses the backgrounds in artificial intelligence, linguistics, and the web. The document outlines several common NLP tasks like part-of-speech tagging, named-entity recognition, word sense disambiguation, and parsing. It also discusses challenges like ambiguity in natural language. The document concludes with a discussion of why NLP is difficult due to ambiguity at both the linguistic and acoustic levels.
Working in NLP in the Age of Large Language ModelsZachary S. Brown
The document provides an overview of recent advances in natural language processing (NLP) and large language models (LLMs). It discusses several key moments and technological developments that have contributed to progress in the field over the past decade, including the introduction of neural networks for language modeling in 2001, word embeddings in 2013, the attention mechanism in 2015, and the transformer architecture in 2017. Recent years have seen massive LLMs like GPT-3 achieve strong performance across many NLP tasks through techniques like self-supervised pre-training and scaling up model sizes. This has led to new tooling ecosystems and commercial applications of generative NLP, though discriminative tasks still rely on smaller, more efficient models when data is available.
SENTIMENT ANALYSIS IN MYANMAR LANGUAGE USING CONVOLUTIONAL LSTM NEURAL NETWORKijnlc
In recent years, there has been an increasing use of social media among people in Myanmar and writing review on social media pages about the product, movie, and trip are also popular among people. Moreover, most of the people are going to find the review pages about the product they want to buy before deciding whether they should buy it or not. Extracting and receiving useful reviews over interesting products is very important and time consuming for people. Sentiment analysis is one of the important processes for extracting useful reviews of the products. In this paper, the Convolutional LSTM neural network architecture is proposed to analyse the sentiment classification of cosmetic reviews written in Myanmar Language. The paper also intends to build the cosmetic reviews dataset for deep learning and sentiment lexicon in Myanmar Language.
Sentiment Analysis In Myanmar Language Using Convolutional Lstm Neural Networkkevig
In recent years, there has been an increasing use of social media among people in Myanmar and writing
review on social media pages about the product, movie, and trip are also popular among people. Moreover,
most of the people are going to find the review pages about the product they want to buy before deciding
whether they should buy it or not. Extracting and receiving useful reviews over interesting products is very
important and time consuming for people. Sentiment analysis is one of the important processes for extracting
useful reviews of the products. In this paper, the Convolutional LSTM neural network architecture is
proposed to analyse the sentiment classification of cosmetic reviews written in Myanmar Language. The
paper also intends to build the cosmetic reviews dataset for deep learning and sentiment lexicon in Myanmar
Language.
Thomas Wolf "An Introduction to Transfer Learning and Hugging Face"Fwdays
In this talk I'll start by introducing the recent breakthroughs in NLP that resulted from the combination of Transfer Learning schemes and Transformer architectures. The second part of the talk will be dedicated to an introduction of the open-source tools released by Hugging Face, in particular our transformers, tokenizers, and NLP libraries as well as our distilled and pruned models.
Analysis of the evolution of advanced transformer-based language models: Expe...IAESIJAI
This document analyzes the evolution of advanced transformer-based language models for opinion mining tasks. It provides background on several transformer models including BERT, GPT, ALBERT, RoBERTa, XLNet, DistilBERT, XLM-RoBERTa, BART, ConvBERT, Reformer, T5, ELECTRA, Longformer, and DeBERTa. The document compares these models based on their architecture, pre-training data, objectives, performance on tasks, and computational costs. It aims to study the behavior of these cutting-edge models on opinion mining and provide guidelines for researchers and engineers on model selection.
The document discusses the development of a chatbot using deep learning models like sequence-to-sequence learning with LSTM. It proposes using a movie dialog corpus consisting of over 220,000 conversational exchanges to train the chatbot model. Transfer learning is used with pre-trained word embeddings to improve the model. The chatbot implementation uses an LSTM encoder-decoder architecture in the sequence-to-sequence model.
This document introduces transfer learning and its importance for natural language processing (NLP). It discusses how transfer learning allows knowledge gained from one task or domain to be applied to another, similar to how humans learn. Large companies can train complex neural networks on vast datasets, but this requires massive resources. Transfer learning addresses this by enabling models pretrained on huge datasets to be fine-tuned and applied to new problems with far fewer resources. This paradigm has been key to advancing and democratizing NLP techniques.
State of the art in Natural Language Processing (March 2019)Liad Magen
2018 was the year of NLP, with many new advances such as ULMFiT, GPT, ELMo and BERT. But as these all come from the industry, are these advances in the 'right' direction? How does it compare to the academy advances? And where next? these topics (and more) are discussed in this presentation.
Feature Extraction and Analysis of Natural Language Processing for Deep Learn...Sharmila Sathish
This document discusses using deep learning techniques for multi-modal feature extraction. It proposes a multi-modal neural network with independent sub-networks for each data mode. It also discusses using a bi-directional GRU network for English word segmentation to effectively solve long-distance dependency issues while reducing training and prediction time compared to bi-directional LSTM. Experimental results showed the proposed multi-modal fusion model can effectively extract low-dimensional fused features from original high-dimensional multi-modal data.
The Smart Way to Invest in Artificial Intelligence and Machine Learning: Lisha Li, Amplify Partners
AI and ML are seeping into every startup, at least into every pitch deck. But what does it mean to build an AI/ML company? Some startups do require a closet filled with five PhD’s in data science, but that doesn’t necessarily mean yours does. Building intelligently with AI and ML.
Neel Sundaresan - Teaching a machine to codeMLconf
1. Recommend using the 'AdamOptimizer' class to optimize the loss since it is commonly used for training neural networks.
2. Suggest mapping the input data to floating point tensors using 'tf.cast()' for compatibility with TensorFlow operations.
3. Advise normalizing the input data to speed up training by using 'tf.keras.utils.normalize()'
IRJET - Automatic Lip Reading: Classification of Words and Phrases using Conv...IRJET Journal
This document presents research on developing an automatic lip reading system using convolutional neural networks. The system takes in video frames of a speaker's face without audio and classifies the words or phrases being spoken. The researchers preprocessed the data by detecting faces in video frames and cropping them. They then trained a CNN model on concatenated frames. Their model achieved 80.44% accuracy on the test set in classifying 10 words and 10 phrases from 17 speakers. The researchers concluded the model could be improved by addressing overfitting to unseen speakers with a larger dataset and regularization techniques.
Applications of Large Language Models in Materials Discovery and DesignAnubhav Jain
The document discusses applications of large language models (LLMs) in materials discovery and design. It describes how LLMs have improved natural language processing tasks related to materials science literature by requiring less custom model training and fine-tuning. As an example, the document discusses how LLMs were used to extract doping information from scientific papers and create a database of over 200,000 doped material compositions. The document suggests LLMs will continue enhancing materials databases and interfaces by integrating search and question-answering capabilities.
GPT and other Text Transformers: Black Swans and Stochastic ParrotsKonstantin Savenkov
Over the last year, we see increasingly more performant Text Transformers models, such as GPT-3 from OpenAI, Turing from Microsoft, and T5 from Google. They are capable of transforming the text in very creative and unexpected ways, like generating a summary of an article, explaining complex concepts in a simple language, or synthesizing realistic datasets for AI training. Unlike more traditional Machine Learning models, they do not require vast training datasets and can start based on just a few examples.
In this talk, we will make a short overview of such models, share the first experimental results and ask questions about the future of the content creation process. Are those models ready for prime time? What will happen to the professional content creators? Will they be able to compete against such powerful models? Will we see GPT post-editing similar to MT post-editing? We will share some answers we have based on the extensive experimenting and the first production projects that employ this new technology.
XAI LANGUAGE TUTOR - A XAI-BASED LANGUAGE LEARNING CHATBOT USING ONTOLOGY AND...kevig
In this paper, we proposed a XAI-based Language Learning Chatbot (namely XAI Language Tutor) by using ontology and transfer learning techniques. To facilitate three levels of language learning, XAI Language Tutor consists of three levels for systematically English learning, which includes: 1) phonetics level for speech recognition and pronunciation correction; 2) semantic level for specific domain conversation, and 3) simulation of “free-style conversation” in English - the highest level of language chatbot communication as “free-style conversation agent”. In terms of academic contribution, we implement the ontology graph to explain the performance of free-style conversation, following the concept of XAI (Explainable Artificial Intelligence) to visualize the connections of neural network in bionics, and explain the output sentence from language model. From implementation perspective, our XAI Language Tutor agent integrated the mini-program in WeChat as front-end, and fine-tuned GPT-2 model of transfer learning as back-end to interpret the responses by ontology graph.
[DSC MENA 24] Nada_GabAllah_-_Advancement_in_NLP_and_Text_Analytics.pptxDataScienceConferenc1
In recent years, NLP and text analytics have witnessed remarkable progress, transforming the way we interact with language data. From sentiment analysis to named entity recognition, these techniques play a pivotal role in understanding and extracting valuable insights from vast amounts of unstructured text. In this session, we’ll delve into the latest advancements, explore state-of-the-art models, and discuss practical applications across domains such as healthcare, finance, and customer service. Join us to unravel the intricacies of NLP and discover how it empowers organizations to unlock the hidden potential of textual information.
1) The document discusses various topics in natural language processing applications including machine translation techniques, sentiment analysis, question answering, text entailment, discourse processing, dialog systems, conversational agents, and natural language generation.
2) Machine translation techniques are discussed including rule-based translation, statistical machine translation, and examples of each. Statistical machine translation uses large corpora and probability to generate translations.
3) Other applications covered include sentiment analysis to determine sentiment in text, question answering systems, text entailment to determine entailment relationships, and discourse processing to analyze linguistic structures in text.
A NOVEL APPROACH FOR NAMED ENTITY RECOGNITION ON HINDI LANGUAGE USING RESIDUA...kevig
Many Natural Language Processing (NLP) applications involve Named Entity Recognition (NER) as an important task, where it leads to improve the overall performance of NLP applications. In this paper the Deep learning techniques are used to perform NER task on Hindi text data as it found that as compared to English NER, Hindi language NER is not sufficiently done. This is a barrier for resource-scarce languages as many resources are not readily available. Many researchers use various techniques such as rule based, machine learning based and hybrid approaches to solve this problem. Deep learning based algorithms are being developed in large scale as an innovative approach now a days for the advanced NER models which will give the best results out of it. In this paper we devise a Novel architecture based on residual network architecture for preferably Bidirectional Long Short Term Memory (BiLSTM) with fasttext word embedding layers. For this purpose we use pre-trained word embedding to represent the words in the corpus where the NER tags of the words are defined as the used annotated corpora. BiLSTM Development of an NER system for Indian languages is a comparatively difficult task. In this paper, we have done the various experiments to compare the results of NER with normal embedding and fasttext embedding layers to analyse the performance of word embedding with different batch sizes to train the deep learning models. Here we present a state-of-the-art results with said approach F1 Score measures.
Similar to Nlp 2020 global ai conf -jeff_shomaker_final (20)
Building RAG with self-deployed Milvus vector database and Snowpark Container...Zilliz
This talk will give hands-on advice on building RAG applications with an open-source Milvus database deployed as a docker container. We will also introduce the integration of Milvus with Snowpark Container Services.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slackshyamraj55
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfMalak Abu Hammad
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Nlp 2020 global ai conf -jeff_shomaker_final
1. Presentation at Global AI Conference
Santa Clara Convention Center
Santa Clara, CA
January 23, 2020
Jeff Shomaker
Founder, 21 SP, Inc.
Natural Language
Processing (NLP):
Future Trends
2. 21 SP, Inc.
Proprietary and Confidential 2
Introduction
• Natural Language Processing (NLP) is making rapid progress
• Talk will cover:
– A little history
– What is NLP and what can you do with it?
– Popular NLP Models
– Major Model Advancements in 2019
– Top 10 AI Use Cases by Revenue
– Commercial Products Announced in 2019
– Future Trends for NLP
3. 21 SP, Inc.
Proprietary and Confidential
How New is NLP?
– In 1637, Descartes wrote that language model-based
machines would likely be possible. He said:
For one may conceive that a machine would be so made
that it could generate speech…but not so that it could adapt
its output in a sufficiently versatile way so as to answer the
meaning of everything that could be said in its presence, as
even the stupidest of man can. 1)
(1) Francois Chollet (2019 Nov 19). @fchollet, Tweet, See Bibliography for full citation.
3
4. 21 SP, Inc.
Proprietary and Confidential
What Can NLP Do?
– NLP can be used for the following:
• Question answering
• Speech recognition
• Text-to-speech and speech-to-text
• Topic modeling
• Sentiment classification
• Language modeling
• Translation
• Others. 2)
(2) Rachael Thomas (2019 May-Jun). fast.ai Code-First Intro to Natural Language
Processing, youtube, accessed 11-11-19. See bibliography.
4
5. 21 SP, Inc.
Proprietary and Confidential
NLP Models
– In the past, machine learning-based NLP processing was done with
support vector machines (SVM) and logistic regression.
– Recently, neural network (NN) models have been increasingly used
and have provided better results on some NLP tasks.
– Deep learning NNs use a “… multi-level automatic feature
representation….” approach.
– In the following, we will discuss four types of models used for NLP:
• Convolutional neural networks (CNNs)
• Recurrent neural networks (RNNs)
• Recursive neural networks (Recur NNs) 3)
• Transformer models. 17)
(3) Tom Young, et al (2018 Nov 25). Deep learning trends for NLP. See bibliography.
(17) Ashish Vaswani, et al (2017 Dec 6). Attention is all you need [On Transformers]. See
bibliography.
5
6. 21 SP, Inc.
Proprietary and Confidential
NLP: Convolutional Neural Networks (CNNs)
– Collobert and Weston proposed CNNs for sentence modeling and
other NLP tasks in 2008 and 2011 respectively.
– Word embeddings - an approach that identifies which words are
similar and the relationships between them.
– After word embeddings came about, it was necessary to extract “…
higher-level features from constituting words or n-grams.” 20)
– NLP models use these for translation, question answering,
summarization and sentiment analysis.
– CNNs are good for capturing semantic information.
– CNNs are not so good when data scarcity exists and they struggle
when modeling some contextual information. 3)
(3) Tom Young, et al (2018 Nov 25). Deep learning trends for NLP. See Bibliography.
(20) N-grams are a sequence of words, eg, a 2-gram might be “please turn”. S. Kapadia (2019 Mar 26).
Language Models: N-Gram, See bibliography. .
6
7. 21 SP, Inc.
Proprietary and Confidential
NLP: Recurrent Neural Networks (RNN)
– RNN design (ie, sequential information processing) makes them good for
NLP since they do the same thing on each instance of the sequence. 3)
– Since RNNs process in a looping fashion and they reuse weights,
information is remembered during processing. 4)
– The process works well for language modeling, machine translation,
speech recognition and image capturing.
– It is not clear whether RNNs or CNNs are best for sentiment
classification, question answering or part-of-speech-tagging.
– One drawback with simple RNNs is they struggle in learning and tuning
an NN’s early layers. LSTMs (ie, RNN variant) overcome this issue. 3)
(3) Tom Young, et al (2018 Nov 25). Deep learning trends for NLP. See bibliography.
(4) Vincent Boucher (2019 Nov 25). AI Overview, academy.montreal.ai. See bibliography.
7
8. 21 SP, Inc.
Proprietary and Confidential
NLP: Recursive Neural Networks (Recur NNs)
– Recursive NNs use a tree-structure to account for the fact that in
language “… words and sub-phrases combine into phrases in a
hierarchical manner.”
– Recur NNs work well for parsing* and there are two types:
• Recursive autoencoders – can reconstruct context
• Recursive neural tensor network – at each node in a tree, they
compute a supervised objective. 3)
– Recur NNs can be used for the following applications:
• NLP
• Audio-to-text transcription
• Image scene decomposition. 5)
* Parsing – “Analyze (a string or text) into logical syntactic components, typically in order to test conformability to a
logical grammar.” LEXICO, lexico.com.
(3) Tom Young, et al (2018 Nov 25). Deep learning trends for NLP. See bibliography.
(5) J. Patterson & A. Gibson (2017). Deep Learning: A Practitioner’s Approach. See bibliography.
8
9. 21 SP, Inc.
Proprietary and Confidential
NLP: Transformers
– The Transformer model is a new approach to processing text that
changed thinking and enabled Google’s BERT, which set the standard.
16), 18)
– The Transformer uses only attention mechanisms, which allow them to
avoid recurrence and convolutions.
– This new architecture makes it possible to process in a parallel fashion,
which can speed up training significantly over RNNs.
– In this paper, which proposed this innovative architecture, Transformers
were tested on translation tasks using BLEU:
• For English-to-German, results showed improvement over all other
models
• For English-to-French, results bested all earlier single models. 17)
(16) Prateek Joshi (2019 Jun 19). How do Transformers Work in NLP? See bibliography.
(17) Ashish Vaswani, et al (2017 Dec 6). Attention is all you need [On Transformers]. See bibliography.
(18) Jacob Devlin, et al (2019 May 24 – 1st version 2018 Oct 11). BERT: Pre-training of Deep Bidirectional
Transformers. See bibliography. On 11 NLP tasks, BERT beat all earlier models.
9
10. 21 SP, Inc.
Proprietary and Confidential
2019 Advancement: XLNET
– XLNET is a “... Generalized autoregressive pretraining method ….” that
allows learning in a bidirectional manner and has improved performance
compared to BERT.
– XLNET has achieved excellent results on:
• Language understanding
• Reading comprehension
• Test classification
• Document ranking.
– Using many benchmarks, XLNET is superior to BERT. 11)
– Potential business applications include:
• Automated customer support
• Sentiment analysis (eg, brand awareness)
• Information searching in document data bases. 12)
(11) Zhilin Yang, et al (2019 Jun 19). XLNET: Generalized Autoregressive Pretraining. See bibliography.
(12) Mariya Yao (2019 Nov 12). Major NLP Achievements in 2019. See bibliography.
10
11. 21 SP, Inc.
Proprietary and Confidential
2019 Advancement: ALBERT – A Lite BERT
– ALBERT is an architecture that uses a transformer encoder.*
– ALBERT scales better than BERT due to parameter reductions and an
improved method for predicting sentences.
– With fewer parameters, memory needs are reduced and training speed is
increased.
– ALBERT delivers state-of-the-art performances on GLUE, RACE, and
SQuAD tests. 15)
– Potential business applications include:
• Chatbots
• Sentiment analysis
• Document mining
• Text classification. 12)
*Transformers are a major improvement over RNN-based sequence-to-sequence models used previously.
(15) Zhenzhong Lan, et al (2019 Oct 30). ALBERT: A Lite BERT for Self-Supervised Learning. See bibliography.
(12) Mariya Yao (2019 Nov 12). Major NLP Achievements in 2019. See bibliography.
11
12. 21 SP, Inc.
Proprietary and Confidential
2019 Advancement: DistilBERT – For the Edge
– Transfer learning is used more and more in NLP; but, these big models
are hard to train to do inference on-the-edge (eg, small devices). 21)
– These large models can have parameters in the hundreds of millions.
– Paper’s authors propose an approach for a “… general purpose language
representation model .…” that can run on-the-edge.
– DistilBERT use knowledge distillation which “… is a compression
technique in which a compact model – the student – is trained to
reproduce the behavior of a larger model – the teacher – or an ensemble
of models.”
– They discuss models that are:
• 40% smaller than BERT
• Keep 97% of BERT’s language understanding
• Are 60% faster than BERT. 13)
(13) Victor Sanh, et al (2019 Oct 16). DistilBERT, A distilled version of BERT. See bibliography.
(21) Dipanjan (DJ) Sarkar (2018 Nov 14). A Comprehensive Hands-on Guide to Transfer Learning. See
bibliography. Transfer learning allows information gained from an earlier model to be used for training a
subsequent model (eg, reuse weights).
12
13. 21 SP, Inc.
Proprietary and Confidential
AI Revenue -Top Ten Use Cases Forecast 6)
13
(6) Mike Quindazzi (2019 Dec 7). Cumulative AI Software Revenue, @tractica via @mikequindazzi,
Tweet, See bibliography.
14. 21 SP, Inc.
Proprietary and Confidential
2019 New Product: Qualcomm’s Chipset
– High end Android phones will soon have a new chipset called the
Snapdragon 865.
– The chip will make possible 5G, digital drivers licenses, a 200 megapixel
camera and improved gaming on smartphones.
– A very notable feature is “Instant live translation and transcription.”
– Google Translate already has very good quality -- Android 10 provides
live video transcriptions and Pixel 4 does translations with it’s voice
recorder app.
– It is reported that the demonstrations with the 865 are far superior. They
show:
• On-the-fly translation in English and Chinese
• Words spoken in English can be translated and then spoken in
Chinese to the listener. 9)
(9) Jessica Dolcourt (2019 Dec 19). 6 Things Coming to Android Phones, CNET. See bibliography.
14
15. 21 SP, Inc.
Proprietary and Confidential
2019 New Product: Automatic Transcription
– Amazon has brought out a product called Transcribe Medical
• Uses a voice app and Amazon’s Web Services to automatically
transcribe patient-doctor discussions
• Once the discussion is complete, a transcription is placed in patient’s
records.
– Patient health records can be further analyzed by Amazon’s Comprehend
Medical product
• This analyzes text for patterns that can be used to highlight possible
diagnoses and approaches to treatment.
– Nuance and Microsoft collaborated earlier and deployed an improved
Dragon Medical Virtual Assistant using Microsoft’s Azure platform.
– Google is also developing a medical transcription product with Stanford
University. 8)
(8) Eric H. Schwartz (2019 Dec 6). Amazon Launches Medical Transcription, voicebot.ai. See bibliography.
15
16. 21 SP, Inc.
Proprietary and Confidential
2019 New Product: AI-enabled Robot w/ Emotion
– CIMON (Crew Interactive Mobile Companion) was the first smart
astronaut assistant used on the International Space Station (ISS).
– CIMON is a collaboration between IBM (Watson AI), Airbus, and the
German Aerospace Center.
– The robotic assistant is designed to retrieve information, track tasks, and
with the new version, CIMON-2, mitigate issues between crew members.
– CIMON-2, which was sent aboard December 2019’s SpaceX ISS
resupply mission, now uses IBM Watson’s Tone Analyzer.
– The goal is to analyze astronaut’s conversations to allow for:
• The mitigation of group-think
• Provide an objective perspective or contrarian viewpoint when
complicated decisions need to be made. 10)
(10) D. Etherington (2019 Dec 5). AI-enabled Assistant Returns to Space, Techcrunch. See bibliography.
16
17. 21 SP, Inc.
Proprietary and Confidential
A 2020 NLP Wish List
– Sebastion Ruder’s list includes:
• Learning from few samples rather than large datasets
• Compact and efficient rather than huge models
• Evaluate on at least another language (from a different language
family)
• New datasets contain at least one other language
• NLP helps to unlock scientific knowledge (see Nature 19)). 7)
(7) Sebastian Ruder (2019 Dec 23). 2020 Wish Lists, NLP News, Issue #48. See bibliography.
(19) Vahe Tshitoyan, et al (2019 Jul 3). Unsupervised learning captures latent knowledge. See bibliography.
17
18. 21 SP, Inc.
Proprietary and Confidential
2020 AI and Machine Learning Trends
– Natural Language Processing
• Pre-training and fine tuning is the approach
• Improvements in NLP are likely to be driven by the use of linguistics
and the incorporation of more human knowledge
• Neural machine translation (ie, simultaneous machine translation) is
now practical and is expected to improve further
– Conversational AI
• Chatbots are widely used for customer service
• To improve Chatbots, researchers are:
– Leveraging conversational history and context
– Working towards providing more varied responses
– Attempting to incorporate emotion and empathy into
conversations. 14)
(14) Kate Koidan (2020 Jan 2). Top AI Trends for 2020, TopBots. See bibliography.
18
19. 21 SP, Inc.
Proprietary and Confidential
End Notes
• 1) Francois Chollet [Google Deep Learning] (2019 Nov 19), @fenollet, Tweet, accessed 12-18-19.
• 2) Rachael Thomas (2019 May-Jun). fast.ai Code-First Intro to Natural Language Processing, What is
NLP? (NLP Video 1), youtube, accessed 11-11-19.
• 3) Tom Young, et al (2018 Nov 25). Recent Trends in Deep Learning Based Natural Language
Processing, ar.Xiv:1708.02709v8 [cs.CL], accessed November 2019.
• 4) Vincent Boucher (2019 Nov 25). Montreal.AI Academy: Artificial Intelligence 101 First World-Class
Overview of AI for All, Montreal.AI, academy.montreal.ai, accessed 12-28-19.
• 5) Josh Patterson and Adam Gibson, Deep Learning: A Practitioner’s Approach, O’Reilly 2017,
accessed 12-22-19 at allite.books.com.
• 6) Mike Quindazzi (2019 Dec 7). Cumulative Artificial Intelligence Revenue, Top Ten Use Cases
World Markets:2017-2025, @tractica via @mikequindazzi, Tweet, accessed 12-29-19.
• 7) Sebastian Ruder [Deep Mind AI] (2019 Dec 23). 2020 Wish Lists, Natural Language Processing
News, Issue #48, newsletter.ruder.io, accessed 12-31-19.
• 8) Eric Hal Schwartz (2019 Dec 6). Amazon Launches Medical Transcription Service in Direct
Competition with Nuance [Nuance partner is Microsoft], voicebot.ai, accessed 12-30-19.
• 9) Jessica Dolcourt (2019 Dec 12). 6 Things Coming to 2020 Android Phones: 8K Video, 5G Uploads,
Two-finger Unlock, CNET, cnet.com, accessed 1-1-20.
19
20. 21 SP, Inc.
Proprietary and Confidential
End Notes (cont.)
• 10) Darrell Etherington (2019 Dec 5). AI-enabled Assistant Robot Returning to the Space Station with
Improved Emotional Intelligence, Techcrunch, techcrunch.com, accessed 12-31-19.
• 11) Zhilin Yang, et al (2019 Jun 19). XLNET: Generalized Autoregressive Pretraining for Language
Understanding. arXiv:1906.08237v1 [cs.CL].
• 12) Mariya Yao (2019 Nov 12). What Are Major NLP Achievements & Papers from 2019, TopBots,
topbots.com, accessed 12-31-19.
• 13) Victor Sanh, et al (2019 Oct 16). DistilBERT, a Distilled Version of BERT: Smaller, Faster,
Cheaper and Lighter, arXiv:1910.01108v2 [cs.CL], accessed 1-8-20.
• 14) Kate Koidan (2020 Jan 2). Top AI & ML Research Trends for 2020, TopBots, topbots.com,
accessed 1-5-20.
• 15) Zhenzhong Lan, et al (2019 Oct 30). ALBERT: A Lite BERT for Self-Supervised Learning of
Language Representations, arXiv:1909.11942v3 [cs.CL], accessed 1-2-20.
• 16) Prateek Joshi (2019 Jun 19). How do Transformers Work in NLP: A Guide to the Latest State-of-
the-Art Models, Analytics Vidhya, analyticsvidhya.com, accessed 1-10-20.
• 17) Ashish Vaswani, et al (2017 Dec 6). Attention Is All You Need [Proposes The Transformer model],
arXiv:1706.03762v5 [cs.CL].
• 18) Jacob Devlin, et al (2019 May 24 – 1st version 2018 Oct 11). BERT: Pre-training of Deep
Bidirectional Transformers for Language Understanding, arXiv:1810.04805v2 [cs.CL].
20
21. 21 SP, Inc.
Proprietary and Confidential
End Notes (cont.)
• 19) Vahe Tshitoyan, et al (2019 Jul 3). Unsupervised Word Embeddings Capture Latent Knowledge
from Materials Science Literature, Nature, nature.com, accessed 1-17-20.
• 20) Shashank Kapadia (2019 Mar 26). Language Models: N-Gram, Towards Data Science,
towardsdatascience.com, accessed 1-2-20.
• 21) Dipanjan (DJ) Sarkar (2018 Nov 14). A Comprehensive Hands-on Guide to Transfer Learning with
Real-World Applications in Deep Learning, Towards Data Science, towardsdatascience.com,
accessed 1-19-20.
•
21
22. 21 SP, Inc.
Proprietary and Confidential
Contacts
• Jeff Shomaker – Founder/President 21 SP, Inc.
– jshomaker@21spinc.com
– www.21spinc.com
– 650-285-8122
• 21 SP, Inc. is a small privately held startup developing and marketing
expert systems-based decision support software to use in genetic-
based personalized medicine. The company's mission is to create
tools that will reduce the use of traditional trial-and-error medicine by
using pharmacogenetics and other evidence-based data, such as the
results of high quality clinical trials, in the medical clinic.
22