My talk at the 4th Annual Global AI Conference in Santa Clara, California on January 23, 2020. Topic: Future Trends in Natural Language Processing.

1. Presentation at Global AI Conference
Santa Clara Convention Center
Santa Clara, CA
January 23, 2020
Jeff Shomaker
Founder, 21 SP, Inc.
Natural Language
Processing (NLP):
Future Trends

2. 21 SP, Inc.
Proprietary and Confidential 2
Introduction
• Natural Language Processing (NLP) is making rapid progress
• Talk will cover:
– A little history
– What is NLP and what can you do with it?
– Popular NLP Models
– Major Model Advancements in 2019
– Top 10 AI Use Cases by Revenue
– Commercial Products Announced in 2019
– Future Trends for NLP

3. 21 SP, Inc.
Proprietary and Confidential
How New is NLP?
– In 1637, Descartes wrote that language model-based
machines would likely be possible. He said:
For one may conceive that a machine would be so made
that it could generate speech…but not so that it could adapt
its output in a sufficiently versatile way so as to answer the
meaning of everything that could be said in its presence, as
even the stupidest of man can. 1)
(1) Francois Chollet (2019 Nov 19). @fchollet, Tweet, See Bibliography for full citation.
3

4. 21 SP, Inc.
Proprietary and Confidential
What Can NLP Do?
– NLP can be used for the following:
• Question answering
• Speech recognition
• Text-to-speech and speech-to-text
• Topic modeling
• Sentiment classification
• Language modeling
• Translation
• Others. 2)
(2) Rachael Thomas (2019 May-Jun). fast.ai Code-First Intro to Natural Language
Processing, youtube, accessed 11-11-19. See bibliography.
4

5. 21 SP, Inc.
Proprietary and Confidential
NLP Models
– In the past, machine learning-based NLP processing was done with
support vector machines (SVM) and logistic regression.
– Recently, neural network (NN) models have been increasingly used
and have provided better results on some NLP tasks.
– Deep learning NNs use a “… multi-level automatic feature
representation….” approach.
– In the following, we will discuss four types of models used for NLP:
• Convolutional neural networks (CNNs)
• Recurrent neural networks (RNNs)
• Recursive neural networks (Recur NNs) 3)
• Transformer models. 17)
(3) Tom Young, et al (2018 Nov 25). Deep learning trends for NLP. See bibliography.
(17) Ashish Vaswani, et al (2017 Dec 6). Attention is all you need [On Transformers]. See
bibliography.
5

6. 21 SP, Inc.
Proprietary and Confidential
NLP: Convolutional Neural Networks (CNNs)
– Collobert and Weston proposed CNNs for sentence modeling and
other NLP tasks in 2008 and 2011 respectively.
– Word embeddings - an approach that identifies which words are
similar and the relationships between them.
– After word embeddings came about, it was necessary to extract “…
higher-level features from constituting words or n-grams.” 20)
– NLP models use these for translation, question answering,
summarization and sentiment analysis.
– CNNs are good for capturing semantic information.
– CNNs are not so good when data scarcity exists and they struggle
when modeling some contextual information. 3)
(3) Tom Young, et al (2018 Nov 25). Deep learning trends for NLP. See Bibliography.
(20) N-grams are a sequence of words, eg, a 2-gram might be “please turn”. S. Kapadia (2019 Mar 26).
Language Models: N-Gram, See bibliography. .
6

7. 21 SP, Inc.
Proprietary and Confidential
NLP: Recurrent Neural Networks (RNN)
– RNN design (ie, sequential information processing) makes them good for
NLP since they do the same thing on each instance of the sequence. 3)
– Since RNNs process in a looping fashion and they reuse weights,
information is remembered during processing. 4)
– The process works well for language modeling, machine translation,
speech recognition and image capturing.
– It is not clear whether RNNs or CNNs are best for sentiment
classification, question answering or part-of-speech-tagging.
– One drawback with simple RNNs is they struggle in learning and tuning
an NN’s early layers. LSTMs (ie, RNN variant) overcome this issue. 3)
(3) Tom Young, et al (2018 Nov 25). Deep learning trends for NLP. See bibliography.
(4) Vincent Boucher (2019 Nov 25). AI Overview, academy.montreal.ai. See bibliography.
7

8. 21 SP, Inc.
Proprietary and Confidential
NLP: Recursive Neural Networks (Recur NNs)
– Recursive NNs use a tree-structure to account for the fact that in
language “… words and sub-phrases combine into phrases in a
hierarchical manner.”
– Recur NNs work well for parsing* and there are two types:
• Recursive autoencoders – can reconstruct context
• Recursive neural tensor network – at each node in a tree, they
compute a supervised objective. 3)
– Recur NNs can be used for the following applications:
• NLP
• Audio-to-text transcription
• Image scene decomposition. 5)
* Parsing – “Analyze (a string or text) into logical syntactic components, typically in order to test conformability to a
logical grammar.” LEXICO, lexico.com.
(3) Tom Young, et al (2018 Nov 25). Deep learning trends for NLP. See bibliography.
(5) J. Patterson & A. Gibson (2017). Deep Learning: A Practitioner’s Approach. See bibliography.
8

9. 21 SP, Inc.
Proprietary and Confidential
NLP: Transformers
– The Transformer model is a new approach to processing text that
changed thinking and enabled Google’s BERT, which set the standard.
16), 18)
– The Transformer uses only attention mechanisms, which allow them to
avoid recurrence and convolutions.
– This new architecture makes it possible to process in a parallel fashion,
which can speed up training significantly over RNNs.
– In this paper, which proposed this innovative architecture, Transformers
were tested on translation tasks using BLEU:
• For English-to-German, results showed improvement over all other
models
• For English-to-French, results bested all earlier single models. 17)
(16) Prateek Joshi (2019 Jun 19). How do Transformers Work in NLP? See bibliography.
(17) Ashish Vaswani, et al (2017 Dec 6). Attention is all you need [On Transformers]. See bibliography.
(18) Jacob Devlin, et al (2019 May 24 – 1st version 2018 Oct 11). BERT: Pre-training of Deep Bidirectional
Transformers. See bibliography. On 11 NLP tasks, BERT beat all earlier models.
9

10. 21 SP, Inc.
Proprietary and Confidential
2019 Advancement: XLNET
– XLNET is a “... Generalized autoregressive pretraining method ….” that
allows learning in a bidirectional manner and has improved performance
compared to BERT.
– XLNET has achieved excellent results on:
• Language understanding
• Reading comprehension
• Test classification
• Document ranking.
– Using many benchmarks, XLNET is superior to BERT. 11)
– Potential business applications include:
• Automated customer support
• Sentiment analysis (eg, brand awareness)
• Information searching in document data bases. 12)
(11) Zhilin Yang, et al (2019 Jun 19). XLNET: Generalized Autoregressive Pretraining. See bibliography.
(12) Mariya Yao (2019 Nov 12). Major NLP Achievements in 2019. See bibliography.
10

11. 21 SP, Inc.
Proprietary and Confidential
2019 Advancement: ALBERT – A Lite BERT
– ALBERT is an architecture that uses a transformer encoder.*
– ALBERT scales better than BERT due to parameter reductions and an
improved method for predicting sentences.
– With fewer parameters, memory needs are reduced and training speed is
increased.
– ALBERT delivers state-of-the-art performances on GLUE, RACE, and
SQuAD tests. 15)
– Potential business applications include:
• Chatbots
• Sentiment analysis
• Document mining
• Text classification. 12)
*Transformers are a major improvement over RNN-based sequence-to-sequence models used previously.
(15) Zhenzhong Lan, et al (2019 Oct 30). ALBERT: A Lite BERT for Self-Supervised Learning. See bibliography.
(12) Mariya Yao (2019 Nov 12). Major NLP Achievements in 2019. See bibliography.
11

12. 21 SP, Inc.
Proprietary and Confidential
2019 Advancement: DistilBERT – For the Edge
– Transfer learning is used more and more in NLP; but, these big models
are hard to train to do inference on-the-edge (eg, small devices). 21)
– These large models can have parameters in the hundreds of millions.
– Paper’s authors propose an approach for a “… general purpose language
representation model .…” that can run on-the-edge.
– DistilBERT use knowledge distillation which “… is a compression
technique in which a compact model – the student – is trained to
reproduce the behavior of a larger model – the teacher – or an ensemble
of models.”
– They discuss models that are:
• 40% smaller than BERT
• Keep 97% of BERT’s language understanding
• Are 60% faster than BERT. 13)
(13) Victor Sanh, et al (2019 Oct 16). DistilBERT, A distilled version of BERT. See bibliography.
(21) Dipanjan (DJ) Sarkar (2018 Nov 14). A Comprehensive Hands-on Guide to Transfer Learning. See
bibliography. Transfer learning allows information gained from an earlier model to be used for training a
subsequent model (eg, reuse weights).
12

13. 21 SP, Inc.
Proprietary and Confidential
AI Revenue -Top Ten Use Cases Forecast 6)
13
(6) Mike Quindazzi (2019 Dec 7). Cumulative AI Software Revenue, @tractica via @mikequindazzi,
Tweet, See bibliography.

14. 21 SP, Inc.
Proprietary and Confidential
2019 New Product: Qualcomm’s Chipset
– High end Android phones will soon have a new chipset called the
Snapdragon 865.
– The chip will make possible 5G, digital drivers licenses, a 200 megapixel
camera and improved gaming on smartphones.
– A very notable feature is “Instant live translation and transcription.”
– Google Translate already has very good quality -- Android 10 provides
live video transcriptions and Pixel 4 does translations with it’s voice
recorder app.
– It is reported that the demonstrations with the 865 are far superior. They
show:
• On-the-fly translation in English and Chinese
• Words spoken in English can be translated and then spoken in
Chinese to the listener. 9)
(9) Jessica Dolcourt (2019 Dec 19). 6 Things Coming to Android Phones, CNET. See bibliography.
14

15. 21 SP, Inc.
Proprietary and Confidential
2019 New Product: Automatic Transcription
– Amazon has brought out a product called Transcribe Medical
• Uses a voice app and Amazon’s Web Services to automatically
transcribe patient-doctor discussions
• Once the discussion is complete, a transcription is placed in patient’s
records.
– Patient health records can be further analyzed by Amazon’s Comprehend
Medical product
• This analyzes text for patterns that can be used to highlight possible
diagnoses and approaches to treatment.
– Nuance and Microsoft collaborated earlier and deployed an improved
Dragon Medical Virtual Assistant using Microsoft’s Azure platform.
– Google is also developing a medical transcription product with Stanford
University. 8)
(8) Eric H. Schwartz (2019 Dec 6). Amazon Launches Medical Transcription, voicebot.ai. See bibliography.
15

16. 21 SP, Inc.
Proprietary and Confidential
2019 New Product: AI-enabled Robot w/ Emotion
– CIMON (Crew Interactive Mobile Companion) was the first smart
astronaut assistant used on the International Space Station (ISS).
– CIMON is a collaboration between IBM (Watson AI), Airbus, and the
German Aerospace Center.
– The robotic assistant is designed to retrieve information, track tasks, and
with the new version, CIMON-2, mitigate issues between crew members.
– CIMON-2, which was sent aboard December 2019’s SpaceX ISS
resupply mission, now uses IBM Watson’s Tone Analyzer.
– The goal is to analyze astronaut’s conversations to allow for:
• The mitigation of group-think
• Provide an objective perspective or contrarian viewpoint when
complicated decisions need to be made. 10)
(10) D. Etherington (2019 Dec 5). AI-enabled Assistant Returns to Space, Techcrunch. See bibliography.
16

17. 21 SP, Inc.
Proprietary and Confidential
A 2020 NLP Wish List
– Sebastion Ruder’s list includes:
• Learning from few samples rather than large datasets
• Compact and efficient rather than huge models
• Evaluate on at least another language (from a different language
family)
• New datasets contain at least one other language
• NLP helps to unlock scientific knowledge (see Nature 19)). 7)
(7) Sebastian Ruder (2019 Dec 23). 2020 Wish Lists, NLP News, Issue #48. See bibliography.
(19) Vahe Tshitoyan, et al (2019 Jul 3). Unsupervised learning captures latent knowledge. See bibliography.
17

18. 21 SP, Inc.
Proprietary and Confidential
2020 AI and Machine Learning Trends
– Natural Language Processing
• Pre-training and fine tuning is the approach
• Improvements in NLP are likely to be driven by the use of linguistics
and the incorporation of more human knowledge
• Neural machine translation (ie, simultaneous machine translation) is
now practical and is expected to improve further
– Conversational AI
• Chatbots are widely used for customer service
• To improve Chatbots, researchers are:
– Leveraging conversational history and context
– Working towards providing more varied responses
– Attempting to incorporate emotion and empathy into
conversations. 14)
(14) Kate Koidan (2020 Jan 2). Top AI Trends for 2020, TopBots. See bibliography.
18

19. 21 SP, Inc.
Proprietary and Confidential
End Notes
• 1) Francois Chollet [Google Deep Learning] (2019 Nov 19), @fenollet, Tweet, accessed 12-18-19.
• 2) Rachael Thomas (2019 May-Jun). fast.ai Code-First Intro to Natural Language Processing, What is
NLP? (NLP Video 1), youtube, accessed 11-11-19.
• 3) Tom Young, et al (2018 Nov 25). Recent Trends in Deep Learning Based Natural Language
Processing, ar.Xiv:1708.02709v8 [cs.CL], accessed November 2019.
• 4) Vincent Boucher (2019 Nov 25). Montreal.AI Academy: Artificial Intelligence 101 First World-Class
Overview of AI for All, Montreal.AI, academy.montreal.ai, accessed 12-28-19.
• 5) Josh Patterson and Adam Gibson, Deep Learning: A Practitioner’s Approach, O’Reilly 2017,
accessed 12-22-19 at allite.books.com.
• 6) Mike Quindazzi (2019 Dec 7). Cumulative Artificial Intelligence Revenue, Top Ten Use Cases
World Markets:2017-2025, @tractica via @mikequindazzi, Tweet, accessed 12-29-19.
• 7) Sebastian Ruder [Deep Mind AI] (2019 Dec 23). 2020 Wish Lists, Natural Language Processing
News, Issue #48, newsletter.ruder.io, accessed 12-31-19.
• 8) Eric Hal Schwartz (2019 Dec 6). Amazon Launches Medical Transcription Service in Direct
Competition with Nuance [Nuance partner is Microsoft], voicebot.ai, accessed 12-30-19.
• 9) Jessica Dolcourt (2019 Dec 12). 6 Things Coming to 2020 Android Phones: 8K Video, 5G Uploads,
Two-finger Unlock, CNET, cnet.com, accessed 1-1-20.
19

20. 21 SP, Inc.
Proprietary and Confidential
End Notes (cont.)
• 10) Darrell Etherington (2019 Dec 5). AI-enabled Assistant Robot Returning to the Space Station with
Improved Emotional Intelligence, Techcrunch, techcrunch.com, accessed 12-31-19.
• 11) Zhilin Yang, et al (2019 Jun 19). XLNET: Generalized Autoregressive Pretraining for Language
Understanding. arXiv:1906.08237v1 [cs.CL].
• 12) Mariya Yao (2019 Nov 12). What Are Major NLP Achievements & Papers from 2019, TopBots,
topbots.com, accessed 12-31-19.
• 13) Victor Sanh, et al (2019 Oct 16). DistilBERT, a Distilled Version of BERT: Smaller, Faster,
Cheaper and Lighter, arXiv:1910.01108v2 [cs.CL], accessed 1-8-20.
• 14) Kate Koidan (2020 Jan 2). Top AI & ML Research Trends for 2020, TopBots, topbots.com,
accessed 1-5-20.
• 15) Zhenzhong Lan, et al (2019 Oct 30). ALBERT: A Lite BERT for Self-Supervised Learning of
Language Representations, arXiv:1909.11942v3 [cs.CL], accessed 1-2-20.
• 16) Prateek Joshi (2019 Jun 19). How do Transformers Work in NLP: A Guide to the Latest State-of-
the-Art Models, Analytics Vidhya, analyticsvidhya.com, accessed 1-10-20.
• 17) Ashish Vaswani, et al (2017 Dec 6). Attention Is All You Need [Proposes The Transformer model],
arXiv:1706.03762v5 [cs.CL].
• 18) Jacob Devlin, et al (2019 May 24 – 1st version 2018 Oct 11). BERT: Pre-training of Deep
Bidirectional Transformers for Language Understanding, arXiv:1810.04805v2 [cs.CL].
20

21. 21 SP, Inc.
Proprietary and Confidential
End Notes (cont.)
• 19) Vahe Tshitoyan, et al (2019 Jul 3). Unsupervised Word Embeddings Capture Latent Knowledge
from Materials Science Literature, Nature, nature.com, accessed 1-17-20.
• 20) Shashank Kapadia (2019 Mar 26). Language Models: N-Gram, Towards Data Science,
towardsdatascience.com, accessed 1-2-20.
• 21) Dipanjan (DJ) Sarkar (2018 Nov 14). A Comprehensive Hands-on Guide to Transfer Learning with
Real-World Applications in Deep Learning, Towards Data Science, towardsdatascience.com,
accessed 1-19-20.
•
21

22. 21 SP, Inc.
Proprietary and Confidential
Contacts
• Jeff Shomaker – Founder/President 21 SP, Inc.
– jshomaker@21spinc.com
– www.21spinc.com
– 650-285-8122
• 21 SP, Inc. is a small privately held startup developing and marketing
expert systems-based decision support software to use in genetic-
based personalized medicine. The company's mission is to create
tools that will reduce the use of traditional trial-and-error medicine by
using pharmacogenetics and other evidence-based data, such as the
results of high quality clinical trials, in the medical clinic.
22