Ai — Externalization of Mind

© Copyright Project10x | Confidential 2
• Lawrence Mills Davis is founder and managing director of
Project10X, a research consultancy known for forward-looking
industry studies; multi-company innovation and market
development programs; and business solution strategy
consulting. Mills brings 30 years experience as an industry
analyst, business consultant, computer scientist, and
entrepreneur. He is the author of more than 50 reports,
whitepapers, articles, and industry studies.
• Mills researches artificial intelligence technologies and their
applications across industries, including cognitive computing,
machine learning (ML), deep learning (DL), predictive analytics,
symbolic AI reasoning, expert systems (ES), natural language
processing (NLP), conversational UI, intelligent assistance (IA),
and robotic process automation (RPA), and autonomous multi-
agent systems.
• For clients seeking to exploit transformative opportunities
presented by the rapidly evolving capabilities of artificial
intelligence, Mills brings a depth and breadth of expertise to help
leaders realize their goals. More than narrow specialization, he
brings perspective that combines understanding of business,
technology, and creativity. Mills fills roles that include industry
research, venture development, and solution envisioning.
Lawrence Mills Davis
Managing Director
Project10X
mdavis@project10x.com
202-667-6400
© Copyright Project10x | Confidential

Direct competitors for Publicis.Sapient include digital agencies, consultants, IT services, which are providing AI
and cognitive platforms as a basis for custom solutions, products/services, and XaaS offerings to markets
addressed by Publicis.Sapient
AI encompasses multiple technologies that can be combined to
sense, think, and act as well as to learn from experience and
adapt over time.
SENSE
Computer vision, audio and
affective processing aim to
actively perceive the world
around them by acquiring and
processing images, sounds,
speech, biometrics, and other
sensory inputs. One example is
identity analytics for facial
recognition. Lie detection is
another.
THINK
Natural language processing
and inference engines enable
AI systems to analyze,
interpret, and understand
information. One example is
speech analytics and language
translation of search engine
results. Another is
interpretation of user intent by
virtual assistants..
ACT
AI systems take action in digital
or physical worlds using
machine learning, expert
systems and inference engines.
Recommendation systems are
one example. Another is auto-
pilot and assisted-braking
capabilities in cars. Cognitive
robotics is another.
3

AI technologies mimic human abilities
to sense, think, and act.
Source: Forrester, TechRadar: Artificial Intelligence Technologies, Q1 2017
A
I-OPTIMIZED CHIPS
Think
Learn
Sense
Act Continuous
iteration and
feedback
HUMAN RECOGNITION
Speech, face, and body
Sensors (e.g., temperature, chemical,
spectral, magnetic) and devices
MACHINE RECOGNITION
Knowledge
representation,
rules engines,
corporate data,
open data, and
external data
KNOW
Virtual agents
and natural
language
generation
INTERFACE
Machine learning platforms, deep
learning platforms, text analytics and
NLP, and image and video analysis
LEARN
Robotic process
automation and decision
management
AUTOMATION
Customer, partner,
employee, robot,
and device

Standard Automation Intelligent Automation
Systems
that do
Systems
that think
Systems
that learn
Robotic Process
Automation
Data Collection/
Data Preparation
Speech, Video &
Image Recognition
Predictive
APIs
Natural
Language
Processing
IT Process
Automation
Deep
Learning
Artificial
Intelligence
IoT & Smart
Devices
Emotional
Recognition
Cognitive
Computing
Machine
Learning
Autonomic
Computing
AI systems learn, think,
and automate doing

• Pattern recognition
• Machine perception
• Speech recognition
• Computer vision
• Affective computing
Overview of 
AI: Sense
7

Pattern recognition
8
PaGern recogniHon involves techniques to dis`nguish
signal from noise through sta`s`cal analyses, Bayesian
analysis, classificaòn, cluster analysis, and analysis of
texture and edges. Pacern recogniòn techniques
apply to sensors, data, imagery, sound, speech,
language.
Automated classificaHon tools dis`nguish,
characterize and categorize data based on a set of
observed features. For example, one might determine
whether a par`cular mushroom is “poisonous” or
“edible” based on its color, size, and gill size.
Classifiers can be trained automa`cally from a set of
examples through supervised learning. Classificaòn
rules discriminate between different contents of a
document or par`òns of a database based on various
acributes within the repository.
StaHsHcal learning techniques construct quan`ta`ve
models of an en`ty based on surface features drawn
from a large corpus of examples. In the domain of
natural language, for example, sta`s`cs of language
usage (e.g., word trigram frequencies) are compiled
from large collecòns of input documents and are
used to categorize or make predicòns about new
text.
Sta`s`cal techniques can have high precision within a
domain at the cost of generality across domains.
Systems trained through sta`s`cal learning do not
require human-engineered domain modeling.
However, they require access to large corpora of
examples and a retraining step for each new domain
of interest.
Source: Gary Larson
Source: Barbara Catania and Anna Maddalena

Machine perception
Enterprises are adopting biometrics based  
AI solutions to determine:
• Identity and authentication — Sensor processing
and biometrics for retinal scanning, fingerprint
scanning, facial recognition, voice recognition,
signature verification.
• Sentiment and emotion — sonic analytics, facial
expressions, text analytics, gesture analytics and
body language
• Veracity — biometric sensors (temperature,
moisture, heart rate, etc.), sonic analytics, facial
expressions, text analytics, gesture analytics and
body language.

• Cybersecurity is the body of technologies, processes and practices designed
to protect networks, computers, programs and data from attack, damage
or unauthorized access.
• User security is shifting from reliance on usernames, passwords and
security questions to incorporate biometric factors including voice
recognition, facial recognition, iris recognition, fingerprints and other
biometric data.
• Biometric security incorporates AI techniques for pattern recognition and
anomaly detection.
• Facial recognition technology is already a big business; it’s being used to
measure the effectiveness of store displays, spot cheaters in casinos, and
tailor digital ads to those passing by.
• Cognitive security analytics provide capabilities for predicting and assessing
threats, recommending best practices for system configuration, automating
defenses, and orchestrating resilient response.
AI machine perception
for user security is
incorporating biometric
factors.

Speech recognition
The ability to automatically
and accurately transcribe
human speech plus natural
language understanding
empowers individuals to
interact with enterprise
systems using voice
commands.
Natural language technology
processes queries, answers
questions, finds information,
and connects users with
various services to
accomplish tasks.
Speak, and
ye shall find
11

Evolution of speech technologies
12
Speaker
dependent
Automatic Speech Recognition
Natural Language Understanding
Text-to-Speech
ASR
NLU
TTS
Speaker
independent
Rules based
grammars
Statistical
grammars
Cloud
ASR
Deep neural
networks
Deep neural networksCloud NLUStatistical NLU
Format based
TTS
Concatenated
TTS
Rules based
TTS
Cloud
TTS
Statistical
TTS models
Deep neural
networks
Source: Nuance

Voice application
development
Text NL
Semantics
ASR
Language
Understanding
Context
Interpretation
Grammar and semantic tags
Voice application
developer

Computer vision
• The ability of computers to iden`fy objects, scenes,
and ac`vi`es in unconstrained (that is, naturalis`c)
visual environments.
• Computer vision has been transformed by the rise of
deep learning.
• The confluence of large-scale computing, especially
on GPUs, the availability of large datasets, especially
via the internet, and refinements of neural network
algorithms has led to dramatic improvements.
• Computers are able to perform some (narrowly
defined) visual classification tasks better than
people. A current research focus is automatic image
and video captioning.

Image annotation  
and captioning using 
deep learning
a man riding a motorcycle  
on a city street
a plate of food with 
meat and vegetables
15

Does My ai really
understand what
he feels and  
what he is  
saying to  
me?
Affective computing
• Detecting emotions from videos, audio, text,
facial expressions and gestures is a growth
market and important part of future cognitive
systems.
• Audio and video analytics for interpreting
sentiment, emotion and veracity

Six universal facial expressions*
* — Anger, happiness, surprise, fear, sadness, disgust
17

To analyze someone’s facial expressions, body temperature, etc. to determine
what that person is feeling, whether they are lying or not, what their gestures
and body language are, etc…
Who are some of the players and what are their top offerings?
18

What can body
temperature, heart
rate, and other
biometrics tell us?
19

Can you tell when
someone is lying
by reading their
facial expressions?
20

To analyze someone’s facial expressions, body temperature, etc. to determine
what that person is feeling, whether they are lying or not, what their gestures
and body language are, etc…
Who are some of the players and what are their top offerings?

Hyper real chatbots and assistants
• Customer service chatbots are about to become
very realistic. A startup gives chatbots and virtual
assistants realistic facial expressions and the
ability to read yours.
• Would your banking experience be more
satisfying if you could gaze into the eyes of the
bank’s customer service chatbot and know it sees
you frowning at your overdraft fees?
Soul Machines made this chatbot for the
Australian government to help people get
information about disability services.
22

Selected vendors* by category of machine perception analytics
AI Platforms
with APIs for  
Image & Text
• Apple (Emotient)
• Facebook
• Google
• IBM
• Microsoft
Facial
Analytics
• Affectiva
• Clarifai
• CrowdEmotion
• Eyeris/EmoVu
• Faciometrics
• Imotions
• Kairos
• Noldus
• nViso
• RealEyes
• Sightcorp/
Sonic
Analytics
• BeyondVerbal
• EMO Speech
• Nemesysco
• NICE
• Verint
• Vokaturi
Gesture
Analytics
• GRT—Gesture
Recognition
Toolkit
Text  
Analytics
• Clarabridge
• Crimson
Hexagon
• IBM Alchemy API
• Indico
• Receptiviti
Document 
Image Analytics
• Cvision
• Parascript
• Signotec
• Topaz Systems
23
* See sections 06 and 07 of this research deck.

• Machine learning
• Deep learning
• Natural language processing
• Knowledge representation
• Reasoning
• Cognitive computing
• What today’s AI technology can and cannot do
Overview of 
AI: Think

Machine learning
• Machine learning is a type of AI that involves using
computerized mathema`cal algorithms that can
learn from data and can depart from strictly
following rule-based, pre-programmed logic.
• Machine learning algorithms build a probabilis`c
model and then use it to make assumpòns and
predicòns about similar data sets
• Machine Learning runs at machine scale: it is data
driven and suited to the complexity of dealing with
disparate data sources and the huge variety of
variables and amounts of data involved.
• Unlike for tradiònal analysis, the more data fed to a
machine learning system, the more it can learn,
resul`ng in higher quality insights.
26© Copyright Project10x | Confidential

Machine learning can
help solve classification,
prediction, and
generation problems
Source McKinsey Global Institute

Machine learning has great
impact potential across
industries and use case types
Source McKinsey Global Institute

Types of machine learning

Machine learning overview
30

Types of machine learning and categories of algorithms
31
Type of machine learning
Target variable
Type of algorithm
Sample application

Clustering
32
Clustering is the process of organizing objects into groups whose
members are similar in some way.
Clustering is an approach to learning that seeks to place objects into
meaningful groups automa`cally based on their similarity. Document
clustering techniques iden`fy topics and group documents into
meaningful classes.
Clustering, unlike classificaòn does not require the categories to be
predefined with the hope that the algorithm will determine useful but
hidden groupings of data points. The hope in applying clustering
algorithms is that they will discover useful but unknown classes of items.

Outlier Detection
• Iden`fying excepòns or rare events can osen
lead to the discovery of unexpected knowledge.  
Outlier detecòn is used to iden`fy anomalous
situaòns.
• Anomalies may be hard-to-find needles in a
haystack, but may nonetheless represent high
value when they are found (or costs if they are
not found). Typical applicaòns include fraud
detecòn, iden`fying network intrusion, faults in
a manufacturing processes, clinical trials, vo`ng
ac`viès and criminal ac`viès in E-commerce.
• Applying machine learning to outlier detecòn
problems brings new insight and becer
detecòn of outlier events. Machine learning
can take into account many disparate sources of
data and find correlaòns that are too obscure
for human analysis to iden`fy.
• Take the example of credit card fraud: with
machine learning online behavior (web site
browsing history) of the purchaser becomes a
part of the fraud detecòn algorithm – rather
than simply considering the history of purchases
made by the card holder. This involves analyzing
huge amounts of data, but it also is a far more
robust approach to E-commerce fraud detecòn.

Machine learning flow — training and prediction
34

Machine learning:
• Supervised— Correct classes of the
training data are known.
• Unsupervised— Correct classes of the
training data are not known
• Reinforcement— Machine or software
agent learns behavior based on feedback
from the environment. This behavior can
be learned once and for all or continue to
adapt as time goes by.
35

Deep learning
A class of machine learning algorithms that:
• Use a cascade of many layers of nonlinear processing
units for feature extracòn and transformaòn.
Successive layer use the output from the previous
layer as input. Algorithms may be supervised or
unsupervised. Applicaòns include pacern analysis
(unsupervised) and classificaòn (supervised).
• Are based on (unsupervised) learning of mul`ple
levels of features or representaòns of the data.
Higher level features are derived from lower level
features to form a hierarchical representaòn.
• Learn mul`ple levels of representaòns that
correspond to different levels of abstracòn; the
levels form a hierarchy of concepts.

Types of Neural Networks
How do neural networks work?
Information flows through a neural network in
two ways. When it's learning (being trained) or
operating normally (after being trained), patterns
of information are fed into the network via the
input units, which trigger the layers of hidden
units, and these in turn arrive at the output units.
Training is the process of modifying the weights in
the connections between network layers so as to
achieve the expected output. This is achieved
through supervised learning, unsupervised
learning, and reinforcement learning.
Operations is the process of applying the
algorithm to make predictions. Result evaluation
feeds back to improve/optimize performance
37

Master algorithm — towards a synthesis of five approaches to machine learning
TRIBE ORIGINS PROBLEM REPRESENTATION EVALUATION OPTIMIZATION
MASTER
ALGORITHM
Symbolists
Logic,
philosophy
Knowledge
composition
Logic Accuracy
Inverse
deduction
Inductive logic
programming
Connectionists Neuroscience
Credit
assignment
Neural networks Squared error
Gradient
descent
Back
propagation
Evolutionaries
Evolutionary
biology
Structure
discovery
Genetic programs Fitness Genetic search
Genetic
programming
Bayesians Statistics Uncertainty Graphical models
Posterior
probability
Bayesian
optimization
Probabilistic
inference
Analogizers Psychology Similarity Support vectors Margin
Constrained
optimization
Kernel machines
Source: Pedro Domingos
38

Natural language processing

This research deck précis information
from the Forrester Digital
Transformation Conference in May
2017. It compiles selected copy and
visuals from conference presentations
and recent Forrester research reports.
Contents are organized into the
following sections:
• Digital transfor
Machine Learning
Human CommunicaHon
ArHficial Intelligence
Natural Language Processing:
NLP|NLU|NLG
Interacòn:
Dialog, gesture, 
emoòn, hap`c
Audible Language: 
Speech, sound
Visual Language: 
2D/3D/4D
Wricen Language: 
Verbal, text
Formal 
Language 
Processing
Symbolic Reasoning
Data
Deep Learning
40
AI for human communication
• Human communication encompasses every way that
people exchange ideas.
• Artificial intelligence is the theory and development of
intelligent machines and software that can sense, learn,
plan, act, understand and reason. AI performs tasks that
normally require human intelligence.
• Natural language processing (NLP) is the confluence of
artificial intelligence (AI) and linguistics.
- A key focus is the analysis, interpretation, and
generation of verbal and written language.
- Other language focus areas include audible & visual
language, data, and interaction.
• Formal programming languages enable computers to
process natural language and other types of data.
• Symbolic reasoning employs rules and logic to frame
arguments, make inferences, and draw conclusions.
• Machine learning (ML) is a area of AI and NLP that solves
problems using statistical techniques, large data sets and
probabilistic reasoning.
• Deep learning (DL) is a type of machine learning that uses
layered artificial neural networks.

nat·u·ral lan·guage proc·ess·ing
/ˈnaCH(ə)rəl//ˈlaNGɡwij//ˈpräˌsesˌiNG/
Natural language is spoken or wricen speech. English, Chinese, Spanish, and
Arabic are examples of natural language. A formal language such as
mathema`cs, symbolic logic, or a computer language isn't.
Natural language processing recognizes the sequence of words spoken by a
person or another computer, understands the syntax or grammar of the words
(i.e., does a syntac`cal analysis), and then extracts the meaning of the words.
Some meaning can be derived from a sequence of words taken out of context
(i.e., by seman`c analysis). Much more of the meaning depends on the context
in which the words are spoken (e.g., who spoke them, under what
circumstances, with what tone, and what else was said, par`cularly before the
words), which requires a pragma`c analysis to extract meaning in context.
Natural language technology processes queries, answers questions, finds
information, and connects users with various services to accomplish tasks.
What is natural
language processing?
NLP

Aoccdrnig to a rseearch taem at Cmabrigde
Uinervtisy, it deosn't mttaer in waht oredr the
ltteers in a wrod are, the olny iprmoatnt tihng is
taht the frist and lsat ltteer be in the rghit pclae.
The rset can be a taotl mses and you can sitll
raed it wouthit a porbelm. Tihs is bcuseae the
huamn mnid deos not raed ervey lteter by istlef,
but the wrod as a wlohe.
42

How natural language interpretation & natural language generation happens
43

Text analytics
44
Text mining is the discovery by computer of new, previously
unknown information, by automatically extracting it from
different written resources. A key element is the linking
together of the extracted information together to form new
facts or new hypotheses to be explored further by more
conventional means of experimentation.
Text analytics is the investigation of concepts, connections,
patterns, correlations, and trends discovered in written
sources. Text analytics examine linguistic structure and apply
statistical, semantic, and machine-learning techniques to
discern entities (names, dates, places, terms) and their
attributes as well as relationships, concepts, and even
sentiments. They extract these 'features' to databases or
semantic stores for further analysis, automate classiﬁcation
and processing of source documents, and exploit visualization
for exploratory analysis.
IM messages, email, call center logs, customer service survey
results, claims forms, corporate documents, blogs, message
boards, and websites are providing companies with enormous
quantities of unstructured data — data that is information-rich
but typically difﬁcult to get at in a usable way.
Text analytics goes beyond search to turn documents and
messages into data. It extends Business Intelligence (BI) and
data mining and brings analytical power to content
management. Together, these complementary technologies
have the potential to turn knowledge management into
knowledge analytics.

Speech I/O vs NLP vs NLU
NLP
NLU
syntactic
parsing
machine
translation
named entity
recognition (NER)
part-of-speech
tagging (POS)
semantic
parsing
relation
extraction
sentiment
analysis
coreference
resolution
dialogue
agents
paraphrase &
natural language
inference
text-to-
speech (TTS) summarization
automatic
speech
recognition (ASR)
text
categorization
question
answering (QA)
Speech I/O

Natural language understanding (NLU)
Natural language understanding (NLU) involves mapping a given
natural language input into useful representations, and analyzing
different aspects of the language.
NLU is critical to making making AI happen. But language is more
than words, and NLU involves more than lots of math to facilitate
search for matching words. Language understanding requires
dealing with ideas, allusions, inferences, with implicit but critical
connections to the ongoing goals and plans.
To develop models of NLU effectively, we must begin with limited
domains in which the range of knowledge needed is well enough
understood that natural language can be interpreted within the
right context.
One example is in mentoring in massively delivered educational
systems. If we want to have better educated students we need to
offer them hundreds of different experiences to choose from
instead of a mandated curriculum. A main obstacle to doing that
now is the lack of expert teachers.
We can build experiential learning based on simulations and
virtual reality enabling student to pursue their own interests and
eliminate the “one size fits all curriculum.”
To make this happen expertise must be captured and brought in to
guide from people at their time of need. A good teacher (and a
good parent) can do that, but they cannot always be available.
A kid in Kansas who wants to be an aerospace engineer should get
to try out designing airplanes. But a mentor would be needed. We
can build AI mentors in limited domains so it would be possible for
a student anywhere to learn to do anything because the AI mentor
would understand what a user was trying to accomplish within the
domain and perhaps is struggling with.
The student could ask questions and expect good answers tailored
to the student’s needs because the AI/NLU mentor would know
exactly what the students was trying to do because it has a perfect
model of the world in which the student was working, the relevant
expertise needed, and the mistakes students often make. NLU gets
much easier when there is deep domain knowledge available.
Source: Roger C Shank
46

Machine reading
& comprehension
AI machine learning is
being developed to
understand social
media, news trends,
stock prices and
trades, and other data
sources that might
impact enterprise
decisions.
47

Natural Language GeneraHon
Natural language generation (NLG) is the process of producing meaningful
phrases and sentences in the form of natural language from some internal
representation, and involves:
• Text planning − It includes retrieving the relevant content from knowledge
base.
• Sentence planning − It includes choosing required words, forming
meaningful phrases, setting tone of the sentence.
• Text realization − It is mapping sentence plan into sentence (or visualization)
structure, followed by text-to-speech processing and/or visualization
rendering.
• The output may be provided in any natural language, such as English, French,
Chinese or Tagalog, and may be combined with graphical elements to provide
a mul`modal presentaòn.
• For example, the log files of technical monitoring devices can be analyzed for
unexpected events and transformed into alert-driven messages; or numerical
`me-series data from hospital paènt monitors can be rendered as hand-over
reports describing trends and events for medical staff star`ng a new shis.
48

Knowledge representation  
(KR)
• Knowledge = theory + information
• Knowledge encoding via patterns
and language
• Spectrum of knowledge
representation and reasoning

This diagram reflects a philosopher's tradiònal picture and our acquired
definiòns of knowledge. The scope of knowledge is everything that has
ever been thought or ever can be.
On the right (in blue) are all observaòns and measurements of the
physical universe, the facts that characterize reality -- past, present, and
future. Within its bounds you find every object, every quantum of energy,
every `me and event perceived or perceivable by our senses and
instrumentaòn. This situaònal knowledge of physical reality is
“informaòn” in the sense of Shannon. It is a world of singular and most
“par`cular” things and facts upon which we might figura`vely scratch
some serial number or other iden`fying mark.
On the other side (in red) are all the “concepts” or ideas ever imagined --
by human's, by animals and plants, by Mickey Mouse, or a can of peas.
We can “imagine” a can of peas thinking. It embraces every mode of
visualizing and organizing and compelling the direcòn of our thoughts
from logic to religion to economics to poli`cs to every reason or raònale
for making dis`ncòns or for pu‚ng one thing before another.
What is knowledge?
“Knowledge is anything that resolves uncertainty. Knowledge is measured
mathematically by the amount of uncertainty removed. Knowledge bases
are defined by the questions they must answer.
Source: R.L. Ballard
50

As the next internet gains momentum, expect rapid progress towards a universal knowledge
technology that provides a full spectrum of informaòn, metadata, seman`c modeling, and
advanced reasoning capabiliès for any who want it.
Large knowledgebases, complex forms of situaòn assessment, sophis`cated reasoning with
uncertainty and values, and autonomic and autonomous system behavior exceed the capabiliès
and performance capacity of current descripòn logic-based approaches.
Universal knowledge technology will be based on a physical theory of knowledge that holds that
knowledge is anything that decreases uncertainty. The formula is:
Knowledge = Theory + InformaMon that reduces uncertainty.
Theories are the condiònal constraints that give meaning to concepts, ideas and thought
pacerns. Theory asserts answers to “how”, “why” and “what if” quesòns. For humans, theory is
learned through enculturaòn, educaòn, and life experience.
InformaHon, or data, provides situaòn awareness — who, what, when, where and how-much
facts of situaòns and circumstances. Informaòn requires theory to define its meaning and
purpose.
Theory persists and always represents the lion’s share of knowledge content — say 85%.
Informaòn represents a much smaller poròn of knowledge — perhaps only 15%
What will dis`nguish universal knowledge technology is enabling both machines and humans to
understand, combine, and reason with any form of knowledge, of any degree of complexity, at any
scale.
Knowledge = theory + informaòn
Knowledge = theory + information that reduces uncertainty
51

• Knowledge representation is the application of
theory, values, logic, and ontology to the task of
constructing computable patterns in some domain.
• Knowledge is “captured and preserved”, when it is
transformed into a perceptible and manipulable
system of representation. Systems of knowledge
representation differ in their fidelity, intuitiveness,
complexity, and rigor.
• The computational theory of knowledge predicts
that ultimate economies and efficiencies can be
achieved through variable-length, n-ary concept
coding and pattern reasoning resulting in designs
that are linear and proportional to knowledge
measure.
What is knowledge
representation?
52

Symbolic methods
• Declarative languages (Logic)
• Imperative languages  
C, C++, Java, etc.
• Hybrid languages (Prolog)
• Rules — theorem provers,
expert systems
• Frames — case-based
reasoning, model-based
reasoning
• Semantic networks, ontologies
• Facts, propositions
Symbolic methods can find
information by inference, can
explain answer
Non-Symbolic methods
• Neural networks — knowledge
encoded in the weights of the
neural network, for
embeddings, thought vectors
• Genetic algorithms
• graphical models — baysean
reasoning
• Support vectors
Neural KR is mainly about
perception, issue is lack of
common sense (there is a lot of
inference involved in everyday
human reasoning
Knowledge Representation 
and Reasoning
Knowledge representation
and reasoning is:
• What any agent—human,
animal, electronic,
mechanical—needs to
know to behave
intelligently
• What computational
mechanisms allow this
knowledge to be
manipulated?
53

Knowledge encoding
Natural language Documents, speech, stories
Visual language Tables, graphics, charts, maps,
illustrations, images
Formal language Models, schema, logic,
mathematics, professional and
scientific notations
Behavior language Software code, declarative
specifications, functions,
algorithms
Sensory language User experience, human-computer
interface, haptic, gestic.
Humans encode thoughts, represent knowledge, and share meanings using
pacerns and language.
PaNerns are knowledge units. A pacern is a compact and rich in seman`cs
representaòn of raw data. Seman`c richness is the knowledge a pacern
reveals that is hidden in the huge quan`ty of data it represents. Compactness
is the correlaòns among data and the synthe`c, high level descripòn of data
characteris`cs. For example, an image.
Language is a system of signs, symbols, gestures, and rules used in
communica`ng. Meaning is something that is conveyed or signified.
Humans have plenty of experience encoding thoughts and meanings using
language in one form or another… Our proficiency varies. We tend to be
becer at some kinds of language, and not so good at others.
Project teams osen combine different skills and exper`se, e.g. to make a
movie; design and construct a building; or coordinate response to an
emergency.
The table to the right gives examples of five forms of human language:
natural, visual, formal, behavioral, and sensory language.
54

Knowledge encoding: a key limitation of natural language is  
the inherent ambiguity resulting from overloaded symbol use.
"A noun is a sound that has meaning
only by convenòn. There is no natural
relaònship between any idea or
observaòn and the sound that you
ucer to describe it."
Aristotle — On InterpretaMon
55
Source: Gary Larson, The Far Side.

Knowledge encoding: visual language consists of words, images and
shapes, tightly integrated into communication units
56
Source: Robert Horn
Visual language is the `ght integraòn of words, images, and shapes to
produce a unified communicaòn. It is a tool for crea`ve problems
solving, problem analysis, and a way of conveying ideas and
communica`ng about the complexiès of our technology and social
ins`tuòns. Visual language can be displayed on different media and
different size communicaòn units. Visual language is being created by
the merger of vocabularies from many different fields as shown in the
diagram to the lower right.
As the world increases in complexity, as the speed at which we need to
solve business and social problems increases, as it becomes increasingly
cri`cal to have the “big picture” as well as mul`ple levels of detail
immediately accessible, visual language will become more prevalent in
our lives.
The internet of subjects, services and things will evolve seman`cally
enabled tools for visual language. Computers will cease being mere
electronic pencils, and be used to author, manage, and generate visual
language as a form of shared executable knowledge.

Theory is any condiònal or uncondiònal asseròn, axiom or constraint  
used for reasoning about the world. It may be any conjecture, opinion, or
speculaòn. In this usage, a theory is not necessarily based on facts and
may or may not be consistent with verifiable descripòns of reality.
We use theories to reason about the world. In this sense, theory is a set of
interrelated constructs — formulas and inference rules and a relaònal
model (a set of constants and a set of relaòns defined on the set of
constants).
"The ontology of a theory consists in the objects theory assumes  
there to be."  
-- Quine -- Word and Object, 1960
Theories are accepted or rejected as a whole. If we choose to accept and
use a theory for reasoning, then we must commit to all the ideas and
relaònships the theory needs to establish its existence.
In science, theory is a proposed raònal descripòn, explanaòn, or model
of the manner of interacòn of a set of natural phenomena.
Scien`fic theory should be capable of predic`ng future occurrences or
observaòns of the same kind, and capable of being tested through
experiment or otherwise falsified through empirical observaòn.
Values for theory construcòn include that theory should: add to our
understanding of observed phenomena by explaining them in the simplest
form possible (parsimony); fit cleanly with observed facts and with
established principles; be inherently testable and verifiable; and imply
further inves`gaòns and predict new discoveries.
Theory
Claude Shannon
57
Turing Science Museum

Structured, semi-structured, and unstructured are types of data representaòns
that seman`c technologies unify.
Structured informaHon is informaòn that is understandable by computers. Data
structures (or data models) include: relaMonal — tabular formats for data are
most prevalent in database systems, and operate best for storage and
persistence; hierarchical — tree-like formats (including XML) are most prevalent
in document models, and operate best in messaging systems (including SOA);
and object — frame systems like Java and C# combine behavior with data
encapsulaòn, and operate best for compiled sosware programs.
Semi-structured informaHon is data that may be irregular or incomplete and
have a structure that changes rapidly or unpredictably. The schema (or plan of
informaòn contents) is discovered by parsing the data, rather than imposed by
the data model, e.g. XML markup of a document.
Unstructured informaHon is not readily understandable by machines. Its sense
must be discovered and inferred from the implicit structure imposed by rules and
convenòns in language use, e.g. e-mails, lecers, news ar`cles.
Data representaòns
Examples of data models.
58

The fundamental shis in the connected intelligence era is from
informaòn-centric to knowledge-centric compu`ng that integrates
four innovaòn dimensions depicted here:
• Intelligent user experience — concerns how I experience things,
demands on my acenòn, my personal values. Trend towards
exploi`ng higher bandwidth content dimensionality, context
sensi`vity, and reasoning power in the user interface.
• SemanMc social compuMng — concerns our lived culture,
intersubjec`ve shared values, & how we communicate. Trend
towards collabora`ve tooling that empowers we humans (and our
computers) to co-develop, share, and exploit knowledge in all its
forms (e.g., content, models, services, and behaviors).
• CogniMve applicaMons, and things — concerns objec`ve things
such as product structure & behavior viewed empirically. Trend
towards hi-bandwidth, intelligent, autonomic, autopoie`c, and
autonomously communica`ng digital products, services, and
intellectual property.
• CogniMve infrastructure — concerns interobjec`ve network-
centric systems and ecosystems. Trend towards, everything self-
aware, somewhat intelligent, connected and socially autopoie`c,
and capable of solving problems of complexity, scale, security,
trust, and change management.
Integra`ng knowledge across different domains.
Source: Ken Wilber, Integral Institute & Mills Davis, Project10X
Concept Computing
Information
Technology
1st Person
(I)
Subjective
2nd Person
(WE)
Social 4th Person
(ITS)
Systemic
3rd Person
(IT)
Objective
Individual
I
n
t
e
r
i
o
r
E
x
t
e
r
i
o
r
Collective
59

Knowledge representation & reasoning: 
from search to knowing
More expressive knowledge
representation (vertical axis)
enables more powerful
reasoning (horizontal axis):
• Representations from lists
to dictionaries, glossaries and
lexicons; to taxonomies; to
thesauri; to models; to
semantic data and models; to
ontologies
• Reasoning from recovery, to
discovery, to intelligence, to
question answering, to smart
behaviors.

Dicònaries, glossaries and lexicons
61
DicHonaries are alphabe`cal lists of terms and their definiòns that
provide variant senses for each term, where applicable. They are more
general in scope than a glossary. They may also provide informaòn about
the origin of the term, variants (both by spelling and morphology), and
mul`ple meanings across disciplines. While a dicònary may also provide
synonyms and through the definiòns, related terms, there is no explicit
hierarchical structure or acempt to group terms by concept.
A gazeNeer is a dicònary of place names. Tradiònal gazeceers have been
published as books or they appear as indexes to atlases. Each entry may
also be iden`fied by feature type, such as river, city, or school. Geospaàlly
referenced gazeceers provide coordinates for loca`ng the place on the
earth’s surface.
A glossary is a list of terms, usually with definiòns. The terms may be from
a specific subject field or those used in a par`cular work. The terms are
defined within that specific environment and rarely have variant meanings
provided.
A lexicon is a knowledge base about some subset of words in the
vocabulary of a natural language. One component of a lexicon is a
terminological ontology whose concept types represent the word senses in
the lexicon. The lexicon may also contain addiònal informaòn about the
syntax, spelling, pronunciaòn, and usage of the words. Besides
convenònal dicònaries, lexicons include large collecòns of words and
word senses, such as WordNet from Princeton University and EDR from the
Japan Electronic Dicònary Research Ins`tute, Ltd. Other examples include
classificaòn schemes, such as the Library of Congress subject headings or
the Medical Subject Headers (MeSH).

Taxonomy
62
A taxonomy is a hierarchical or associa`ve
ordering of terms represen`ng categories.
A taxonomy takes the form of a tree or a
graph in the mathema`cal sense.
A taxonomy typically has minimal nodes,
represen`ng lowest or most speciﬁc
categories in which no sub-categories are
included as well as a top-most or maximal
node or la‚ce, represen`ng the maximum
or general category.
Source: Denise Bedford, World Bank
Examples of taxonomy.

Folk taxonomy
63
A folk taxonomy is a category hierarchy with 5-6 levels
that has its most cogni`vely basic categories in the
middle.
In folk taxonomies, categories are not merely organized
in a hierarchy from the most general to the most specific,
but are also organized so that the categories that are
most cogni`vely basic are “in the middle” of a general-
to-specific hierarchy. Generalizaòn proceeds upward
from the basic level and specializaòn proceeds down.
A basic level category is somewhere in the middle of a
hierarchy and is cogni`vely basic. It is the level that is
learned earliest. Usually has a short name and is used
frequently. It is the highest level at which a single mental
image can reflect the category. Also, there is no defini`ve
basic level for a hierarchy – it is dependent on the
audience. Most of our knowledge is organized around
basic level categories.
Source: George Lakoff

Thesaurus
64
A thesaurus is a compendium of synonyms and related
terms. It organizes knowledge based on concepts and
relaònships between terms.
Relaònships commonly expressed in a thesaurus include
hierarchy, equivalence, and associa`ve (or related). These
relaònships are generally represented by the notaòn BT
(broader term), NT (narrower term), SY (synonym), and RT
(associa`ve or related). Associa`ve relaònships may be
more granular in some schemes.
For example, the Unified Medical Language System (UMLS)
from the Naònal Library of Medicine has defined over 40
relaònships across more than 80 vocabularies, many of
which are associa`ve in nature. Preferred terms for
indexing and retrieval are iden`fied. Entry terms (or non-
preferred terms) point to the preferred terms that are to
be used for each concept.

Model
65
A model is a representaòn of an actual or
conceptual system. It involves mathema`cs, logical
expressions, or computer simulaòns that can be
used to predict how the system might perform or
survive under various condiòns or in a range of
hos`le environments.
A simulaHon is a method for implemen`ng a
model. It is the process of conduc`ng experiments
with a model for the purpose of understanding the
behavior of the system modeled under selected
condiòns or of evalua`ng various strategies for
the operaòn of the system within the limits
imposed by developmental or operaònal criteria.
Simulaòn may include the use of analog or digital
devices, laboratory models, or “testbed” sites.
Examples of models.

Seman`c graph
67
Seman`c networks are ontologies. They are like and unlike other IT models.
Like databases, ontologies are used by applicaòns at run `me (queried and
reasoned over). Unlike convenònal databases, relaònships are first-class
constructs.
Like object models, ontologies describe classes and acributes (properès).
Unlike object models, ontologies are set-based and dynamic.
Like business rules, seman`c models encode event-based behaviors. Unlike
business rules, ontologies organize rules using axioms.
Like XML schemas, they are na`ve to the web (and are in fact serialized in
XML). Unlike XML schemas, ontologies are graphs not trees, and used for
reasoning.
People some`mes refer to ontologies as the“O” word, thinking that
knowledge models are abstract and scary. Actually, seman`cs is something
that every human being already knows very well. We’ve been figuring out
what things mean all our lives. Don’t let the notaòn fool you. Any ontology
can be expressed clearly in plain English (or other natural language of your
choosing).

Ontology
68
An ontology is a formal explicit specification of a shared conceptualization.
An ontology defines the terms and axioms used to describe, represent, and
reason about an area of knowledge (subject matter). It is the model (set of
concepts) for the meaning of those terms. It defines the vocabulary and the
meaning of that vocabulary as well as the assertions, rules, and constraints
used in reasoning about this subject matter. An ontology is used by people,
databases, and applications that need to share domain information.
A domain is a specific subject area or area of knowledge, like medicine, tool
manufacturing, real estate, automobile repair, financial management, etc.
Ontologies include computer-usable definitions of basic concepts in the domain
and the relationships among them. They encode domain knowledge (modular).
Knowledge that spans domains (composable). They make knowledge available
(reusable).
Ontologies are usually expressed in a logic-based language that enables
detailed, sound, meaningful distinctions to be made among the classes,
properties, & relations as well as inferencing across the knowledge model.
Source: Leo Obrst
Source: Tom Gruber
Source: Andreas Schmidt
The diagram above shows that shared ideas and knowledge can
be expressed with different degrees of formality.

REASONING

• Reasoning is the derivation of inferences and
the warranting of conclusions through
application of heuristics, rules, analogies,
mathematics, logic, and values.
• Reasoning requires knowledge representation.
We choose more powerful forms of
representation to enable more powerful kinds
of reasoning and problem solving.
• A broad range of reasoning capabilities exist
including pattern detection and machine
learning; deep linguistics; ontology and model
based inferencing; and reasoning with
uncertainties, conflicts, causality, analogies,
and values.
What is reasoning?
70

Continuum of machine
reasoning and decision
making
Artificial intelligence
describes software that
dynamically choses the
optimal combination of
methods for the solution
of a problem, often at
very short temporal
scales.

▪ Statistics is the study of the collection, analysis, interpretation,
presentation, and organization of data.
▪ Two main statistical methodologies are used in data analysis: descriptive
statistics, which summarizes data from a sample using indexes such as
the mean or standard deviation, and inferential statistics, which draws
conclusions from data that are subject to random variation (e.g.,
observational errors, sampling variation).
▪ Descriptive statistics are most often concerned with two sets of properties
of a distribution (sample or population): central tendency (or location)
seeks to characterize the distribution's central or typical value, while
dispersion (or variability) characterizes the extent to which members of the
distribution depart from its center and each other.
▪ Inferences on mathematical statistics are made under the framework of
probability theory, which deals with the analysis of random phenomena.
72
Statistical inference
“He told me I was average. 
I told him he was mean.”

Similarity Search
• Similarity search provides a way to find the objects that are the
most similar, in an overall sense, to the object(s) of interest.
• A typical example is that of a doctor finding the top 10 past paènts
who are most similar to the current paènt of interest. This could
be used for diagnosis, but also adds the human judgement that
some other machine learning methods do not necessarily offer.
Another example of approximate similarity search is for finding the
song in a database corresponding to a given sound sample, or
finding the person in a database corresponding to a face photo.
• Similarity searches can be thought of as mul`dimensional analogs
to SQL queries. SQL queries are composed of condiòns on
individual variables, for example “Find all customers whose age is
within a certain range and whose income is greater than a certain
amount”, whereas similarity searches are more like “Find all the
customers most like this one”.
73

Analy`cs
74
AnalyHcs studies the cons`tuent parts of something and their relaòn to the
whole.
Analy`cs seeks to iden`fy and interpret pacerns in informaòn derived from
mul`ple sources. Seman`c technologies provide the capability to dynamically
map together heterogeneous data sets, bridging silos and making
interrelaònships explicit and computable.
The chart to the right maps different types of analy`cs by purpose and `me
horizon. The purpose axis (ver`cal) dis`nguishes "exploraòn vs. control" and
highlights the difference between analysis and repor`ng. Analysis is about
digging deep into data to discover relaònships, find causaòn, and describe
phenomena. Repor`ng, in contrast, is used to track performance and iden`fy
variaòn from goals. The temporal axis (horizontal) dis`nguishes “backward
looking vs. forward looking vs. real-`me). Most analy`cs is backward looking
— in an acempt to understand what has happened, and therefore be
equipped to make becer decisions in the future. Alterna`vely, analy`cs can
focus explicitly on predic`ng future performance. Increasingly, however, the
requirement is to provide informaòn and insight to support operaònal
decisions in real-`me.
Source: Zach Gemignani

Predictive Analytics
• PredicHve analyHcs is the science of analyzing current
and historical facts/data to make predicòns about future
events.
• Unlike tradiònal business intelligence prac`ces, which
are more backward-looking in nature, predic`ve analy`cs
is focused on helping companies derive acònable
intelligence based on past experience.
• A typical applicaòn is in insurance: predic`ng which
policy holders (or potenàl policy holders) will make a
claim and how long it will be un`l they make the claim.
The more data available on the history of claims and
‘extraneous’ informaòn about the policy holder the
more variables a predic`ve analy`cs algorithm can take
in to account.
• For example, a machine learning algorithm could easily
take into account the impact of when a parent has
children on claim rates. Iden`fying if such a relaònship
exists (amongst ALL the other possibiliès) is too complex
for human analysts.
• Another example is a health insurance company using
predic`ve analy`cs to iden`fy when paènts are likely to
have a hospital stay – and to direct health care providers
to take preventa`ve acòns to avoid the hospital stay.
With a growing base of health care data, this sort of data
science is set to improve the nature of health care
delivery.
• Other examples include predicòn of product demand,
opòns prices, or turnover likelihood of sales leads.
75

• Predic`ve analy`cs is the science of analyzing current and historical facts/data to make predicòns about future events.
• Unlike tradiònal business intelligence prac`ces, which are more backward-looking in nature, predic`ve analy`cs is focused on helping companies
derive acònable intelligence based on past experience.
• A typical applicaòn is in insurance: predic`ng which policy holders (or potenàl policy holders) will make a claim and how long it will be un`l they
make the claim. The more data available on the history of claims and ‘extraneous’ informaòn about the policy holder the more variables a
predic`ve analy`cs algorithm can take in to account.
• For example, a machine learning algorithm could easily take into account the impact of when a parent has children on claim rates. Iden`fying if such
a relaònship exists (amongst ALL the other possibiliès) is too complex for human analysts.
• Another example is a health insurance company using predic`ve analy`cs to iden`fy when paènts are likely to have a hospital stay – and to direct
health care providers to take preventa`ve acòns to avoid the hospital stay. With a growing base of health care data, this sort of data science is set
to improve the nature of health care delivery.
• Other examples include predicòn of product demand, opòns prices, or turnover likelihood of sales leads.

Four kinds of reasoning
77
The four methods of reasoning include:
(1) DeducHon: deriving implicaòns from premises.
(2) InducHon: deriving general principles from examples.
(3) AbducHon: Forming a hypothesis that must be tested by inducòn and deducòn.
It involves inferring the best or most plausible explanaòn from a given set of facts or
data.
(4) Analogy: Besides these three types of reasoning there is a fourth, analogy, which
combines the characters of the three, yet cannot be adequately represented as
composite. Analogy is more primi`ve, but more flexible than logic. The methods of
logic are disciplined ways of using analogy.
Although deducòn is important, it is only one of the four methods of reasoning.
Inducòn, abducòn, and analogy are at least as important, and they are necessary
for learning or acquiring new knowledge. Current computer systems come close to
human ability in deducòn. But they are far inferior in learning, which depends
heavily on the other three methods of reasoning.
Source: John Sowa

Evidence-based reasoning
• This evidence-based reasoning framework shows how a premise and data are
processed through three distinct steps (analysis, interpretation, and application)
to produce a claim as the output.
• A claim is a statement about a specific outcome or state phrased as either a
prediction of what something will do in the future (e.g., “This box will sink”), an
observation of what something has done in the past (e.g., “This box sank”), or a
conclusion about what something is in the present (e.g., “This box sinks”).
• The premise consists of one or more statements describing the specific
circumstances acting as an input that will result in the outcome described by the
claim.
• Rules link the premise and the claim, asserting a general relationship that
justifies how the latter follows from the former. Application is the process by
which the rule are brought to bear in the specific circumstances described by the
premise.
• Evidence consist of statements describing observed relationships. Interpretation
of evidence is a process of generalization, grounded in a specific context.
• Data are discrete reports of past or present observations, and are collected and
related by analysis to produce a statement of evidence.
78
Source: EBR Framework: Assessing Scientific Reasoning, 
Taylor & Francis Group

Axiology, logic & probability
• Value is the foundaòn of meaning. It is the measure of the
worth or desirability (posi`ve or nega`ve) of something, and of
how well something conforms to its concept or intension.
• Value formaòn and value-based reasoning are fundamental to
all areas of human endeavor. Theories embody values. The
axiom of value is based on “concept fulfillment.”
• Most areas of human reasoning require applicaòn of mul`ple
theories; resoluòn of conflicts, uncertainès, compe`ng
values; and analysis of trade-offs. For example, quesòns of guilt
or innocence require judgment of far more than logical truth or
falsity.
• Axiology is integral to the evoluòn of AI. Axiology is the branch
of philosophy that studies value and value theory. Things like
honesty, truthfulness, objec`veness, novelty, originality,
“progress,” people sa`sfacòn, etc. The word ‘axiology’, derived
from two Greek roots 'axios’ (worth or value) and ‘logos’ (logic
or theory), means the theory of value, and concerns the process
of understanding values and valuaòn.

• Most predictive analysis today is done with machine
learning and statistical methods, so using this alone is not
novel.
• Semantic reasoning can be used alongside to guide and
validate machine learning methods, catch outliers, and
explain how and why predictions were made.
• A large class of problems exist where hybrid AI
approaches can be applied to improve outcomes
provided that means are available to:
- Standardize access to multiple AI engines and enable
collaboration between them to answer the same query
- Compare and combine the results to improve accuracy
- Explain the “why” of the results and recommend ways
to improve them with different methods and data
Neural-symbolic fusion combines statistical
correlation and semantic reasoning to deliver
unique insight.
People Who Will
People Who Will Not
People Predicted By
Semantic Reasoning
People Predicted By
Statistical Correlation
MAXIMIZE
Combining statistical and symbolic reasoning
80

Yann LeCun 
Director, AI Research 
Facebook
“Predictive learning is the next frontier for AI”
Deep learning has been at the root of significant progress in many
application areas, such as computer perception and natural language
processing. But almost all of these systems currently use supervised
learning with human-curated labels.
The challenge of the next several years is to enable machines to learn by
themselves about any domain from raw, unlabeled data, such as images,
videos and text.
The problem is that intelligent systems today do not possess "common
sense", which humans and animals acquire by observing the world, acting in
it, and understanding the physical constraints of it.
Enabling machines to learn predictive models of the world where
predictability is only partial, is key to making significant progress in artificial
intelligence, and a necessary component of model-based planning and
reinforcement learning. Predictive learning is the next frontier for AI.

/ˈk.ɡnədiv//kəmˈpyo͞odiNG/
noun
Cognition is the mental action or process of acquiring knowledge and
understanding through thought, experience, and the senses.
Cognitive computing is the simulation of human thought processes in a
computerized model. It involves self-learning systems that use data
mining, pattern recognition, natural language processing, and statistical
and symbolic reasoning to mimic the way that humans think and learn.
cog.ni.tive com.put.ing
What is  
cognitive computing?

This diagram maps
cognitive technologies by
how autonomously they
work, and the tasks they
perform.
It shows the current state
of smart machines—and
anticipates how future
technologies might
unfold.
SPRING 2016 MIT SLOAN MANAGEMENT REVIEW
WHAT TODAY’S COGNITIVE TECHNOLOGIES CAN — AND CAN’T — DO
Mapping cognitive technologies by how autonomously they work and the tasks they perform shows the current
state of smart machines — and anticipates how future technologies might unfold.
LEVELS OF INTELLIGENCE
TASK
TYPE
SUPPORT FOR
HUMANS
REPETITIVE TASK
AUTOMATION
CONTEXT
AWARENESS
AND LEARNING SELF-AWARENESS
T
G
C
Analyze
Numbers
Business intelligence,
data visualization,
hypothesis-driven
analytics
Operational analytics,
scoring, model
management
Machine learning,
neural networks
Not yet
Analyze
Words
and
Images
Character and
speech recognition
Image recognition,
machine vision
IBM Watson, natural
language processing
Not yet
Perform
Digital
Tasks
Business process
management
Rules engines, robotic
process automation
Not yet Not yet
Perform
Physical
Tasks
Remote operation
of equipment
Industrial robotics,
collaborative robotics
Autonomous robots,
vehicles
Not yet
Source: MIT Sloan Management Review, Spring 2016
What today’s cognitive technologies can and cannot do
83

• Building blocks and levels of intelligent action
• Levels of intelligent action
• Search and question answering
• Rules engines
• Expert systems
• Recommender systems
• Automated planning and scheduling systems
• Robotic process automation
• Autonomic computing
• Autonomous systems
Overview of 
AI: Act

Four building blocks for intelligent action
Source: HfS
Software development toolkit
allows non-engineers quickly
to create software robots to
automate rules-driven
business processes.
E.g. digitize process of
collecting of unpaid invoices,
mimicking manual activities in
the RPA software, the
integration of electronic
documents and generation of
automated emails to ensure
the whole collections, process
is run digitally and can be
repeated in a hi-throughput,
high intensity model.
Simulating human thought
in a set of processes. It
involves self-learning
systems, data mining,
pattern recognition, and
natural language processing
to mimic the way the
human brain works.
E.g. an insurance
adjudication system that
assesses claims, based on
scanned documents and
available data from similar
claims and evaluates
payment awards.
Self-learning and self-
remediating engines.
System makes autonomous
decisions, using hi-level
policies. Constantly
monitors, adapts and
optimizes its performance
as conditions change and
business rules evolve.
E.g. a virtual support agent
continuously learning to
handle queries and creating
new rules/exceptions as
products evolves and
queries change.
Intelligent automation systems
go beyond routine business
and IT process activity to
make decisions and
orchestrate processes.
E.g. an AI system managing a
fleet of self-driving cars or
drones to deliver goods to
clients, manage aftermarket
warranties and continuously
improve the supply chain.
Artificial IntelligenceRobotic Process Automation AutonomicsCognitive Computing
86

What intelligent systems need to possess
Source: GRAKN.AI 87

Four levels of intelligent action
88
FIXED
SEMANTIC
AUTONOMIC
AUTONOMOUS
• Fixed interfaces
• Hard-wired design `me stack
• Black boxes
• H2M interoperability
VALUE
KNOWLEDGE INTENSIVITYLow Hi
LowHi
•Model-based, seman`c APIs, dynamic
interfaces
•Self-declaring, self-defining components
•Glass boxes
•M2M integraòn and interoperability
•Self* (awareness, provisioning, configuring, diagnosing)
•Pervasive adap`vity (sense, interpret, respond)
•Mobile dynamics, granular security,
•M2M performance op`mizaòn
• Goal-oriented,  
• Mul`-domain knowledge
• Cogni`ve automaòn
• Mul`-agent
• M2M & predic`ve learning

What makes intelligent systems different?
Goal-orientaòn
Constraint-based  
models
Event-driven 
inferencing
Context 
awareness
Adap`vity
Self-op`mizaòn
89

Search
90

Search engines
91
Search engines look for relevant informaòn based
on some criteria.
Full-text search is fast, efficient, and simple, but
delivers poor relevance in the absence of an exact
keyword match.
StaMsMcal search mechanisms focus on the
frequency of keywords, but provide imperfect
results: a keyword may be misspelled in some
target documents; it may appear in a plural or
conjugated form; it may be replaced by a
synonym; it may have different meanings
according to context. Osen, sta`s`cally-based
searches return results that prove either too
voluminous or too restricted to be helpful.
Natural language search uses linguis`c analysis,
rules, and reference knowledge to improve named
en`ty extracòn, seman`c analysis of word
senses, and meaning of texts.
SemanMc search expands keyword search by
understanding the meaning of concepts and
context of the query. It exploits knowledge about
the context of the quesòn. It looks at the
meaning of full sentences and documents as well
as equivalent ways of saying the same thing. It
recognizes the gramma`cal role played by words
in a sentence (e.g., subject or object), detects the
relaònship between the parts of a sentence
(objects, subjects, verbs, acributes, etc.). It
exploits reference knowledge about relaònships
between concepts. Seman`c search can be cross-
lingual (queries in the user language, answers in all
languages).
Search technologies Definition Sample vendors
Boolean
(extended Boole a n )
Retrieves documents based on
the number of times the
keywords appear in the text .
Virtually all search
engine s
Clusteri ng Dynamically creates “clusters”
of documents grouped by
similarity, usually based on a
statistical analysis .
Autonomy,
GammaSite,
Vivisim o
Linguistic analysis
(stemming,
morphology, synonym-
handling, spell-
checki n g )
Dissects words using
grammatical rules and
statistics.
Finds roots, alternate tenses,
equivalent terms and likely
misspellings.
Virtually all search
engines
Natural language
processing
(named entity
extraction, semantic
analysi s )
Uses grammatical rules to find
and understand words in a
particular category. More
advanced approaches classify
words by parts of speech to
interpret their meaning.
Albert, Inxight,
InQuira
Ontology
(knowledge
representatio n )
Formally describes the terms,
concepts and interrelationships
in a particular subject area.
Endeca, InQuira,
iPhrase, Verity
Probabilistic
(belief networks,
inference networks,
Naive Baye s )
Calculates the likelihood that
the terms in a document refer
to the same concept as the
query.
Autonomy,
Recommind,
Microsoft
Taxonomy
(categorization)
Establishes the hierarchical
relationships between concepts
and terms in a particular search
area.
GammaSite, H5
Technologies,
YellowBrix
Vector-based
(vector support
machine)
Represents documents and
queries as arrows on a
multidimensional graph— and
determines relevance based on
their physical proximity in that
space.
Convera, Google,
Verity
Source: Forrester Research

Search engine parts
This diagram looks under the hood of a
search engine to iden`fy the component
parts: search inputs, query matching
algorithms, and types of search outputs
that the user sees.

Question answering
Quesòn answering (QA) is more than search, more than
discovery, and more than document retrieval.
The first step is to analyze the natural language quesòn to
determine the informaòn needed and the form that the
answer should take, e.g. a factoid for “who discovered
oxygen?”, a list for “what countries export oil?”, a
definiòn for “what is a quasar?”.
Quesòn analysis involves both linguis`c and seman`c
processing, and results in targeted informaòn retrieval
queries to search engines to return documents that are
likely to contain elements of the answer sought.
The next step is to extract and aggregate informaòn
passages from these sources, then to reason about the
content in order to extract or synthesize the answer, and to
present it to user.
Machine
Learning
Question
Analysis
Feature
Engineering
Ontology
Analysis
Question
& Answer
Natural
Language
Processing
Question answering involves analyzing questions,
retrieving documents, retrieving passages, and
extracting answers.

Rules engine
• A rules engine is a sosware
system that executes one or
more business rules in a run`me
producòn environment.
• A business rule system enables
company policies and other
operaònal decisions to be
defined, tested, executed and
maintained separately from
applicaòn code.
• Rules-based systems use
databases of knowledge and if-
then rules to automate the
process of making inferences
about informaòn.
94

Expert systems
KNOWLEDGE BASE
Knowledge Models
Smart DataRules & Theory
Machine Learning
& Data Analytics
Knowledge
Authoring
& Curation
Question AnsweringKnowledge Management Virtual Assistance
INFERENCE 
ENGINE
Decisions
Actions
Explanations
Reporting
Self-Documentation
SMART USER INTERFACE • An expert system (ES) employs knowledge about its
application domain and uses inferencing (reasoning)
procedures to solve problems that require human
competence or expertise.
• An expert system contains three subsystems: an inference
engine, a knowledge base, and a user interface.
• The expert system reasons with the knowledge base. Its
reasoning engine interprets directions, answers
questions, and executes commands that result in decisions,
actions, reporting, explanations, and self-documentation.
• Expert systems assist and augment human decision
makers. Application areas include classification, diagnosis,
monitoring, process control, design, scheduling and
planning, and generation of options.
95

Recommender system
• Recommend — to put forward (someone or
something) with approval as being suitable for a
par`cular purpose or role.
• RecommendaHon engines automate the process of
making real-`me recommendaòns to customers.
• A simple example: an online customer who is browsing
a store for one item (e.g. a power drill), places the
item in their shopping cart, and is then recommended
to buy a complementary item (e.g., a set of drill bits).
This example is trivial. Machine learning can go
further, osen uncovering unexpected buying pacerns,
based on unforeseen relaònships between different
customers and between different products.
• Recommender systems take into account where on
the site the customer had visited, their history of
purchases at the site and even their social network
history. It may be that the customer browsed for
mortar on the last visit to the site. Perhaps the user
also asked friends about selec`ng bathroom `les on
Facebook. In this case it might make sense to
recommend a mortar mixing acachment – since it is
clear the customer is doing a `ling project. For a
machine learning algorithm, iden`fying non-explicit
relaònships like this is typical.
• A machine learning recommender system improves
with `me. It learns from successful, and unsuccessful
recommendaòns. The same underlying technology
can be used to provide customers with many other
kinds of personalized experiences, based on data of
many kinds.
96

Automated planning  
and scheduling system
AI systems that devise
strategies and sequences of
actions to meet goals and
observe constraints, typically
for execution by intelligent
agents, autonomous robots
and unmanned vehicles.
97

Robotic process
automation (RPA)
Captures and interprets
existing means for
conducting a task,
processing a transaction,
manipulating data,
triggering responses, and
communicating with other
systems. This may include
manual, repetitive tasks,
intelligent automation of
processes, and
augmentation of resources.
98

Robotics
• A robot is a programmable mechanical or software
device that can perform tasks and interact with its
environment, without the aid of human interaction.
• Robotics is embracing cognitive technologies to create
robots that can work alongside, interact with, assist, or
entertain people. Such robots can perform many
different tasks in unpredictable environments,
integrating cognitive technologies such as computer
vision and automated planning with tiny, high-
performance sensors, actuators, and hardware. Current
development efforts focus how to train robots to interact
with the world in generalizable and predictable ways.
• Deep learning is being used in robotics. Advances in
machine perception, including computer vision, force,
and tactile perception are key enablers to advancing the
capabilities of robotics. Reinforcement learning helps
obviate the need for large labeled data sets.
99

au·to·ma·tion
/ˌôdəˈmāSH(ə)n/
The use of software and equipment in a system or production process so
that it works largely by itself with little or no direct human control.
Robotic process automation and intelligent automation are the
combination of AI and automation.What is automation?

• “Automation” today can be defined as including any functional activity
that was previously performed manually and is now handled via
technology platforms or process automation tools like robotic process
automation (RPA) platforms.
• With increasing computer processing power, technology has reached a
point where its ability to perform human-like tasks has become possible.
• There are various names for referring to robotics in service industries
such as Rapid Automation (RA), Autonomics, Robotic Process
Automation, software bots, Intelligent Process Automation or even plain
Artificial Intelligence.
• These terms refer to the same concept: letting organizations automate
current tasks as if a real person was doing them across applications and
systems.
• A primary opportunity for robotic process automation in the enterprise is
to augment the creative problem-solving capabilities and productivity of
human beings and deliver superior business results.
Automation: letting
organizations automate
current tasks as if a real
person was doing them
across applications and
systems.

Source: HfS - 2016
Evolving landscape of
service agents and
intelligent automation:
• From desktop automation
to RPA, to chatbot, to
assistant, to virtual agent.
• From enhancement of
data, to augmentation of
human agents, to
substitution of digital
labor for the human agent.
Example
vendors:
102

Robotic Desktop
Automation (RDA)
• Personal robots for
every employee
• Call center, retail, branches,
back office
• 20-50% improvement across
large workforce groups
• RDA also provides dashboards
and UI enhancements
Robotic Process
Automation (RPA)
• Unattended robots replicating
100% of work
• Back office, operations,
repetitive
• 100% improvement across
smaller sub-groups
• Runs on a virtual server farm
(or under your desk)
Comparing robotic
desktop automation
(RDA) and robotic
process automation
(RPA)

• Robotic process automation gives humans the potential of attaining new
levels of process efficiency, such as improved operational cost, speed,
accuracy and throughput volume, and leaving behind the repetitive and time
consuming low added-value tasks.
• Top drivers for implementing robotic automation beyond cost savings include:
- High quality by a reduction of error rates
- Time savings via better management of repeatable tasks
- Scalability by improving standardization of process workflow
- Integration by reducing the reliance on multiple systems/screens to
complete a process
- Reducing friction (increasing straight-through processing)
• For example, back-office tasks do not require direct interaction with
customers and can be performed more efficiently and effectively off-site or by
robots. It is feasible to re-engineer hundreds of business processes with
software robots that are configured to capture and interpret information
from systems, recognize patterns, and run business processes across multiple
applications to execute activities including data entry and validation,
automated formatting, multi-format message creation, text mining, workflow
acceleration, reconciliations and currency exchange rate processing among
others.
Robotic process
automation (RPA)

Intelligent process automation is smart software with machine-learning
capabilities:
• Unlike RPA, which must be programmed to perform a task, AI can train
itself or be trained to automate more complex and subjective work
through pattern recognition
• Unlike RPA, which requires a human expert to hard code a script or
workflow into a system, AI can process natural language and unstructured
data
• Unlike RPA, AI responds to a change in the environment, adapts and
learns the new way
Intelligent process
automation (IPA)

Intelligent automation stages
Source: Shahim Ahmed, CA Technologies
107

Trigger based
Rules-based
dynamic language
Rules-based
standardized language
Structured
CHARACTERISTIC OF DATA / INFORMATION
Unstructured without patternsUnstructured patterned
Data Center
Automation:
Runbook
Scripting
Scheduling
Job control
Workload
automation
Process
orchestration
SOA
Virtualization
Cloud services
RPA
Cognitive
Computing
Artificial
Intelligence
BPM
Workflow
ERP
Autonomics
PROCESS CHARACTERISTICS
Source: HfS - 2016
Intelligent automation
continuum
The spectrum of intelligent
process automation spans
robotic process automation,
cognitive computing,
autonomics, and artificial
intelligence.
The direction of travel is  
along three dimensions.
Stages overlap.

Four aspects of self-management as they are now  
and as they become with autonomic computing
Concept Current computing Autonomic computing
Self-configuration Corporate data centers have multiple
vendors and platforms. Installing,
configuring, and integrating systems is
time consuming and error prone.
Automated configuration of components
and systems follows high-level policies.
Rest of system adjusts automatically and
seamlessly.
Self-optimization Systems have hundreds of manually set,
nonlinear tuning parameters, and their
number increases with each release.
Components and systems continually seek
opportunities to improve their own
performance and efficiency.
Self-healing Problem determination in large, complex
systems can take a team of programmers
weeks.
System automatically detects, diagnoses,
and repairs localized software and
hardware problems.
Self-protection Detection of and recovery from attacks and
cascading failures is manual.
System automatically defends against
malicious attacks or cascading failures. It
uses predictive analytics and early
warning to anticipate and prevent
systemwide failures.
Autonomic computing
Autonomic computing
refers to the self-managing
characteristics of AI-based
distributed computing
resources, adapting to
unpredictable changes
while hiding intrinsic
complexity to operators
and users.

Autonomous vehicles
Everyone (Google, Baidu,
Apple, NVidia, Uber, Tesla,
Volvo, Kamaz, Mercedes-
Benz, etc.) is developing
their own autonomous car.
Automobiles will soon
become really auto-
mobile. The main
restriction here seems to
be laws and regulations.
110

DRONE
Autonomous drones
Controlling different
things now seems efficient
using deep learning (e.g.,
games, game characters,
drones, autonomous cars,
robotic control.)
111

Artificial intelligence — issues
In order to work on real AI, as opposed to the hype presented by large
companies and the media these days, the following problems must be
worked on.
1. Knowledge Representation: This has always been the biggest problem in
AI but serious work on it stopped on it in the mid 80’s in favor of easy to
extract large, shallow libraries of lexical information.
2. Complex Models of Goals and Plans: In order to help and learn, an
intelligent system (a dog, a human or a computer) needs to know about
goals, and plans to achieve those goals, common mistakes with plans it has
tried in the past, and how to explain and learn from those mistakes.
3. Human-Like Models of Memory: Humans update their memory with every
interaction. They learn. Every experience changes their world model. In
order to build real AI we need to focus on limited domains of knowledge in
which the goals and plans of actors are represented and understood so
that they can be acted upon or acted against. AI systems must learn from
their own experiences, not learn by having information fed into them.
4. Conversational Systems: In practice, this means being able to build a
program that can hold up its end of a conversation with you. (unlike Siri or
any travel planning program). Such systems, should be linked to a memory
of stories (typically no more than 1.5 minutes in length and in video) from
the best and brightest people in the world. Those stories should “find” the
user when the program knows that they would be helpful. This happens
every day in human interaction. One person talks to another person about
what they are thinking or working on and the other person reacts with a
just-in-time reminding, a story that came to mind because it seemed
relevant to tell at the time, a story meant to help the other person think
things out.
5. Reminding: A computer in a situation must get reminded of relevant
situations it has previously experienced to guide it in its actions. This is real
AI. Done on a massive scale, this means capturing the expertise in a any
given domain by inputting stories and indexing those stories with respect
to what goals and plans and contexts they might pertain so that they can
be delivered just in time to a user. We can do this now to some extent, but
we need to start working on the real AI problems of automated indexing of
knowledge. (Although this may be what machine learning people say they
can do, they are talking about words and they are not trying to build an
ever increasingly complex world model as humans do through daily life.)
Source: Roger C Shank
112

AI encompasses multiple technologies that can be combined to sense, think,
and act as well as to learn from experience and adapt over time. Sense refers to
pattern recognition, machine perception, speech recognition, computer vision
and affective computing. Think refers to natural language processing, knowledge
representation and reasoning, machine learning and deep learning, and
cognitive computing. Act refers to search engines and question answering, rules
engines, expert systems, recommender systems, automated planning and
scheduling, autonomic computing, and autonomous systems.
113

Ai — Externalization of Mind

More Related Content

Similar to Ai — Externalization of Mind

More from Mills Davis

Recently uploaded

Ai — Externalization of Mind