Generative AI: Past, Present, and Future – A Practitioner's Perspective As the academic realm grapples with the profound implications of generative AI and related applications like ChatGPT, I will present a grounded view from my experience as a practitioner. Starting with the origins of neural networks in the fields of logic, psychology, and computer science, I trace its history and align it within the wider context of the pursuit of artificial intelligence. This perspective will also draw parallels with historical developments in psychology. Against this backdrop, I chart a proposed trajectory for the future. Finally, I provide actionable insights for both academics and enterprising individuals in the field.

1. GENERATIVE AI: PAST,
PRESENT, AND FUTURE
– A PRACTITIONER'S
PERSPECTIVE
Huahai Yang
Co-founder & CTO, Juji, Inc.
August 14, 2023

2. INTERNET CHANGED THE WORLD
What would Generative AI do?
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 2

3. AGENDA
Rise of GenAI
How we got here
Assessment from psychology
Paths forward
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 3

4. GENERATIVE AI
Generative AI is a subset of artificial intelligence
that focuses on creating new content.
It is often based on the frameworks of machine
learning and deep learning.
The systems learn patterns, features, and
correlations from massive amounts of data, and
they can generate output such as images, music,
voice, text, code, or other types of content that
mirrors the learned data.
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 4

5. GENERATIVE AI APPLICATIONS
Fastest Growing App in
History
Re ac h e d 1 0 0 m illion
m ont hly ac t ive use r s in
t wo m ont hs
Open Models
L ar g e L ang uag e Mode ls
wit h a g r owin g
e c osyst e m
Fast Growing Text to
Image App
1 5 m illion use r s
Open Models
T e x t t o Im ag e Mode ls
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 5

6. INFLATED EXPECTATION?
Beginning of artificial general intelligence (AGI)?
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 6

7. ORIGIN OF AI
Goals
The study is to proceed on the basis of the
conjecture that every aspect of learning or any
other feature of intelligence can in principle be
so precisely described that a machine can be
made to simulate it.
An attempt will be made to find how to make
machines use language, form abstractions and
concepts, solve kinds of problems now reserved
for humans, and improve themselves.
Some Participants
• John McCarthy
• Marvin Minsky
• Oliver Selfridge
• Claude Shannon
• John Nash
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 7
1956 Dartmouth Summer Research Project on Artificial Intelligence
• Herbert Simon
• Allan Newell
• John H. Holland
• William Ross Ashby
• Warren S. McCulloch

8. LANDMARKS OF GENERATIVE AI
1940s
1950s-1970s
1980s-2006
2006-present
McCulloch, Warren S., and Walter Pitts. "A logical calculus of the ideas immanent in
nervous activity." The bulletin of mathematical biophysics 5 (1943).
• Rosenblatt, F. "The perceptron: a probabilistic model for information storage
and organization in the brain." Psychological review 65.6 (1958).
• Minsky, M. L. and Papert, S. A. Perceptrons: an Introduction to
Computational Geometry. MIT Press (1969).
• Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. "Learning
representations by back-propagating errors." nature 323.6088 (1986).
• Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning
algorithm for deep belief nets." Neural computation 18.7 (2006).
• Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "ImageNet
classification with deep convolutional neural networks." Advances
in neural information processing systems 25 (2012).
• Vaswani, Ashish, et al. "Attention is all you need." Advances in
neural information processing systems 30 (2017).
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 8

9. MCCULLOCH, WARREN S., AND WALTER PITTS. (1943) A LOGICAL
CALCULUS OF THE IDEAS IMMANENT IN NERVOUS ACTIVITY.
• All-or-nothing nature of neural action potentials
(threshold)
• “Response of any neuron is equivalent to a logic
proposition which proposed its adequate stimulus.”
• Physiological relations of nervous activities
correspond to relations among the propositions.
• Neuron is a logic expression of disjunction,
conjunction, negation, i.e. a Boolean function
• Learning (facilitation or inhabitation) as changing
the function expression
• Prove some neural activities are realizable by this
calculus
• Equivalent to Turing machine (1936), hence an
universal computing machine
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 9

10. ROSENBLATT F. (1958) THE PERCEPTRON: A PROBABILISTIC MODEL
FOR INFORMATION STORAGE AND ORGANIZATION IN THE BRAIN.
• Propositional logic expression -> vector dot product
• Probabilistic interpretation of stimuli contribution
p(xi)p(d|xi)
• Permit geometrical interpretation of tasks
• Proposed a learning algorithm
• Given a series of stimuli samples and corresponding
outcome
• Calculate predicted values (1)
• Update weights by prediction error (2)
• Iterate
• Converge when sample features are linearly
separable
• All these become basic ingredients of GenAI today
• “Father of Deep Learning” – C.C. Tappert, 2019
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 10
(1)
(2)
Sum f y
x1
x2
x3
w1
w3
w2

11. MINSKY, M. L. AND PAPERT, S. A. (1969, 1988) PERCEPTRONS: AN
INTRODUCTION TO COMPUTATIONAL GEOMETRY.
• Formal treatment of Rosenblatt’s perceptrons
• Deal with simple (single layer, no loops) perceptron only
• Prove some theorems regarding the ability of simple
perceptrons to recognize some global patterns
• Connectedness (figure-ground)
• Parity (odd, even)
• Notably, simple perceptrons with limited number of A-
unites that has local connections only, cannot handle XOR
patterns
• People (incl. authors) extrapolated this limit to all
perceptrons
• First AI winter
• “This is quite a humorous turn of events. The
psychologists offer an implemented information-
processing model, which the computer scientists reject in
favor of a creative psychological theory!” - J.B. Pollack,
1988
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 11
XOR

12. • RUMELHART, DAVID E., GEOFFREY E. HINTON, AND RONALD J.
WILLIAMS. (1986) LEARNING REPRESENTATIONS BY BACK-
PROPAGATING ERRORS.
• Improve Rosenblatt’s learning algorithm:
backpropagation
• Learning non-linearly separable features
needs multiple layer perceptrons
• Difficult to converge in multiple layers
• Introduce these changes
• Linear threshold function f is replaced by
non-linear function, e.g. sigmod (3)
• Error function is replaced by Mean Squared
Error (4)
• Pass back partial derivatives of error (5)
• Update weights only after going through all
samples (6)
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 12
(3)
(4)
(5)
(6)

13. • HINTON, GEOFFREY E., SIMON OSINDERO, AND YEE-WHYE TEH.
(2006) A FAST LEARNING ALGORITHM FOR DEEP BELIEF NETS.
• Some problems in training of deep network
• Vanishing/exploding gradients
• Overfitting
• Explaining away
• Solution
• Contrastive diverge learning for Restricted Boltzmann
Machines (RBM)
• Single layer -> two layers, one up, one down
• Gibbs sampling back and forth until convergence
• Measure KL divergence between input data and
generated data to update weights
• Stacked RBMs build up abstractions layer by layer
• Able to see the generated images at each layer
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 13

14. • KRIZHEVSKY, ALEX, ILYA SUTSKEVER, AND GEOFFREY E. HINTON.
(2012) IMAGENET CLASSIFICATION WITH DEEP CONVOLUTIONAL
NEURAL NETWORKS.
• Breakout moment of deep learning (DL)
• Won ImageNet Large-Scale Visual
Recognition Challenge
• Reduce top-5 error from 26% to 15.3%
• Machine learning is dominated by DL
since
• Techniques
• GPU enable training of large DL
network
• Rectified Linear Unit (ReLU) non-linear
function
• Dropout to reduce overfitting
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 14

15. • VASWANI, ASHISH, ET AL. (2017) ATTENTION IS ALL YOU
NEED.
• Transformer for predicting next token in a sequence
• self-attention mechanisms handle long range dependencies
• process the entire sequence at once, highly parallelizable
• less gradient vanishing/exploding, flexible
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 15

16. INSTRUCTGPT, CHATGPT, GPT-4, GPT-5…
• Larger models, GTP-3 has 175 billion parameters
• Align GPT to user tasks and requirements
• Supervised fine-tuning (SFT)
• Reinforcement learning from human feedback (RLHF)
• Mixture of experts (MoE)
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 16

17. ASSESSMENT
What has Gen AI achieved?
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 17

18. TURING TEST IS INADEQUATE
• PARRY has passed Turing Test in 1972
• PARRY simulated a person with paranoid
schizophrenia
• One group of experienced psychiatrists
interacted with either PARRY or real patients
• Another group of 33 psychiatrists were shown
transcripts
• They were asked to tell which were human,
which were computer programs
• They could tell correctly 48% of the time
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 18

19. GENERATIVE AI IS BEHAVIORISM
Behaviorism
Objective observable behaviors
Environment determines behaviors
Adapt via learning only
All organisms learn in similar ways
Behaviors can be shaped via
reinforcement
No such thing as mind *
Gen AI
End to end
Data driven
Model change is via training only
Learn all tasks in similar ways
Behaviors can be shaped via reinforcement
Gen AI has mind already; No such thing as
mind
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 19
* B.F. Skinner, Can Psychology Be a Science of Mind? American Psychologist, November 1990, Vol. 45, No. 11, 1206-1210

20. CHOMSKY’S ATTACK ON BEHAVIORISM DOES NOT
WORK ON GEN AI
Criticisms
Lack of innate abilities
Absence of creativity
Reject internal mental states
Response
Pretrained models considered innate
Gen AI is shown to be creative
Gen AI does have internal mental states
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 20

21. ASSESSMENT IN COGNITIVE CAPABILITIES
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 21

22. WHAT HAS GENERATIVE AI ACHIEVED?
Able to do
Perception
Memory
Language
Unable to do
Organization of knowledge
Mental images and propositions
Attention and consciousness
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 22
Depends
Problem solving and creativity
Decision making and reasoning

23. CORE COMPETENCE: LEARN MAPPING IN VECTOR SPACES
• Image classification: pixels => labels
• Image generation: labels => pixels
• Regression: raw data => numbers
• Embedding: raw data => vectors
• Prompting: text sequence => text
sequence
• Perception: forward mapping
• Generation: backward mapping
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 23

24. PERCEPTION SOLVING REASONING PROBLEMS
• Reasoning problems -> perceptual problems
• After seeing enough cases
• Chunking
• Deep learning builds abstraction layer by layer
• Training with enough data
• Also chunking
• Learned knowledge representation may not be
the same as human’s
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 24
• Reporter: How many moves do you see
ahead while playing chess?
• Capablanca: Only one, but it’s always the
right one

25. WHAT GENERATIVE AI CAN DO: PERCEPTUAL TASKS
PERCEPTUAL
Perception
Memory
Language
NON-PERCEPTUAL
Organization of knowledge
Mental images and propositions
Attention and consciousness
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 25
DEPENDS
Problem solving and creativity
Decision making and reasoning

26. ORGANIZATION OF KNOWLEDGE
At Odds with Human
Hallucination
Accidental properties
Against societal values
High cost
Poverty of Representation
Expressiveness
Mental images
Propositions
Flexibility and velocity of updates
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 26

27. HALLUCINATION IS UNAVOIDABLE
• No distinction between fact and fiction
• Generation is based on probabilistic sampling
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 27

28. ACCIDENTAL PROPERTIES MAY BE LEARNED
• Minor perturbation in input space, may result in big change in concept space
• Learned features may not match human conception
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 28

29. ADVERSARIAL ATTACKS AGAINST INSTRUCTIONS
• Universal, transferable adversarial attacks can be systematically trained
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 29

30. HUGE COST
• Power inefficient
• GPT-3: 1024 GPUs, 34 days to train, 1287 MWh of power
• Information inefficient
• GPT-3: 175B parameters, 800GB to store
• Wikipedia: 22GB to store
• Exponential growth of model size
• Labor intensive
• SFT, RFHL take huge human effort
• OpenAI paid Keyan less than $2 per hour , had to read and label
between 150 and 250 passages of text per 9-hour shift
• Only huge commercial entities can afford to train large models
• Monopolization hampers innovation
• Lack of openness and accountability
• Exacerbate digital divide
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 30

31. LACK OF EXPRESSIVENESS
• Geometric representation is the only game in town
• Elegant and powerful, but does not cover everything
• Godel’s incomplete theorem
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 31
• In principle, these can be represented in vector
space, but only inefficiently
• Propositions
• Graphs
• World models
• Mental images
• Knowledge of GenAI is not human interpretable

32. CHANGE AND COMMUNICATION ARE DIFFICULT
• The only way to change a Gen AI model is to train with raw
data
• Risk of new data breaks existing behaviors
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 32
• The only way to communicate with a Gen AI model
is via the same type of data as training data
• Prompt ”engineering” is actual an art
• Model output are uncertain
• Stochastic generation
• Setting temperature to 0 does not guarantee same
results
• No real time active learning from experience
• Models are frozen, there is only nature, no nurture

33. ATTENTION AND CONSCIOUSNESS
Lack of Agency
Top-down processing
Proactive action
Individual differences
Lack of Embedment
Knowledge of physicality
Empathy
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 33

34. LACK OF AGENCY
• Reactive, not proactive
• No explicit goals, other than please users
• Only responsive to external changes, not internal
• Will not ask user questions, only answer them
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 34
• There are only data-driven, bottom-up processes
• Top down attention mechanism would be nice
• No individual differences
• Models are essentially the average of training data
• May be told to play certain role, no true identity
• Cannot detect user’s individual differences

35. LACK OF EMBEDMENT
• Lack sensory input and output
• Sight, sound, touch, taste, smell
• Spatial relations, movement, temperature
• Understand physical limitations
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 35
• Lack emotion understanding
• Fatigue, fear, nervous, happy
• Social norm, body language
• Low empathy
• Limited persuasion and influence

36. WOULD GEN AI TAKE MY JOB?
Most Jobs are Safe
Increased personal productivity
New job categories will appear
Enterprise use is challenging
Low Level Creative Jobs Loss
Copy writers
Illustrators
Translators
…
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 36
Impact Similar to Search Engine

37. PATHS FORWARD
Practice: Integrated AI
Common Ingredients
Juji ways
Science
Beyond Current State of Gen AI
Human AI collaboration
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 37

38. COGNITION REQUIRES TWO WAY PROCESSING
Bottom-up Top-down
Data-driven Goal-driven
Sub-symbolic Symbolic
Machine knowledge Human knowledge
Generative Curated
Frozen Fluid
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 38

39. GOAI WAS DEAD, LONG LIVES THE GOAI
• Good Old AI (GOAI),
• Expert Systems, knowledge graph, semantic Web, etc.
• Victim of 2nd AI winter
• Failure due to the weak perceptual foundation
• Gen AI now provides a solid perceptual foundation.
• The same forces leading to rise of Gen AI, apply to GOAI
• Powerful GPU => better graph search
• Abundant realistic data => better knowledge base
• Better software tools and practices
• Need to integrate Gen AI with GOAI.
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 39

40. TWO ROADS TO INTEGRATION
GOAI as Basis
Engineer in nature
Pragmatic
Gen AI as Basis
Reductionist in nature
Impractical
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 40

41. INGREDIENTS FOR ENTERPRISE GEN AI DEPLOYMENT
Agent Framework
Perceptors and
Actuators
Central Control Unit
Memory
Customized Plugins
No-Code Platform
Friendly Graphical
Interface
Test and Evaluation Tools
Performance Reporting
Live Human Integration
Infrastructure
Scalable Platform
DevOps
Security
Compliance
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 41

42. JUJI: RESPONSIBLE EMPATHETIC PERSONA (REP)
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 42
Agency
• Each conversation backed by own
REP
• Stateful, keep context
• Has its own event loop
• Not just reactive, also proactive
• Has agenda
• Can loop back to agenda
• Interruptible
• React properly to
interruptions
• Resume after interruption
Individual Difference
• Proprietary psychometric models
• Measure in serendipity
• Good validity
• Better reliability than paper-
pencil instruments
• Individualized experience
• Messaging tailored to users
• Conversation path customized
to users

43. JUJI: SYMBOLIC AS BONES, DATA-DRIVEN AS FLESH
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 43
Production Rules
• Rule engine as the backbone
• Match -> Action
• Parallel processing
• Best match fires
• Match & action can be anything
• Patterns
• Function calls
Gen AI Functions
• Gen AI is a function, use it as
such in rules
• Measure similarity to input
• Approve a response
• Choose from multiple matches
• Choose from multiple responses
• Extract entities
• Test if a question is asked
• Verify a question is answered
• Query user data
• …

44. JUJI: AUTOMATIC DIALOG MANAGEMENT
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 44
Topic Abstraction
• Topic as basic dialog unit
• Meaningful
• Flexible
• Composable
• Agenda include topics
• Tracked to ensure completion
• Allow out of order topics
• Topics can have own agenda
Society of Minds
• Topics all have chances
• Tried in parallel
• Compete for taking effect
• All can also contribute
• Domain specific language
• Topic generation
• Topic manipulation

45. BEYOND CURRENT STATE OF GEN AI
GEN AI Usability
Smaller
Faster
More robust
Run in more
devices
Theory
Efficient
algorithms
Continuous
learning
Geometric
information
theory?
Integration with
GOAI
Top-down
attention
Knowledge
sharing
New AI
Paradigms
Causal AI
Evolutionary AI
Embedded AI
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 45

46. RESEARCH ON HUMAN AI COLLABORATION
EVALUATION
Model
performance
System usability
Domain use of AI
DESIGN
Interaction
methods
Human in the loop
Humanist AI
OPENNESS
Explainability
Accessibility
Education
ALIGNMENT
Safety
Privacy
Trust
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 46

47. SUMMARY
Generative AI, once a whisper, now a song,
Overcoming perception, proving skeptics wrong.
Behaviorist at heart, in a system hybrid aligned,
With old AI, a novel dance designed.
Shining bright, productivity's new dawn,
A dream realized, a promise drawn.
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 47

48. THANK YOU
Huahai Yang
hyang@juji-inc.com
Juji.io
2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 48