What new developments and capabilities we'll be seeing by the end of 2023 and 2024 in AI. Multimodal AI can make video, control robots. Techniques for amplicifaction and distillation improves quality from little data. Brains can be decoded with AI. What future do we want to create with this power?

1. The Near Future
New AI capabilities for 2024
consumer products
Dr. Josiah Seaman

2. Overview
Human
and
AI Teaming
Distillation
Tree of Thoughts, Autonomy,
Energy vs. Quality
Amplifying
Training Data
What is AI,
really?
Multimodal AI
Text, Images, Video, Sound, Robotics

3. Personal Experience: AlphaFold 2
Gangulli et al. 2022

4. Skills in a Box
What is AI, really?
• A useful paradigm for thinking about AI:
• “An AI model is a single skill from many people distilled into software” –
Josiah Seaman
• Implications:
• (Almost) All AI is Human Derived
• AI models start out very narrow and can be composed more broadly
• While a skill remains in a human mind it cannot be shared
• It is challenging for AI to rise above human skill maximums
• More on that later

5. Overview
Human
and
AI Teaming
Distillation
Tree of Thoughts, Autonomy,
Energy vs. Quality
Amplifying
Training Data
What is AI,
really?
Multimodal AI
Text, Images, Video, Sound, Robotics

6. Text
Large Language Models capture much
of the human experience in text. This
includes reasoning and planning.
1
Robotics
Dexterity, movement, and
understanding of the physical world.
Often tactile sense.
2
Video
Moving pictures with consistency over
time.
3
Audio
Both listening, understanding, and
producing sounds. Not limited to
speech, music, and sound effects.
4
• Expanding the range of skills replicated by AI
• New abilities have a multiplicative effect
Multimodal AI

7. Multimodal AI
ChatGPT with Images Combining capabilities has a multiplicative effect

8. Text guided Video to AI Video
Joel Haver Jan 2021

9. Text guided Video to AI Video
Corridor Digital – Feb 2023

10. Text guided Video to AI Video
Martin Haerlin – June 2023

11. Pure Text to Video
Source:
2 Minute
Papers
2023

12. Multimodal AI
• LLM is a drop-in language capability for any context
• Gato - the one model to rule them all
• “Game Over” tweet

13. Multimodal AI
Final Thoughts
• Robotics will be covered in the next talk
• Existing models can be composited into Multimodal AI
• Multimodal is necessary to reach AGI - far future – 2029

14. Overview
Human
and
AI Teaming
Distillation
Tree of Thoughts, Autonomy,
Energy vs. Quality
Amplifying
Training Data
What is AI,
really?
Multimodal AI
Text, Images, Video, Sound, Robotics
Superhuman Skill

15. Amplification in Training
“The core characteristic is that amplification uses the original process as a starting point and applies more
computational resources to create a more powerful agent.
Distillation can be any process whereby we compress this more expensive amplified agent into something that we
can call cheaply.” – Robert Miles, AI Safety Expert
Source:
OpenAI, 2022
• Amplification is useful and essential for quality and
quantity. AI imagines combinations or extrapolations of
training data.
• RLHF – Reinforcement Learning with Human Feedback

16. Amplification Prompting
• AutoGPT (with vector database memory)
• SmartGPT - With 3 fixed solutions then
pick best solution
• Tree of Thoughts - tree traversal +
scoring candidates
• Simulate multiple steps in the
future, try out branching outcomes.
Test which outcomes are the best,
pick the solution
Yao et al. 2023

17. Amplification Downsides
• AI is not completely immune to software problems and gets stuck in loops
• All AI have blind spots which can be detected if adversaries gain access to the network
Adversarial Policies beat
Superhuman GO AIs – ICLR
2023
“KataGo is, in this
instance, less
robust than human
amateurs, despite
having
superhuman
capabilities.”
Beaten by a novice using double enclosure
White AI surrenders

18. Amplification Downsides
Training Data Biases
• “Over time, mistakes in generated data compound and ultimately force models that learn from generated data to
misperceive reality even further,” -- Ilia Shumailov, AI Researcher
• “Just as we’ve strewn the oceans with plastic trash and filled the atmosphere with carbon dioxide, so we’re about
to fill the Internet with blah. This will make it harder to train newer models by scraping the web” -- Ross Anderson
Glitch Tokens

19. Overview
Human
and
AI Teaming
Distillation
Tree of Thoughts, Autonomy,
Energy vs. Quality
Amplifying
Training Data
What is AI,
really?
Multimodal AI
Text, Images, Video, Sound, Robotics
Superhuman Skill

20. Distillation
“Distillation can be any process whereby we compress this more expensive amplified agent into something that we
can call cheaply” – Robert Miles, AI Safety Expert
“If I had more time, I would have written a shorter letter.”
--Blaise Pascal
• Distillation = train a predictor which picks the correct solution first before the multi-step simulation is necessary.
Robert Miles:
Used with Permission

21. Distillation
• AlphaZero showed us that it’s possible for an AI to teach itself to become better when they’re already the best
player in the world.
• ChatGPT used to train Vicuna
Source: LMSYS.org
Large Model Systems Organization
The False Promise of Imitating Proprietary LLMs
Gudibande et al. 2023 May 2023
ORCA 13-B: Progressive Learning from Complex
Explanation Traces of GPT-4. Mukherjee et al.
June 2023

22. Overview
Human
and
AI Teaming
Distillation
Tree of Thoughts, Autonomy,
Energy vs. Quality
Amplifying
Training Data
What is AI,
really?
Multimodal AI
Text, Images, Video, Sound, Robotics

23. Skills in a Box
A good outcome and a bad
outcome
Current Policy: Efficiency is King
1. Companies need skills, but employees are costly…
2. AIs need to maximize goals, but humans are costly…
Good Policy: Abundance enables Luxury
1. Companies increase productivity at lower cost and pay employees
more
2. Customers are also employees who now have more to spend
3. AIs optimizers reflect abundance policy

24. Human and AI Teaming
• The Human Experience is a combination of our awareness, our will, and our skills.
• AI allows us to share and benefit from the skills of the whole planet, which also makes our skills less special.
• Personal story: dementia is so difficult because it undermines our awareness, even if our skills remain.
• It’s difficult to exercise our skills in the world when we are not aware.
The Bright Future
• Humans  Will
• AI  Skills
• We become more capable of accomplishing what we want

25. Brain-Machine Interfaces
• “The least scary future I can think of is one where we have at least democratized AI” -- Elon Musk,
Neuralink Founder
• fMRI can now decode accurate full sentence train of thought and visual cortex decoding - Semantic
reconstruction of continuous language from non-invasive brain recordings
Takagi and Nishimoto / bioRxiv, 2022

26. Brain-Machine Interfaces
• Neuralink receives FDA approval for Human trials – June 2023
• Limited success on non-invasive decoding is improving now, some direct to consumer products available
Avatar - 2154
Cyberpunk - 2077
Green Mars - 2127
Sci-Fi Reality
fMRI - 2023
Thought to Text
Thought to Image
Synchron – July 2022
Neurosity Crown - 2023
Neuralink - 2024
Semantic reconstruction of
continuous language from non-
invasive brain recordings
Jerry Tang, Amanda LeBel, Shailee Jain &
Alexander G. Huth. Nature Neuroscience,
May 2023
Graphic by Presented by NBC News

27. Brain-Machine Interfaces
Semantic reconstruction of continuous language from non-invasive brain
recordings
Jerry Tang, Amanda LeBel, Shailee Jain & Alexander G. Huth. Nature Neuroscience,

28. THANK YOU
Hire Me
Software Consulting
Biotechnology
Business Automation
Generative AI
Josiah.seaman@gmail.com
@josiahseaman
josiahseaman.com

29. THANK YOU
Hire Me
Software Consulting
Biotechnology
Business Automation
Generative AI
Josiah.seaman@gmail.com
@josiahseaman
josiahseaman.com