AI in Physics - University of Washington, Jan 2024

AI in Physics
University of Washington
Incubator for Quantum
Simulation – IQuS
Peter Morgan
www.deeplp.com
UW IQuS – AI in Physics
© Peter Morgan 17 Jan 2024

About Me
• MSc (physics), University of Auckland
• PhD (physics), UMass Amherst with Barry Holstein (ABD)
• Founded company
• Then IT Consultant (10 years)
• Academia again as Research Associate – 3 years studying neutrino
double beta decay (mass)
• AI Consultant (last 10 years)
• Founded Deep Learning Partnership
• AI consulting mostly for businesses (also govt, education)
• Some quantum computing consulting (bit early for commercial application)

Motivation
• Contribute to accelerating science
• Applying AI to science – very impactful
• Due to my physics background, I am
motivated to use AI to solve science, as well
as business, problems
• Naturally inclined to combine AI and science
• Hopefully provide UW Physics with
motivation and research starting points

AI Hierarchy & Terminology
• We will use these three terms somewhat
interchangeably (even though we shouldn’t)
• Deep learning = ANNs (neural networks)
• AI – three categories
1. Narrow – ANI*
2. General – Human level, AGI**
3. Super – ASI
• Beyond human
• Narrow ASI*
• General ASI**
* Solved
**Not solved

So why AI now?
• Data, hardware and AI models have all been increasing exponentially over the
past 30 years
• We are now at trillions of tokens (words), trillions of model parameters, and
Exaflops of compute (Exa = 10^18)
• In 2012 we were at millions of tokens, millions of parameters and Teraflops
(Tera = 10^12)
• So a factor of ~million in all three dimensions !
• Recall human brain has ~100 billion neurons & ~1000 trillion synapses
• Uses ~ 1million times less energy but contains a lot less information (memory)
• LLMs contain & have processed virtually the whole of the Internet
• So we are in interesting times – let’s make the most of AI!

ANN Model Growth*
* Data sets and hardware are on similar exponentials (but also cost)

Intelligent Capabilities Emerge with Scale
https://blog.research.google/2022/04/pathways-language-model-palm-scaling-to.html

AI Models
• Proprietary
• GPT-4, Copilot, Gemini, Claude, Cohere, …
• OpenAI, Microsoft, Google, Anthropic, Cohere, …
• Access via API calls for a cost
• Best performance so far
• Open source
• Mistral, Falcon, Llama-2, Bloom, …
• Companies & university research groups
• Download model weights for free
• Lower performance than proprietary but catching up
• See leaderboards
• e.g., https://huggingface.co/spaces/lmsys/chatbot-arena-
leaderboard

How do they work? (we don’t completely know)

ANNs – High Level
• General Purpose Technology
• Discriminative vs Generative
• Many types of architectures and
approaches
• Use back propagation
• Data can be multimodal
• Some mysteries remain, e.g.,
emergence
• Lacking a complete mathematical
theory
• Complex statistical systems
• See references at end

Landscape of Generative AI
GANs
Variational
Autoencoders
Energy-based Models
Normalizing Flows
Transformers
Diffusion Models

Active Areas of AI Research
• Open vs closed source models
• Large vs small (trillions vs billions of parameters)
• Architectures
• Encoder only (autoencoder)
• Decoder only (autoregressive)
• Encoder-Decoder (seq2seq)
• Compression – quantization, distillation, etc.
• Fine-tuning  custom models
• Multimodality
• Retrieval (RAG)
• Reasoning
• AI Agents
• Scaling Laws
• Emergence – scale is all you need
• AGI (human-like intelligence)

AI can be used in all of science
• Cosmology
• Quantum gravity
• Particle physics
• Nuclear physics
• Materials science
• Chemistry
• Molecular dynamics
• Biology

Nuclear Physics Example – NNPDF
Collaboration
• Neural Networks for Parton Distribution Function
• “The NNPDF collaboration determines the structure of the
proton using contemporary methods of artificial intelligence”
• Use data from many particle experiments, including RHIC,
Tevatron, LHC
• Global collaboration going back to 2004
• Homepage: https://nnpdf.mi.infn.it
• Overview paper: Ball, R. et al, The Path to Proton Structure at
One-Percent Accuracy, 31 May 2022,
https://arxiv.org/abs/2109.02653
• Opensource code available: https://github.com/NNPDF/nnpdf
• NNPDF collaboration papers (23 so far):
https://arxiv.org/search/hep-
ph?searchtype=author&query=NNPDF+Collaboration

Can expect lots more data, more capable AI frameworks, plus more powerful hardware

References
References

AI Reasoning Capabilities
• Trinh, T. et al, Solving Olympiad geometry without human demonstrations, 17 Jan2024,
https://www.nature.com/articles/s41586-023-06747-5 Google Deepmind
• Sun, J. et al, A Survey of Reasoning with Foundation Models, 26 Dec 2023,
https://arxiv.org/abs/2312.11562v4
• Gurnee, W. & M. Tegmark, Language Models Represent Space and Time, 14 Dec 2023,
https://arxiv.org/abs/2310.02207, MIT
• Yao, S. et al, Tree of Thoughts: Deliberate Problem Solving with Large Language Models, 3 Dec 2023,
https://arxiv.org/abs/2305.10601, Google Deepmind
• Shi, W. et al, In-Context Pretraining: Language Modeling Beyond Document Boundaries, 30 Nov
2023, https://arxiv.org/abs/2310.10638, UW CS/ML & Allen Institute
• Sumers, T. et al, Cognitive Architectures for Language Agents, 27 Sept 2023,
https://arxiv.org/abs/2309.02427, Princeton
• Greyling, C. LangChain, LangSmith & LLM Guided Tree-of-Thought, 13 Sept 2023,
https://cobusgreyling.medium.com/langchain-langsmith-llm-guided-tree-of-thought-47a2cd5bcfca
• Bubeck, S. et al, Sparks of Artificial General Intelligence: Early experiments with GPT-4, April 2023,
https://arxiv.org/abs/2303.12712, Microsoft Research
• Paranjape, B. et al, ART: Automatic multi-step reasoning and tool-use for large language models, 16 Mar
2023, https://arxiv.org/abs/2303.09014, UW, MSR & Allen Institute

AI4Science Research Groups
All the major cloud providers are investing heavily in AI for science:
Google Deepmind
• Materials science: Merchant, A. et al, Scaling deep learning for materials discovery, 29 Nov 2023,
https://www.nature.com/articles/s41586-023-06735-9
• Fusion: Degrave, J. et al, Magnetic control of tokamak plasmas through deep reinforcement learning, 16 Feb
2022, https://www.nature.com/articles/s41586-021-04301-9
• Mathematics: Romera-Paredes, B. et al, Mathematical discoveries from program search with large language
models, 14 Dec 2023, https://www.nature.com/articles/s41586-023-06924-6
• Computer science: Mankowitz, D. et al, Faster sorting algorithms discovered using deep reinforcement
learning, 7 June 2023, https://www.nature.com/articles/s41586-023-06004-9
• Biology: Jumper, J. et al, Highly accurate protein structure prediction with AlphaFold, 15 July 2021,
https://www.nature.com/articles/s41586-021-03819-2
MSR
• AI4Science: https://www.microsoft.com/en-us/research/lab/microsoft-research-ai4science/
• PNNL Collaboration: https://www.pnnl.gov/pnnl-microsoft-collaboration
AWS, https://www.amazon.science/about
ByteDance, https://www.forbes.com/sites/alexandralevine/2024/01/02/tiktok-bytedance-pharmaceuticals-
drug-discovery-science-biology-chemistry-ai-china/?sh=5d5072615087

How is AI being used in nuclear physics?
• Zhang, X. et al, Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems, 15 Nov 2023,
• Matchev, K.T., et al, Seeking Truth and Beauty in Flavor Physics with Machine Learning, 31 Oct 2023,
• Cranmer, K. et al, Advances in machine-learning-based sampling motivated by lattice quantum chromodynamics, 3 Sept
2023, https://arxiv.org/abs/2309.01156
• Abbott, R. et al, Normalizing flows for lattice gauge theory in arbitrary space-time dimension, 3 May 2023,
• Liu, Z. et al, GenPhys: From Physical Processes to Generative Models, 5 April 2023, https://arxiv.org/abs/2304.02637
• He, Y-H. et al, Machine Learning in Physics and Geometry, 30 Mar, 2023, https://arxiv.org/abs/2303.12626
• Butter, A. et al, Machine Learning and LHC Event Generation, 28 Dec 2022, https://arxiv.org/abs/2203.07460
• Liu, Z. et al, AI Poincaré 2.0: Machine Learning Conservation Laws from Differential Equations, 30 Oct 2022,
• Shanahan, P. et al, Snowmass 2021 CompF03 Topical Group Report: Machine Learning, 15 Sept 2022,
• Boehnlein, A. et al, Machine Learning in Nuclear Physics, 2 May 2022, https://arxiv.org/abs/2112.02309 <-- review paper

Historic Papers of NNs in Particle Physics
• Denby, B., Neural networks and cellular automata in experimental high
energy physics, 3(49), June 1988!
https://www.sciencedirect.com/science/article/abs/pii/0010465588900045
• Cutts, D. et al, Applications of neural networks in high energy physics, Aug
1990, https://www.osti.gov/biblio/5954034
• Lonnblad, L. et al, Using neural networks to identify jets, Nucl. Phys. B349
(1991) 675–702
• Kanev, Y.A., Application of neural networks and genetic algorithms in high-
energy physics, UMI-99-05968
• Forte, S. et al, Neural network parametrization of deep-inelastic structure
functions, JHEP 05 (2002) 062, https://arxiv.org/abs/hep-ph/0204232

ML and Quantum Computing
• Castelvecchi, D., The AI–quantum computing mash-up: will it revolutionize
science?, 2 Jan 2024, https://www.nature.com/articles/d41586-023-04007-0
• Bausch, J. et al, Learning to Decode the Surface Code with a Recurrent,
Transformer-Based Neural Network, 9 Oct 2023,
• Machine Learning Aids Classical Modeling of Quantum Systems, Quanta
Magazine, 14 Sept 2023, https://www.quantamagazine.org/machine-
learning-aids-classical-modeling-of-quantum-systems-20230914
• Huang, H-Y. et al, Learning to predict arbitrary quantum processes, 15 April
2023, https://arxiv.org/abs/2210.14894
• Lewis, L. et al, Improved machine learning algorithm for predicting ground
state properties, 30 Jan 2023, https://arxiv.org/abs/2301.13169
• Moon, H. et al, Machine learning enables completely automatic tuning of a
quantum device faster than human experts, 8 Jan 2020,

Books & Online Resources
• Erdmann, M. et al, Deep Learning for Physics Research, World Scientific,
2021
• Tanaka, A. et al, Deep Learning and Physics, Springer Link, 2021
• Roberts, D. et al, Principles of Deep Learning Theory, CUP, 2021, or
arXiv, https://arxiv.org/abs/2106.10165
• Simon Prince, Understanding Deep Learning, MIT Press, Dec 2023 (also
free online)
• A Living Review of Machine Learning for Particle Physics, https://iml-
wg.github.io/HEPML-LivingReview/
• AI for Science, https://ai4sciencecommunity.github.io
• Undermind, an AI enabled research paper search engine, undermind.ai

Some Physics ML Research Groups
• MIT IAIFI, https://iaifi.org
• CERN Openlab, https://openlab.cern
• LANL, https://discover.lanl.gov/news/1017-ai-machine-learning/
• FNAL, https://computing.fnal.gov/artificial-intelligence/
• PNNL, https://www.pnnl.gov/artificial-intelligence
• Argonne, https://www.alcf.anl.gov/alcf-ai-testbed
• Flatiron, https://www.simonsfoundation.org/machine-learning-at-the-
flatiron-institute/
• NNPDF, https://nnpdf.mi.infn.it
• MSU FRIB, https://frib.msu.edu

AI in Physics Workshops
• Aspen Winter Conference Jan 2024, Fields, Strings & Deep learning,
https://indico.cern.ch/event/1299185/
• MIT IAIFI Summer Workshops (every year), https://iaifi.org/summer-workshop
• CERN 2023, AI4Science Workshop, https://indico.cern.ch/event/1326114/
• NeurIPS 2023, AI for Scientific Discovery: From Theory to Practice,
https://ai4sciencecommunity.github.io/neurips23.html
• ICML 2022, AI for Science: Theories and Foundations,
https://ai4sciencecommunity.github.io/neurips23.html
• UCLA IPAM 2019, Machine Learning for Physics and the Physics of Learning,
http://www.ipam.ucla.edu/programs/workshops/machine-learning-for-
physics-and-the-physics-of-learning-tutorials/

Further Resources
Physicists (check out their papers)
• Max Tegmark – MIT
• Kyle Cranmer – University of Wisconsin-Madison
• Danilo Rezende – Google Deepmind
AI Research in Seattle*
• UW Computer Science, e.g., Luke Zettlemoyer
• Allen Institute for AI, e.g., Noah Smith
• Microsoft Research, Redmond
*Collaboration is fruitful for both physicists and AI researchers

Questions & Discussion

AI in Physics - University of Washington, Jan 2024

Recommended

Recommended

More Related Content

Similar to AI in Physics - University of Washington, Jan 2024

Similar to AI in Physics - University of Washington, Jan 2024 (20)

More from Peter Morgan

More from Peter Morgan (12)

Recently uploaded

Recently uploaded (20)

AI in Physics - University of Washington, Jan 2024