Physics-Informed Machine Learning Methods for Data Analytics and Model Diagnostics

1. Physics-Informed Machine Learning Methods for
Data Analytics and Model Diagnostics
Velimir V. Vesselinov (monty) (vvv@lanl.gov)
Earth and Environmental Sciences Division, Los Alamos National Laboratory, NM, USA
http://tensors.lanl.gov
LA-UR-19-24562
ML Studies Climate: EU Summary

2. Machine Learning (ML)
We have developed a series of novel
Machine Learning (ML) methods
Data Analytics: Identify features
(signals) in datasets
Model Analytics: Identify processes
(features) in model outputs
Physics-Informed Machine Learning:
Coupled Data/Model Analytics
ML Studies Climate: EU Summary

3. Physics-Informed Machine Learning (PIML)
Extracting common hidden features present in data (observations/experiments) and
model outputs
Incorporating physics constraints in ML analyses
Incorporating physics processes in ML analyses
ML Studies Climate: EU Summary

4. Physics-Informed Machine Learning (PIML)
PIML is of critical importance for data- and physics-driven science and engineering
applications
PIML models can replace computationally intensive numerical simulators
PIML models can be applied to predict complex processes when:
• physics is not well understood
• numerical models cannot be developed (due to physics complexity)
Training PIML models is 1-2 orders of magnitude faster than traditional ML models
ML Studies Climate: EU Summary

5. Physics-Informed Machine Learning (PIML): Cartpole
“Deep Learning is dead. Long Live Differentiable Programming!”
Yann LeCun, Chief AI Scientist at Facebook, and one of the leading ML experts in the
world
Differentiable Programming for PIML can be done efficiently only in
C/C++, FORTRAN, Python, scikit-learn, TensorFlow, PyTorch cannot do it
ML Studies Climate: EU Summary

6. Physics-Informed Machine Learning (PIML): Cartpole
ML Studies Climate: EU Summary

7. ML Analyses
Field Data:
Groundwater contamination
US Climate data
Geothermal data
Seismic data
Lab Data:
X-ray Spectroscopy
UV Fluorescence Spectroscopy
Microbial population analyses
Operational Data:
LANSCE: Los Alamos Neutron
Accelerator
Oil/gas production
Model Outputs:
Reactive mixing A + B → C
Phase separation of co-polymers
Molecular Dynamics of proteins
EU Climate modeling
ML Studies Climate: EU Summary

8. ML Publications
Vesselinov, Munuduru, Karra, O’Maley, Alexandrov, Unsupervised Machine Learning
Based on Non-Negative Tensor Factorization for Analyzing Reactive-Mixing, Journal of
Computational Physics, (in review), 2019.
Stanev, Vesselinov, Kusne, Antoszewski, Takeuchi, Alexandrov, Unsupervised Phase
Mapping of X-ray Diffraction Data by Nonnegative Matrix Factorization Integrated with
Custom Clustering, Nature Computational Materials, 2018.
Vesselinov, O’Malley, Alexandrov, Nonnegative Tensor Factorization for Contaminant
Source Identification, Journal of Contaminant Hydrology, 2018.
O’Malley, Vesselinov, Alexandrov, Alexandrov, Nonnegative/binary matrix factorization
with a D-Wave quantum annealer, PLOS ONE, 2018.
Vesselinov, O’Malley, Alexandrov, Contaminant source identification using
semi-supervised machine learning, Journal of Contaminant Hydrology,
10.1016/j.jconhyd.2017.11.002, 2017.
Alexandrov, Vesselinov, Blind source separation for groundwater level analysis based
on nonnegative matrix factorization, WRR, 10.1002/2013WR015037, 2014.
ML Studies Climate: EU Summary

9. Climate model of Europe
TerrSysMP: coupled climate simulator
(CLM/ParFlow) of atmospheric, surface
and subsurface processes
ML Studies Climate: EU Summary

10. Climate model of Europe: 2003 air temperature
Daily fluctuations in the air temperature
[◦
C]
Tensor: (424 × 412 × 365)
(columns × rows × days)
NTFk applied to extract dominant
hidden (latent) features
ML Studies Climate: EU Summary

11. Climate model of Europe: 2003 temperature fluctuations represented by 3 features
ML Studies Climate: EU Summary

12. Climate model of Europe: 2003 temperature fluctuations represented by 3 features
ML Studies Climate: EU Summary

13. Climate model of Europe: 2003 water-table depth
Daily fluctuations in the water-table
depth [m]
Tensor: (424 × 412 × 365)
(columns × rows × days)
NTFk applied to extract dominant
hidden (latent) features
ML Studies Climate: EU Summary

14. Climate model of Europe: 2003 water-table fluctuations represented by 3 features
ML Studies Climate: EU Summary

15. Climate model of Europe: Water-table fluctuations represented by 2 features
ML Studies Climate: EU Summary

16. Climate model of Europe: Water-table fluctuations represented by 3 features
ML Studies Climate: EU Summary

17. Climate model of Europe: Water-table fluctuations represented by 4 features
ML Studies Climate: EU Summary

18. Climate model of Europe: Water-table fluctuations represented by 5 features
ML Studies Climate: EU Summary

19. Climate model of Europe: Water-table fluctuations represented by 6 features
ML Studies Climate: EU Summary

20. Climate model of Europe: air temperature 1989-2017
Monthly fluctuations in the air
temperature from 1989 to 2017 [◦
C]
Tensor: (316 × 316 × 348)
(columns × rows × months)
NTFk applied to extract dominant
hidden (latent) features
ML Studies Climate: EU Summary

21. Climate model of Europe: 2003 air temperature reconstruction by 3 features
Original Reconstruction Error
ML Studies Climate: EU Summary

22. Climate model of Europe: 2003 air temperature reconstruction by 4 features
Original Reconstruction Error
ML Studies Climate: EU Summary

23. Climate model of Europe: 2003 air temperature reconstruction by 5 features
Original Reconstruction Error
ML Studies Climate: EU Summary

24. Climate model of Europe: 2003 air temperature reconstruction by 6 features
Original Reconstruction Error
ML Studies Climate: EU Summary

25. Climate model of Europe: 2003 air temperature reconstruction by 7 features
Original Reconstruction Error
ML Studies Climate: EU Summary

26. Climate model of Europe: 2003 air temperature reconstruction by 8 features
Original Reconstruction Error
ML Studies Climate: EU Summary

27. Climate model of Europe: 2003 air temperature reconstruction by 9 features
Original Reconstruction Error
ML Studies Climate: EU Summary

28. Climate model of Europe: 2003 air temperature reconstruction by 10 features
Original Reconstruction Error
ML Studies Climate: EU Summary

29. Climate model of Europe: 2003 air temperature reconstruction by 15 features
Original Reconstruction Error
ML Studies Climate: EU Summary

30. Climate model of Europe: 2003 air temperature reconstruction by 20 features
Original Reconstruction Error
ML Studies Climate: EU Summary

31. Climate model of Europe: 2003 air temperature reconstruction by 25 features
Original Reconstruction Error
ML Studies Climate: EU Summary

32. Climate model of Europe: 2003 air temperature reconstruction by 30 features
Original Reconstruction Error
ML Studies Climate: EU Summary

33. Climate model of Europe: 2003 air temperature reconstruction by 35 features
Original Reconstruction Error
ML Studies Climate: EU Summary

34. Climate model of Europe: 2003 air temperature reconstruction by 40 features
Original Reconstruction Error
ML Studies Climate: EU Summary

35. Climate model of Europe: 2003 air temperature reconstruction by 45 features
Original Reconstruction Error
ML Studies Climate: EU Summary

36. Climate model of Europe: 2003 air temperature reconstruction by 50 features
Original Reconstruction Error
ML Studies Climate: EU Summary

37. Climate model of Europe: air temperature reconstruction errors
ML Studies Climate: EU Summary

38. Climate model of Europe: air temperature features (8)
ML Studies Climate: EU Summary

39. Climate model of Europe: air temperature features (8) 1989-2017
ML Studies Climate: EU Summary

40. Climate model of Europe: air temperature features (8) 2002-2003
ML Studies Climate: EU Summary

41. Climate model of Europe: air temperature reconstruction by 8 features
Original Reconstruction Error
ML Studies Climate: EU Summary

42. Climate model of Europe: air temperature reconstruction by 50 features
Original Reconstruction Error
ML Studies Climate: EU Summary

43. Climate model of Europe: analyze all model outputs (>40) simultaneously
Water-table depth Temperature Evaporation Sensible heat flux
Find interconnections and common dominant features in model outputs
Evaluate impacts of different model setups (initial conditions, resolution, etc.)
Find dominant processes impacting model predictions
(e.g., occurrence of heat waves, climate impacts on groundwater resources, climate
impacts of subsurface processes on atmospheric conditions)
ML Studies Climate: EU Summary

44. Summary
Developed a series of novel ML methods and
computational tools
Our ML methods have been applied to solve
various real-world problems at various scales
(from molecular to global)
(brought breakthrough discoveries related to
human cancer research)
Modeldata
Sensor data Experimentaldata
Robust Unsupervised
Machine Learning
ML Studies Climate: EU Summary

45. Machine Learning (ML) Algorithms / Codes developed by our team
Codes:
NMFk.jl Mads.jl NTFk.jl
Examples:
http://madsjulia.github.io/Mads.jl/Examples/blind_source_separation
http://tensors.lanl.gov
http://tensordecompositions.github.io
ML Studies Climate: EU Summary