The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big Data
•
1 like•667 views
In this Deck from the 2018 Swiss HPC Conference, Dave Turek from IBM presents: The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big Data. "There is a shift underway where HPC is beginning to be addressed with novel techniques and technologies including cognitive and analytic approaches to HPC problems and the arrival of the first quantum systems. This talk will showcase how IBM is merging cognitive, analytics, and quantum with classic simulation and modeling to create a new path for computational science." Watch the video: https://wp.me/p3RLHQ-ik7 Learn more: http://ibm.com and http://www.hpcadvisorycouncil.com/events/2018/swiss-workshop/agenda.php Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Recommended
Recommended
More Related Content
What's hot
What's hot (16)
Similar to The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big Data
Similar to The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big Data (20)
More from inside-BigData.com
More from inside-BigData.com (20)
Recently uploaded
Recently uploaded (20)
The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big Data
- 1. Data Centric Systems David Turek VP High Performance and Cognitive Computing
- 2. © 2016 International Business Machines Corporation R&D in the IT Era Theory/Knowledge Experiment Simulation Massive improvements: • Applicability & ease of use • Simulation fidelity • Scalability • Throughput thanks to Supercomputing
- 3. Data Centric Systems 3Source: Top500.org Implementing Exact-Exchange in CPMD >99% Parallel Efficiency to over 6.2M threads Studying Li-Air Batteries, 1736 atoms, 70Ry cuttof V. Weber, T. Laino, C. Bekas, A. Curioni, A. Bertsch, S. Futral IPDPS 13
- 4. Data Centric Systems 4 ACM Gordon Bell Prize 2013 14.4 PFLOP/S @73% of peak perf., with I/O 2 orders of magnitude improvement in • scale of the problem (from 128 to 15K bubbles) • time to solution Compute specifics: 13 Trillion elements, 1.2TBytes compressed I/O per time step, 6.4 M threads IBM, ETHZ, TUM, LLNL Success in Petascale computing: CFD can achieve Linpack like sustained performance
- 5. Data Centric Systems 5 ACM Gordon Bell Prize 2015 97% of sustained scalability for a fully implicit solver. 1.6M cores 3.2M MPI processes 602B DoF, IBM, UT Austin, NYU, CALTECH Success in Petascale computing: Implicit linear solvers do scale!
- 6. © 2016 International Business Machines Corporation 6 New Product Opportunistic Discovery by Humans Simulation Experiments R&D Today But we cannot beat complexity with brute force simulation. Traditional discovery has limits: We need a new, data driven, holistic approach
- 10. © 2016 International Business Machines Corporation 1 0 Companies need to easily access quickly growing and widely diverse information sources. • Highly unstructured/dark • Current human based approach not scalable Domain related inference is largely missing. Setting up and deploying the right simulations is very hard. • Human capital intensive, non scalable Internal evidence and experiments are driven primarily empirically, often brute force, and their results are isolated from wider knowledge space. Knowledge Evidence & Experiments Inference & Simulation
- 11. © 2016 International Business Machines Corporation 1 1 Create technical area specific knowledge space from all relevant sources. Link with company data. Use knowledge space to • Drastically augment internal know-how & modeling • Focus on which experiment is relevant • Embed results in knowledge base Use inference on the knowledge space & simulation on the models • To augment the knowledge space • Sharpen simulation models • Make precise decisions Cognitive Discovery Drastically accelerate pace of systematic discovery and maximize ROI for R&D Rapid and Precise Materials R&D drives new value for our clients Pharma Materials Engineering & Manufacturing Science, Products & Economics Experimental Results Knowledge Inference & Simulation Evidence & Experiments Simulation
- 12. Data Centric Systems Document Ingestion: PDF Domain Specific Knowledge Graphs Domain Specific ML + Inference NLQ + ML Driven Simulations Automatic Hypothesis Discovery Fully Automated Reasoning Fully Automated Discovery mature Ideation KNOWLEDGE EXTRACTION & REPRESENTATION INFERENCE DRIVEN SIMULATIONS AUTOMATED TECHNICAL REASONING
- 13. Data Centric Systems 1 3 Literature Review Non scalable, human based outsourcing: • Limited sources • Non-systematic; limited re-use 1 Chemical/Physical/Eng. modeling & simulations • Expert material scientists • Empirical: no inference • Trial and error based: no systematic knowledge buildup 2 Lab tests Time/money costly • Empirical (slow: many tests) • No systematic knowledge buildup & connection 3 Years Months Months Months INGESTION SIMULATION ANALYSIS
- 14. Data Centric Systems Pdf-parser: • Parses the pdf-code and presents the raw data of the pdf (text-cells, embedded images and vector-graphics in consumable format) Pdf-interpreter: • Captures ground truth by massive Crowd-sourcing big Data system • Uses HPC for ML-techniques (Deep Leaning), to train automatic annotation models Semantic-representation: • Uses HPC & Big Data systems to to obtain a semantic representation in JSON-format of the original text Billions of documents Millions of concurrent users
- 15. Data Centric Systems Weeks Deep Search Lab tests experiments data Simulation & Inference Scientific literature & internal reports Design alloys to avoid catastrophic failure that can lead to huge liabilities • Corrosion • Cracks • Special environmental and deployment conditions DAYS Knowledge space • Atomistic simulations • Deep Learning based property prediction
- 16. Data Centric Systems • Typically HPC development is focused on increased speed. • The fastest calculation is the one which you don’t run! • Can we use machine learning to make better decisions on which simulations give the most value? • Can we use machine learning to improve resolution of information? ‘Cognitive’ workflow uses 1/3 of the calculations to achieve 4 orders of magnitude resolution increase
- 17. Data Centric Systems On-prem, customer managed (Bluemix Local) IBM Cloud private X86, Power & Z X86 based systems On-prem, IBM managed Off-prem, IBM managed (Bluemix Public or Dedicated) Linux 4/11/2018IBM Confidential 17 kube- arbitrator GPFS/Parallel object store Spectrum MPI Spectrum LSF Conductor w/Spark Symphony XLc/C/Fortran Compute Accelerators (GPUs, AI, FPGA, etc.)//High Performance Network (RoCE, IB, RRC)//NVMe,Flash Math libraries ESSL, GPU, AI AI frameworks (PowerAI,DLaaS) Workflow Managers (TCaaS) HPC, AI Applications xCATProvisioning Ubiquity Storage drivers
- 18. Data Centric Systems Knowledge Space Simulation Weeks Evidence/Experiments • Supercomputing • Quantum and new computing paradigms • Inference (ML) Ingest data and create massive knowledge spaces Link evidence with knowledge spaces. Drive deep search