SlideShare a Scribd company logo
Tuning Parallel Applications to
Accelerate Scientific Discoveries

              Robert Henschel
           rhensche@indiana.edu


                October 2009
Contents
•   PTI / High Performance Applications
•   Performance of Scientific Codes
•   IU and TeraGrid Compute Resources
•   Optimizing for IU's HPC Systems
•   Using TeraGrid HPC Systems
•   HPA is Here to Help




                                          Robert Henschel
What this talk will be about
• Making you aware of compute resources that you
  can use for your work, to make you more
  productive.

• Introducing the High Performance Applications
  group and how we can help get research done
  faster.

• Give you examples of what we have done for
  researchers to make them more competitive in their
  field.



                                            Robert Henschel
PTI and High Performance Applications
• Pervasive Technology Institute
   – Develop and deliver innovative information technology
     to advance research, education, industry and society.
   – School of Informatics
   – School of Law
   – University Information Technology Services

• High Performance Applications
  – Part of the Digital Science Center of PTI
  – Part of the Research Technologies of UITS
  – Seven people that help IU researchers make efficient
    use of IU and TeraGrid compute resources

                                                Robert Henschel
Performance of Scientific Codes
• Supercomputing, or High Performance Computing (HPC),
  is not just for computer geeks!

• Performance for computer scientists
   – Amdahls law and scalability
   – Efficient usage of functional units of processors
   – Optimally using memory bandwidth
   – Trying to avoid I/O as much as possible

• Performance for researchers
   – When do I get the answer to my problem?
   – When does my job run and when is it done?

                                                   Robert Henschel
IU and TeraGrid Compute Resources
• Two HPC systems at IU
   – BigRed    30 TFLOPS            (3000 cores)
   – Quarry     7 TFLOPS            (1000 cores)

• Several special purpose systems
   – Small Cell B.E. Cluster
   – MDGRAPE-2 machine

• Several storage resources
   – IU Data Capacitor
   – GPFS, RFS, HPSS

• Policy of open access to compute resources
                                               Robert Henschel
IU and TeraGrid Compute Resources cont'd




                                     Robert Henschel
IU and TeraGrid Compute Resources cont'd
• TeraGrid
   – NSF funded HPC systems and support infrastructure
   – 11 resource providers
   – More than 1,500 TFLOPS      (150,000 cores)

• Central allocation and support structure




                                              Robert Henschel
Optimizing for IU's HPC Systems
• Help researchers access the central systems and
  determine what system to use
• Install and optimize applications
• Provide guidance on compiler and library optimization
• Help with job submission, especially running many
  thousands of jobs




                                                Robert Henschel
Using TeraGrid HPC Systems
• Low barrier of entry

• Identify if a problem and workflow will work on the
  TeraGrid
• Get a startup allocation
• Use it and identify if it is worth pursuing this further
• Submit a full allocation request




                                                      Robert Henschel
Contents – HPA is Here to Help
• HPA is Here to Help
  – What We Do

• Recent Examples
  – Integrating HPC Systems into an Electron
    Microscope Workflow
  – Migrating Research in Gas Giant Planets from IU
    to TeraGrid HPC Systems
  – Developing Computational Models to Predict
    Drug-Drug Interactions



                                           Robert Henschel
What We Do
• Consulting about HPC system usage
  – From start to finish
  – Optimize source code for architectures

• Help with TeraGrid allocation proposals
• Adapting and creating workflows for new environments
• Consulting for grant proposals




                                               Robert Henschel
HPC Systems and an Electron Microscope
General Case
  – Users have an instrument that produces a lot of data
    on a daily basis
  – This data needs to be stored and analyzed

Electron Microscope in Simon Hall (IU Bloomington)
   – Microscope stores data on a Windows workstation
   – Researcher does quality checks on local workstation
   – IU Data Capacitor links workstations, IU HPC systems
     and the IU long term archive together




                                                Robert Henschel
Gas Giant Planets on the TeraGrid
General Case
  – Users have a set workflow for analyzing data
  – Locally available compute resources are not big
    enough to keep up with demand

Understanding Gas Giant Planets
  – IDL is used to visualize simulation data
     • Commercial software, IU Astronomy has a license
  – Simulation application needs to run on a large shared
    memory system
  – TeraGrid and IU Data Capacitor tie this workflow
    together


                                                Robert Henschel
Predicting Drug-Drug Interactions
General Case
  – Researchers implement proof of concept research
    algorithms
  – Scaling from proof of concept to production science is
    difficult
  – The ability to add HPC expertise to grant proposals will
    make the proposal more competitive

Computational Models to Predict Drug-Drug Interactions
  – Drug exposure model developed in R
  – Scaling to real world data sets not possible without
    using HPC systems
  – Porting to C and running on UITS hardware

                                                 Robert Henschel
What this talk was about
• Made you aware of compute resources that you can
  use for your work, to make you more productive.

• Introduced the High Performance Applications
  group and how we can help get research done
  faster.

• Gave you examples of what we have done for
  researchers to make them more competitive in their
  field.




                                            Robert Henschel
Acknowledgments
This material is based upon work supported by the National Science
Foundation under Grant Numbers 0116050 and 0521433. Any
opinions, findings and conclusions or recommendations expressed in
this material are those of the author and do not necessarily reflect the
views of the National Science Foundation (NSF).

This work was support in part by the Indiana Metabolomics and
Cytomics Initiative (METACyt). METACyt is supported in part by Lilly
Endowment, Inc.

This work was support in part by the Indiana Genomics Initiative. The
Indiana Genomics Initiative of Indiana University is supported in part by
Lilly Endowment, Inc.

This work was supported in part by Shared University Research grants
from IBM, Inc. to Indiana University.

                                                             Robert Henschel

More Related Content

Viewers also liked

Ed tech story book ...my community.1
Ed tech story book ...my community.1Ed tech story book ...my community.1
Ed tech story book ...my community.1
Carrie Chang
 
GeneIndex: an open source parallel program for enumerating and locating words...
GeneIndex: an open source parallel program for enumerating and locating words...GeneIndex: an open source parallel program for enumerating and locating words...
GeneIndex: an open source parallel program for enumerating and locating words...PTIHPA
 
Aca Talent
Aca TalentAca Talent
Aca Talent
dsargeant
 
5 Vampir Configuration At IU
5 Vampir Configuration At IU5 Vampir Configuration At IU
5 Vampir Configuration At IUPTIHPA
 
Big Iron and Parallel Processing, USArray Data Processing Workshop
Big Iron and Parallel Processing, USArray Data Processing WorkshopBig Iron and Parallel Processing, USArray Data Processing Workshop
Big Iron and Parallel Processing, USArray Data Processing WorkshopPTIHPA
 
Github:fi Presentation
Github:fi PresentationGithub:fi Presentation
Github:fi PresentationPTIHPA
 
2010 05 hands_on
2010 05 hands_on2010 05 hands_on
2010 05 hands_onPTIHPA
 
A Common Sense Approach Electronic
A Common Sense Approach   ElectronicA Common Sense Approach   Electronic
A Common Sense Approach Electronic
Regina Henderson
 
2010 vampir workshop_iu_configuration
2010 vampir workshop_iu_configuration2010 vampir workshop_iu_configuration
2010 vampir workshop_iu_configurationPTIHPA
 
Trace Visualization
Trace VisualizationTrace Visualization
Trace VisualizationPTIHPA
 
Air Traffic
Air TrafficAir Traffic
Air Traffichumair73
 
Appraisers Direct, Inc.
Appraisers Direct, Inc.Appraisers Direct, Inc.
Appraisers Direct, Inc.
ClintCornett
 
How to Win the Moment in Real Time Events
How to Win the Moment in Real Time EventsHow to Win the Moment in Real Time Events
How to Win the Moment in Real Time Events
Tammy Gordon
 
1 Vampir Overview
1 Vampir Overview1 Vampir Overview
1 Vampir OverviewPTIHPA
 
Overview: Event Based Program Analysis
Overview: Event Based Program AnalysisOverview: Event Based Program Analysis
Overview: Event Based Program AnalysisPTIHPA
 
3 Vampir Trace In Detail
3 Vampir Trace In Detail3 Vampir Trace In Detail
3 Vampir Trace In DetailPTIHPA
 

Viewers also liked (18)

Ed tech story book ...my community.1
Ed tech story book ...my community.1Ed tech story book ...my community.1
Ed tech story book ...my community.1
 
GeneIndex: an open source parallel program for enumerating and locating words...
GeneIndex: an open source parallel program for enumerating and locating words...GeneIndex: an open source parallel program for enumerating and locating words...
GeneIndex: an open source parallel program for enumerating and locating words...
 
Aca Talent
Aca TalentAca Talent
Aca Talent
 
5 Vampir Configuration At IU
5 Vampir Configuration At IU5 Vampir Configuration At IU
5 Vampir Configuration At IU
 
Big Iron and Parallel Processing, USArray Data Processing Workshop
Big Iron and Parallel Processing, USArray Data Processing WorkshopBig Iron and Parallel Processing, USArray Data Processing Workshop
Big Iron and Parallel Processing, USArray Data Processing Workshop
 
Ciclismo Neiva
Ciclismo NeivaCiclismo Neiva
Ciclismo Neiva
 
Github:fi Presentation
Github:fi PresentationGithub:fi Presentation
Github:fi Presentation
 
2010 05 hands_on
2010 05 hands_on2010 05 hands_on
2010 05 hands_on
 
A Common Sense Approach Electronic
A Common Sense Approach   ElectronicA Common Sense Approach   Electronic
A Common Sense Approach Electronic
 
critical thinking
critical thinkingcritical thinking
critical thinking
 
2010 vampir workshop_iu_configuration
2010 vampir workshop_iu_configuration2010 vampir workshop_iu_configuration
2010 vampir workshop_iu_configuration
 
Trace Visualization
Trace VisualizationTrace Visualization
Trace Visualization
 
Air Traffic
Air TrafficAir Traffic
Air Traffic
 
Appraisers Direct, Inc.
Appraisers Direct, Inc.Appraisers Direct, Inc.
Appraisers Direct, Inc.
 
How to Win the Moment in Real Time Events
How to Win the Moment in Real Time EventsHow to Win the Moment in Real Time Events
How to Win the Moment in Real Time Events
 
1 Vampir Overview
1 Vampir Overview1 Vampir Overview
1 Vampir Overview
 
Overview: Event Based Program Analysis
Overview: Event Based Program AnalysisOverview: Event Based Program Analysis
Overview: Event Based Program Analysis
 
3 Vampir Trace In Detail
3 Vampir Trace In Detail3 Vampir Trace In Detail
3 Vampir Trace In Detail
 

Similar to Statewide It Robert Henschel

Intel life sciences_personalizedmedicine_stanford biomed 052214 dist
Intel life sciences_personalizedmedicine_stanford biomed 052214 distIntel life sciences_personalizedmedicine_stanford biomed 052214 dist
Intel life sciences_personalizedmedicine_stanford biomed 052214 dist
Ketan Paranjape
 
big_data_casestudies_2.ppt
big_data_casestudies_2.pptbig_data_casestudies_2.ppt
big_data_casestudies_2.ppt
vishal choudhary
 
HPE and Hortonworks join forces to Deliver Healthcare Transformation
HPE and Hortonworks join forces to Deliver Healthcare TransformationHPE and Hortonworks join forces to Deliver Healthcare Transformation
HPE and Hortonworks join forces to Deliver Healthcare Transformation
Hortonworks
 
HPC at NIBR
HPC at NIBRHPC at NIBR
HPC at NIBR
inside-BigData.com
 
High Performance Data Analytics and a Java Grande Run Time
High Performance Data Analytics and a Java Grande Run TimeHigh Performance Data Analytics and a Java Grande Run Time
High Performance Data Analytics and a Java Grande Run Time
Geoffrey Fox
 
Dell High-Performance Computing solutions: Enable innovations, outperform exp...
Dell High-Performance Computing solutions: Enable innovations, outperform exp...Dell High-Performance Computing solutions: Enable innovations, outperform exp...
Dell High-Performance Computing solutions: Enable innovations, outperform exp...
Dell World
 
Using The Hadoop Ecosystem to Drive Healthcare Innovation
Using The Hadoop Ecosystem to Drive Healthcare InnovationUsing The Hadoop Ecosystem to Drive Healthcare Innovation
Using The Hadoop Ecosystem to Drive Healthcare Innovation
Dan Wellisch
 
Data-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and CloudData-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and Cloud
Ola Spjuth
 
Challenges and Issues of Next Cloud Computing Platforms
Challenges and Issues of Next Cloud Computing PlatformsChallenges and Issues of Next Cloud Computing Platforms
Challenges and Issues of Next Cloud Computing Platforms
Frederic Desprez
 
OpenStack at SJTU: Predictive Data Mining in Clinical Medicine with Dynamical...
OpenStack at SJTU: Predictive Data Mining in Clinical Medicine with Dynamical...OpenStack at SJTU: Predictive Data Mining in Clinical Medicine with Dynamical...
OpenStack at SJTU: Predictive Data Mining in Clinical Medicine with Dynamical...
Shuquan Huang
 
Enterprise data science at scale
Enterprise data science at scaleEnterprise data science at scale
Enterprise data science at scale
Carolyn Duby
 
Analyzing Big Data in Medicine with Virtual Research Environments and Microse...
Analyzing Big Data in Medicine with Virtual Research Environments and Microse...Analyzing Big Data in Medicine with Virtual Research Environments and Microse...
Analyzing Big Data in Medicine with Virtual Research Environments and Microse...
Ola Spjuth
 
Big Data
Big Data Big Data
The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...
The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...
The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...
inside-BigData.com
 
Tragedy of the Data Commons (ODSC-East, 2021)
Tragedy of the Data Commons (ODSC-East, 2021)Tragedy of the Data Commons (ODSC-East, 2021)
Tragedy of the Data Commons (ODSC-East, 2021)
James Hendler
 
How HPC and large-scale data analytics are transforming experimental science
How HPC and large-scale data analytics are transforming experimental scienceHow HPC and large-scale data analytics are transforming experimental science
How HPC and large-scale data analytics are transforming experimental science
inside-BigData.com
 
Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...
Ola Spjuth
 
High Performance Computing and the Opportunity with Cognitive Technology
 High Performance Computing and the Opportunity with Cognitive Technology High Performance Computing and the Opportunity with Cognitive Technology
High Performance Computing and the Opportunity with Cognitive Technology
IBM Watson
 
HPC Resource Accounting
HPC Resource AccountingHPC Resource Accounting
HPC Resource Accounting
Ken Schumacher
 
Collins seattle-2014-final
Collins seattle-2014-finalCollins seattle-2014-final
Collins seattle-2014-final
inside-BigData.com
 

Similar to Statewide It Robert Henschel (20)

Intel life sciences_personalizedmedicine_stanford biomed 052214 dist
Intel life sciences_personalizedmedicine_stanford biomed 052214 distIntel life sciences_personalizedmedicine_stanford biomed 052214 dist
Intel life sciences_personalizedmedicine_stanford biomed 052214 dist
 
big_data_casestudies_2.ppt
big_data_casestudies_2.pptbig_data_casestudies_2.ppt
big_data_casestudies_2.ppt
 
HPE and Hortonworks join forces to Deliver Healthcare Transformation
HPE and Hortonworks join forces to Deliver Healthcare TransformationHPE and Hortonworks join forces to Deliver Healthcare Transformation
HPE and Hortonworks join forces to Deliver Healthcare Transformation
 
HPC at NIBR
HPC at NIBRHPC at NIBR
HPC at NIBR
 
High Performance Data Analytics and a Java Grande Run Time
High Performance Data Analytics and a Java Grande Run TimeHigh Performance Data Analytics and a Java Grande Run Time
High Performance Data Analytics and a Java Grande Run Time
 
Dell High-Performance Computing solutions: Enable innovations, outperform exp...
Dell High-Performance Computing solutions: Enable innovations, outperform exp...Dell High-Performance Computing solutions: Enable innovations, outperform exp...
Dell High-Performance Computing solutions: Enable innovations, outperform exp...
 
Using The Hadoop Ecosystem to Drive Healthcare Innovation
Using The Hadoop Ecosystem to Drive Healthcare InnovationUsing The Hadoop Ecosystem to Drive Healthcare Innovation
Using The Hadoop Ecosystem to Drive Healthcare Innovation
 
Data-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and CloudData-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and Cloud
 
Challenges and Issues of Next Cloud Computing Platforms
Challenges and Issues of Next Cloud Computing PlatformsChallenges and Issues of Next Cloud Computing Platforms
Challenges and Issues of Next Cloud Computing Platforms
 
OpenStack at SJTU: Predictive Data Mining in Clinical Medicine with Dynamical...
OpenStack at SJTU: Predictive Data Mining in Clinical Medicine with Dynamical...OpenStack at SJTU: Predictive Data Mining in Clinical Medicine with Dynamical...
OpenStack at SJTU: Predictive Data Mining in Clinical Medicine with Dynamical...
 
Enterprise data science at scale
Enterprise data science at scaleEnterprise data science at scale
Enterprise data science at scale
 
Analyzing Big Data in Medicine with Virtual Research Environments and Microse...
Analyzing Big Data in Medicine with Virtual Research Environments and Microse...Analyzing Big Data in Medicine with Virtual Research Environments and Microse...
Analyzing Big Data in Medicine with Virtual Research Environments and Microse...
 
Big Data
Big Data Big Data
Big Data
 
The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...
The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...
The Transformation of HPC: Simulation and Cognitive Methods in the Era of Big...
 
Tragedy of the Data Commons (ODSC-East, 2021)
Tragedy of the Data Commons (ODSC-East, 2021)Tragedy of the Data Commons (ODSC-East, 2021)
Tragedy of the Data Commons (ODSC-East, 2021)
 
How HPC and large-scale data analytics are transforming experimental science
How HPC and large-scale data analytics are transforming experimental scienceHow HPC and large-scale data analytics are transforming experimental science
How HPC and large-scale data analytics are transforming experimental science
 
Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...
 
High Performance Computing and the Opportunity with Cognitive Technology
 High Performance Computing and the Opportunity with Cognitive Technology High Performance Computing and the Opportunity with Cognitive Technology
High Performance Computing and the Opportunity with Cognitive Technology
 
HPC Resource Accounting
HPC Resource AccountingHPC Resource Accounting
HPC Resource Accounting
 
Collins seattle-2014-final
Collins seattle-2014-finalCollins seattle-2014-final
Collins seattle-2014-final
 

More from PTIHPA

2010 02 instrumentation_and_runtime_measurement
2010 02 instrumentation_and_runtime_measurement2010 02 instrumentation_and_runtime_measurement
2010 02 instrumentation_and_runtime_measurementPTIHPA
 
2010 03 papi_indiana
2010 03 papi_indiana2010 03 papi_indiana
2010 03 papi_indianaPTIHPA
 
Switc Hpa
Switc HpaSwitc Hpa
Switc HpaPTIHPA
 
2 Vampir Trace Visualization
2 Vampir Trace Visualization2 Vampir Trace Visualization
2 Vampir Trace VisualizationPTIHPA
 
4 HPA Examples Of Vampir Usage
4 HPA Examples Of Vampir Usage4 HPA Examples Of Vampir Usage
4 HPA Examples Of Vampir UsagePTIHPA
 
Implementing 3D SPHARM Surfaces Registration on Cell B.E. Processor
Implementing 3D SPHARM Surfaces Registration on Cell B.E. ProcessorImplementing 3D SPHARM Surfaces Registration on Cell B.E. Processor
Implementing 3D SPHARM Surfaces Registration on Cell B.E. ProcessorPTIHPA
 

More from PTIHPA (6)

2010 02 instrumentation_and_runtime_measurement
2010 02 instrumentation_and_runtime_measurement2010 02 instrumentation_and_runtime_measurement
2010 02 instrumentation_and_runtime_measurement
 
2010 03 papi_indiana
2010 03 papi_indiana2010 03 papi_indiana
2010 03 papi_indiana
 
Switc Hpa
Switc HpaSwitc Hpa
Switc Hpa
 
2 Vampir Trace Visualization
2 Vampir Trace Visualization2 Vampir Trace Visualization
2 Vampir Trace Visualization
 
4 HPA Examples Of Vampir Usage
4 HPA Examples Of Vampir Usage4 HPA Examples Of Vampir Usage
4 HPA Examples Of Vampir Usage
 
Implementing 3D SPHARM Surfaces Registration on Cell B.E. Processor
Implementing 3D SPHARM Surfaces Registration on Cell B.E. ProcessorImplementing 3D SPHARM Surfaces Registration on Cell B.E. Processor
Implementing 3D SPHARM Surfaces Registration on Cell B.E. Processor
 

Recently uploaded

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Thierry Lestable
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
DianaGray10
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
Sri Ambati
 
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
Prayukth K V
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Jeffrey Haguewood
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
91mobiles
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Alan Dix
 
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMsTo Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
Paul Groth
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Tobias Schneck
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
Fwdays
 
ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User Group
CatarinaPereira64715
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
Alison B. Lowndes
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
Cheryl Hung
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
Guy Korland
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
Laura Byrne
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
Product School
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Product School
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Product School
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
Thijs Feryn
 

Recently uploaded (20)

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
 
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
 
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMsTo Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
 
ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User Group
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
 

Statewide It Robert Henschel

  • 1. Tuning Parallel Applications to Accelerate Scientific Discoveries Robert Henschel rhensche@indiana.edu October 2009
  • 2. Contents • PTI / High Performance Applications • Performance of Scientific Codes • IU and TeraGrid Compute Resources • Optimizing for IU's HPC Systems • Using TeraGrid HPC Systems • HPA is Here to Help Robert Henschel
  • 3. What this talk will be about • Making you aware of compute resources that you can use for your work, to make you more productive. • Introducing the High Performance Applications group and how we can help get research done faster. • Give you examples of what we have done for researchers to make them more competitive in their field. Robert Henschel
  • 4. PTI and High Performance Applications • Pervasive Technology Institute – Develop and deliver innovative information technology to advance research, education, industry and society. – School of Informatics – School of Law – University Information Technology Services • High Performance Applications – Part of the Digital Science Center of PTI – Part of the Research Technologies of UITS – Seven people that help IU researchers make efficient use of IU and TeraGrid compute resources Robert Henschel
  • 5. Performance of Scientific Codes • Supercomputing, or High Performance Computing (HPC), is not just for computer geeks! • Performance for computer scientists – Amdahls law and scalability – Efficient usage of functional units of processors – Optimally using memory bandwidth – Trying to avoid I/O as much as possible • Performance for researchers – When do I get the answer to my problem? – When does my job run and when is it done? Robert Henschel
  • 6. IU and TeraGrid Compute Resources • Two HPC systems at IU – BigRed 30 TFLOPS (3000 cores) – Quarry 7 TFLOPS (1000 cores) • Several special purpose systems – Small Cell B.E. Cluster – MDGRAPE-2 machine • Several storage resources – IU Data Capacitor – GPFS, RFS, HPSS • Policy of open access to compute resources Robert Henschel
  • 7. IU and TeraGrid Compute Resources cont'd Robert Henschel
  • 8. IU and TeraGrid Compute Resources cont'd • TeraGrid – NSF funded HPC systems and support infrastructure – 11 resource providers – More than 1,500 TFLOPS (150,000 cores) • Central allocation and support structure Robert Henschel
  • 9. Optimizing for IU's HPC Systems • Help researchers access the central systems and determine what system to use • Install and optimize applications • Provide guidance on compiler and library optimization • Help with job submission, especially running many thousands of jobs Robert Henschel
  • 10. Using TeraGrid HPC Systems • Low barrier of entry • Identify if a problem and workflow will work on the TeraGrid • Get a startup allocation • Use it and identify if it is worth pursuing this further • Submit a full allocation request Robert Henschel
  • 11. Contents – HPA is Here to Help • HPA is Here to Help – What We Do • Recent Examples – Integrating HPC Systems into an Electron Microscope Workflow – Migrating Research in Gas Giant Planets from IU to TeraGrid HPC Systems – Developing Computational Models to Predict Drug-Drug Interactions Robert Henschel
  • 12. What We Do • Consulting about HPC system usage – From start to finish – Optimize source code for architectures • Help with TeraGrid allocation proposals • Adapting and creating workflows for new environments • Consulting for grant proposals Robert Henschel
  • 13. HPC Systems and an Electron Microscope General Case – Users have an instrument that produces a lot of data on a daily basis – This data needs to be stored and analyzed Electron Microscope in Simon Hall (IU Bloomington) – Microscope stores data on a Windows workstation – Researcher does quality checks on local workstation – IU Data Capacitor links workstations, IU HPC systems and the IU long term archive together Robert Henschel
  • 14. Gas Giant Planets on the TeraGrid General Case – Users have a set workflow for analyzing data – Locally available compute resources are not big enough to keep up with demand Understanding Gas Giant Planets – IDL is used to visualize simulation data • Commercial software, IU Astronomy has a license – Simulation application needs to run on a large shared memory system – TeraGrid and IU Data Capacitor tie this workflow together Robert Henschel
  • 15. Predicting Drug-Drug Interactions General Case – Researchers implement proof of concept research algorithms – Scaling from proof of concept to production science is difficult – The ability to add HPC expertise to grant proposals will make the proposal more competitive Computational Models to Predict Drug-Drug Interactions – Drug exposure model developed in R – Scaling to real world data sets not possible without using HPC systems – Porting to C and running on UITS hardware Robert Henschel
  • 16. What this talk was about • Made you aware of compute resources that you can use for your work, to make you more productive. • Introduced the High Performance Applications group and how we can help get research done faster. • Gave you examples of what we have done for researchers to make them more competitive in their field. Robert Henschel
  • 17. Acknowledgments This material is based upon work supported by the National Science Foundation under Grant Numbers 0116050 and 0521433. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation (NSF). This work was support in part by the Indiana Metabolomics and Cytomics Initiative (METACyt). METACyt is supported in part by Lilly Endowment, Inc. This work was support in part by the Indiana Genomics Initiative. The Indiana Genomics Initiative of Indiana University is supported in part by Lilly Endowment, Inc. This work was supported in part by Shared University Research grants from IBM, Inc. to Indiana University. Robert Henschel