SlideShare a Scribd company logo
What Is It Good For?!
                      Real World Examples of
                       Vampir Usage at IU
                                   Robert Henschel
                                rhensche@indiana.edu

Thanks to Scott Teige, Don Berry, Scott Michael, Huian Li, Judy Qiu for providing traces.

                                        May 2009
Contents
• IPP – Repeated Execution of an Application
• Runtime Anomaly on the iDataPlex (Tachyon)
• Cell B.E. Tracing
• Particle Swarm Optimizer – Memory Tracing
• Finding an I/O Bottleneck (EMAN)
• Cluster Challenge
   – WPP, GAMESS
• Tracing on Windows
• Swarp – Pthread Tracing
• MD Simulation



                                          Robert Henschel
What is this all about
• Provide a feeling for what Vampir can be used for
• Show a few of Vampirs features
• Raise the awareness that tracing software is a
  complex undertaking




                                             Robert Henschel
IPP - Repeated Execution of an Application
• Image analysis pipeline
• Every module/binary is called a few hundred times
  during a single run
• Setting VT_FILE_UNIQUE=yes allows for normal
  execution of the pipeline, while traces are produced
• We ended up with 640 traces, that we did not want to
  look at
• otfprofile was used in bash script to retrieve the
  profile of every trace, and then combine to a profile of
  the entire run




                                                   Robert Henschel
IPP - Repeated Execution of an Application
 ppImage_103.0.def.z
 ppImage_103.0.marker.z
 ppImage_103.1.events.z
 ppImage_103.otf
 ppImage_103.otf_collop.csv
 ppImage_103.otf_data.csv
 ppImage_103.otf_func.csv
 ppImage_103.otf_p2p.csv
 ppImage_103.otf_result.aux
 ppImage_103.otf_result.dvi
 ppImage_103.otf_result.log
 ppImage_103.otf_result.ps
 ppImage_103.otf_result.tex

                                        Robert Henschel
IPP - Repeated Execution of an Application




                                        Robert Henschel
IPP - Repeated Execution of an Application




                                        Robert Henschel
IPP - Repeated Execution of an Application




                                        Robert Henschel
IPP - Repeated Execution of an Application




                                        Robert Henschel
IPP - Repeated Execution of an Application




                                        Robert Henschel
IPP - Repeated Execution of an Application




                                        Robert Henschel
IPP - Repeated Execution of an Application




                                        Robert Henschel
IPP - Repeated Execution of an Application




                                        Robert Henschel
IPP - Repeated Execution of an Application




                                        Robert Henschel
IPP - Repeated Execution of an Application




                                        Robert Henschel
IPP - Repeated Execution of an Application




                                        Robert Henschel
IPP - Repeated Execution of an Application




                                        Robert Henschel
IBM iDataPlex Testing
• We are testing an IBM iDataPlex system
   – 84 dual quad-core node cluster
   – Infiniband interconnect
• Tachyon, a part of the SPEC MPI2007 benchmark suite
  had a strange runtime behavior
   – 100 vs. 200 seconds
• Tracing the application, made the problem go away,
  stable runtime at 110 seconds

• There will be a deeper interaction of OpenMPI and
  VampirTrace in the future, OpenMPI 1.3 now contains
  VampirTrace


                                              Robert Henschel
IBM iDataPlex Testing




                        Robert Henschel
IBM iDataPlex Testing




                        Robert Henschel
IBM iDataPlex Testing




                        Robert Henschel
IBM iDataPlex Testing




                        Robert Henschel
IBM iDataPlex Testing




                        Robert Henschel
Cell B.E. Tracing
• Code exists as serial version, MPI version and as Cell
  B.E. version
• Great to be able to use the same tracing tool on both
  platforms and for different parallelization strategies
• We used a beta version of VampirTrace for Cell B.E.




                                                 Robert Henschel
Cell B.E. Tracing




                    Robert Henschel
Cell B.E. Tracing




                    Robert Henschel
Cell B.E. Tracing




                    Robert Henschel
Cell B.E. Tracing




                    Robert Henschel
Cell B.E. Tracing




                    Robert Henschel
Cell B.E. Tracing




                    Robert Henschel
Cell B.E. Tracing




                    Robert Henschel
Cell B.E. Tracing




                    Robert Henschel
Cell B.E. Tracing




                    Robert Henschel
Cell B.E. Tracing




                    Robert Henschel
Cell B.E. Tracing




                    Robert Henschel
Cell B.E. Tracing




                    Robert Henschel
Cell B.E. Tracing




                    Robert Henschel
PSO – Memory Footprint
• Determining the memory footprint of a particle swarm
  optimizer




                                                Robert Henschel
PSO – Memory Footprint




                         Robert Henschel
PSO – Memory Footprint




                         Robert Henschel
EMAN Tracing
• Application ran very slow on regular hardware, NFS
  mounted file system
• Application was faster on Lustre
• Application was really fast on shared memory filesystem
• Suspected I/O problem
• However, the actual I/O problem was not visible in
  Vampir, as this had to do with file locking




                                                Robert Henschel
EMAN Tracing




               Robert Henschel
EMAN Tracing




               Robert Henschel
EMAN Tracing




               Robert Henschel
Cluster Challenge
• WPP (Wave Propagation Program)
• Vampir helped to understand the code (few comments,
  not sure what the code does, and how it does it)
• PAPI counter helped to show that on X86, peak floating
  point performance was already achieved, thus, not
  much room for tuning, besides rewriting the algorithm




                                                Robert Henschel
Cluster Challenge




                    Robert Henschel
Cluster Challenge




                    Robert Henschel
Cluster Challenge




                    Robert Henschel
Cluster Challenge




                    Robert Henschel
Cluster Challenge




                    Robert Henschel
Cluster Challenge




                    Robert Henschel
Cluster Challenge




                    Robert Henschel
Cluster Challenge




                    Robert Henschel
Cluster Challenge
• GAMESS
• Problem with MPI communication, while socket
  communication worked just fine




                                                 Robert Henschel
Tracing on Windows
• On Windows, the trace creation is handled by MS MPI
• Unfortunately, not a lot of details are provided, thus ,the
  traces look a bit different to traces that are created on
  Linux
   – MPI collective operations are not marked as such!
• However, on Windows, you are able to trace C#
  application in addition to C/C++/Fortran applications
   – And possibly other applications, that are build on top
     of MS MPI




                                                    Robert Henschel
Tracing on Windows




                     Robert Henschel
Tracing on Windows




                     Robert Henschel
Tracing on Windows




                     Robert Henschel
Tracing on Windows




                     Robert Henschel
Tracing on Windows




                     Robert Henschel
Tracing on Windows




                     Robert Henschel
Tracing on Windows




                     Robert Henschel
Tracing on Windows




                     Robert Henschel
Tracing on Windows




                     Robert Henschel
Swarp - Pthread Tracing
• SWARP can be run as a Pthread parallel application
• Tracing of Pthread parallel applications is a new feature
  of the latest VampirTrace version
• Enabled by default, if Pthread flag is detected on the
  compile and link commands, but can be forced as well

• Compiling with -DVTRACE_PTHREAD and including
  vt_user.h in the appropriate file
   – Will trace the overhead of Pthread functions




                                                  Robert Henschel
Swarp - Pthread Tracing




                          Robert Henschel
Swarp - Pthread Tracing




                          Robert Henschel
Swarp - Pthread Tracing




                          Robert Henschel
Swarp - Pthread Tracing




                          Robert Henschel
Swarp - Pthread Tracing




                          Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid
• MD application, part of the research of physics
  professor Chuck Horowitz
• Used for studying dense nuclear matter in supernovae,
  white dwarf and neutron stars
• In production use on a daily basis
• Exists as serial, MPI, OpenMP, Hybrid and
  MDGRAPE-2 application
• MDGRAPE-2 tracing is not supported out of the box,
  but could be implemented using manual instrumentation




                                              Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel
MD Application, Serial, OpenMP, MPI, Hybrid




                                       Robert Henschel

More Related Content

More from PTIHPA

Github:fi Presentation
Github:fi PresentationGithub:fi Presentation
Github:fi PresentationPTIHPA
 
2010 05 hands_on
2010 05 hands_on2010 05 hands_on
2010 05 hands_onPTIHPA
 
Trace Visualization
Trace VisualizationTrace Visualization
Trace VisualizationPTIHPA
 
2010 vampir workshop_iu_configuration
2010 vampir workshop_iu_configuration2010 vampir workshop_iu_configuration
2010 vampir workshop_iu_configurationPTIHPA
 
Overview: Event Based Program Analysis
Overview: Event Based Program AnalysisOverview: Event Based Program Analysis
Overview: Event Based Program AnalysisPTIHPA
 
Switc Hpa
Switc HpaSwitc Hpa
Switc HpaPTIHPA
 
Statewide It Robert Henschel
Statewide It Robert HenschelStatewide It Robert Henschel
Statewide It Robert HenschelPTIHPA
 
5 Vampir Configuration At IU
5 Vampir Configuration At IU5 Vampir Configuration At IU
5 Vampir Configuration At IUPTIHPA
 
2 Vampir Trace Visualization
2 Vampir Trace Visualization2 Vampir Trace Visualization
2 Vampir Trace VisualizationPTIHPA
 
1 Vampir Overview
1 Vampir Overview1 Vampir Overview
1 Vampir OverviewPTIHPA
 
GeneIndex: an open source parallel program for enumerating and locating words...
GeneIndex: an open source parallel program for enumerating and locating words...GeneIndex: an open source parallel program for enumerating and locating words...
GeneIndex: an open source parallel program for enumerating and locating words...PTIHPA
 
Implementing 3D SPHARM Surfaces Registration on Cell B.E. Processor
Implementing 3D SPHARM Surfaces Registration on Cell B.E. ProcessorImplementing 3D SPHARM Surfaces Registration on Cell B.E. Processor
Implementing 3D SPHARM Surfaces Registration on Cell B.E. ProcessorPTIHPA
 
Big Iron and Parallel Processing, USArray Data Processing Workshop
Big Iron and Parallel Processing, USArray Data Processing WorkshopBig Iron and Parallel Processing, USArray Data Processing Workshop
Big Iron and Parallel Processing, USArray Data Processing WorkshopPTIHPA
 

More from PTIHPA (13)

Github:fi Presentation
Github:fi PresentationGithub:fi Presentation
Github:fi Presentation
 
2010 05 hands_on
2010 05 hands_on2010 05 hands_on
2010 05 hands_on
 
Trace Visualization
Trace VisualizationTrace Visualization
Trace Visualization
 
2010 vampir workshop_iu_configuration
2010 vampir workshop_iu_configuration2010 vampir workshop_iu_configuration
2010 vampir workshop_iu_configuration
 
Overview: Event Based Program Analysis
Overview: Event Based Program AnalysisOverview: Event Based Program Analysis
Overview: Event Based Program Analysis
 
Switc Hpa
Switc HpaSwitc Hpa
Switc Hpa
 
Statewide It Robert Henschel
Statewide It Robert HenschelStatewide It Robert Henschel
Statewide It Robert Henschel
 
5 Vampir Configuration At IU
5 Vampir Configuration At IU5 Vampir Configuration At IU
5 Vampir Configuration At IU
 
2 Vampir Trace Visualization
2 Vampir Trace Visualization2 Vampir Trace Visualization
2 Vampir Trace Visualization
 
1 Vampir Overview
1 Vampir Overview1 Vampir Overview
1 Vampir Overview
 
GeneIndex: an open source parallel program for enumerating and locating words...
GeneIndex: an open source parallel program for enumerating and locating words...GeneIndex: an open source parallel program for enumerating and locating words...
GeneIndex: an open source parallel program for enumerating and locating words...
 
Implementing 3D SPHARM Surfaces Registration on Cell B.E. Processor
Implementing 3D SPHARM Surfaces Registration on Cell B.E. ProcessorImplementing 3D SPHARM Surfaces Registration on Cell B.E. Processor
Implementing 3D SPHARM Surfaces Registration on Cell B.E. Processor
 
Big Iron and Parallel Processing, USArray Data Processing Workshop
Big Iron and Parallel Processing, USArray Data Processing WorkshopBig Iron and Parallel Processing, USArray Data Processing Workshop
Big Iron and Parallel Processing, USArray Data Processing Workshop
 

Recently uploaded

The Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official PublicationThe Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official Publication
Delapenabediema
 
Supporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptxSupporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptx
Jisc
 
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXXPhrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
MIRIAMSALINAS13
 
Unit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdfUnit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdf
Thiyagu K
 
The geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideasThe geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideas
GeoBlogs
 
Operation Blue Star - Saka Neela Tara
Operation Blue Star   -  Saka Neela TaraOperation Blue Star   -  Saka Neela Tara
Operation Blue Star - Saka Neela Tara
Balvir Singh
 
The French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free downloadThe French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free download
Vivekanand Anglo Vedic Academy
 
Guidance_and_Counselling.pdf B.Ed. 4th Semester
Guidance_and_Counselling.pdf B.Ed. 4th SemesterGuidance_and_Counselling.pdf B.Ed. 4th Semester
Guidance_and_Counselling.pdf B.Ed. 4th Semester
Atul Kumar Singh
 
A Strategic Approach: GenAI in Education
A Strategic Approach: GenAI in EducationA Strategic Approach: GenAI in Education
A Strategic Approach: GenAI in Education
Peter Windle
 
Thesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.pptThesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.ppt
EverAndrsGuerraGuerr
 
The Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdfThe Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdf
kaushalkr1407
 
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup   New Member Orientation and Q&A (May 2024).pdfWelcome to TechSoup   New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
TechSoup
 
Polish students' mobility in the Czech Republic
Polish students' mobility in the Czech RepublicPolish students' mobility in the Czech Republic
Polish students' mobility in the Czech Republic
Anna Sz.
 
Home assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdfHome assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdf
Tamralipta Mahavidyalaya
 
1.4 modern child centered education - mahatma gandhi-2.pptx
1.4 modern child centered education - mahatma gandhi-2.pptx1.4 modern child centered education - mahatma gandhi-2.pptx
1.4 modern child centered education - mahatma gandhi-2.pptx
JosvitaDsouza2
 
Additional Benefits for Employee Website.pdf
Additional Benefits for Employee Website.pdfAdditional Benefits for Employee Website.pdf
Additional Benefits for Employee Website.pdf
joachimlavalley1
 
Instructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptxInstructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptx
Jheel Barad
 
The approach at University of Liverpool.pptx
The approach at University of Liverpool.pptxThe approach at University of Liverpool.pptx
The approach at University of Liverpool.pptx
Jisc
 
678020731-Sumas-y-Restas-Para-Colorear.pdf
678020731-Sumas-y-Restas-Para-Colorear.pdf678020731-Sumas-y-Restas-Para-Colorear.pdf
678020731-Sumas-y-Restas-Para-Colorear.pdf
CarlosHernanMontoyab2
 
Sha'Carri Richardson Presentation 202345
Sha'Carri Richardson Presentation 202345Sha'Carri Richardson Presentation 202345
Sha'Carri Richardson Presentation 202345
beazzy04
 

Recently uploaded (20)

The Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official PublicationThe Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official Publication
 
Supporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptxSupporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptx
 
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXXPhrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
 
Unit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdfUnit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdf
 
The geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideasThe geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideas
 
Operation Blue Star - Saka Neela Tara
Operation Blue Star   -  Saka Neela TaraOperation Blue Star   -  Saka Neela Tara
Operation Blue Star - Saka Neela Tara
 
The French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free downloadThe French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free download
 
Guidance_and_Counselling.pdf B.Ed. 4th Semester
Guidance_and_Counselling.pdf B.Ed. 4th SemesterGuidance_and_Counselling.pdf B.Ed. 4th Semester
Guidance_and_Counselling.pdf B.Ed. 4th Semester
 
A Strategic Approach: GenAI in Education
A Strategic Approach: GenAI in EducationA Strategic Approach: GenAI in Education
A Strategic Approach: GenAI in Education
 
Thesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.pptThesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.ppt
 
The Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdfThe Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdf
 
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup   New Member Orientation and Q&A (May 2024).pdfWelcome to TechSoup   New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
 
Polish students' mobility in the Czech Republic
Polish students' mobility in the Czech RepublicPolish students' mobility in the Czech Republic
Polish students' mobility in the Czech Republic
 
Home assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdfHome assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdf
 
1.4 modern child centered education - mahatma gandhi-2.pptx
1.4 modern child centered education - mahatma gandhi-2.pptx1.4 modern child centered education - mahatma gandhi-2.pptx
1.4 modern child centered education - mahatma gandhi-2.pptx
 
Additional Benefits for Employee Website.pdf
Additional Benefits for Employee Website.pdfAdditional Benefits for Employee Website.pdf
Additional Benefits for Employee Website.pdf
 
Instructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptxInstructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptx
 
The approach at University of Liverpool.pptx
The approach at University of Liverpool.pptxThe approach at University of Liverpool.pptx
The approach at University of Liverpool.pptx
 
678020731-Sumas-y-Restas-Para-Colorear.pdf
678020731-Sumas-y-Restas-Para-Colorear.pdf678020731-Sumas-y-Restas-Para-Colorear.pdf
678020731-Sumas-y-Restas-Para-Colorear.pdf
 
Sha'Carri Richardson Presentation 202345
Sha'Carri Richardson Presentation 202345Sha'Carri Richardson Presentation 202345
Sha'Carri Richardson Presentation 202345
 

4 HPA Examples Of Vampir Usage