An Advanced Simulation


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

An Advanced Simulation

  1. 1. FLASH Future March 21, 2005 Anshu Dubey
  2. 2. FLASH3 Motivation <ul><li>The FLASH Center won’t be around forever </li></ul><ul><li>Put FLASH in a position to be a community developed and supported code </li></ul><ul><li>First opportunity to focus on design rather than capabilities </li></ul>
  3. 3. FLASH 3 Goals <ul><li>To create robust, reliable, efficient and expandable code, that stands the test of time and users </li></ul><ul><li>Build upon lessons learned in FLASH 2 </li></ul><ul><ul><li>Simplify the architecture </li></ul></ul><ul><ul><li>Decentralize data management </li></ul></ul><ul><ul><li>Define rules for inheritance and data movement </li></ul></ul><ul><ul><li>Eliminate assumptions </li></ul></ul><ul><li>Incorporate newer software engineering practices </li></ul><ul><ul><li>Design by contract / Unit test </li></ul></ul><ul><li>Exploit technologies </li></ul><ul><ul><li>Tools </li></ul></ul><ul><li>Create a community code </li></ul><ul><ul><li>Easy to add new functionality </li></ul></ul><ul><ul><li>Policy for external contributions </li></ul></ul>
  4. 4. The Design <ul><li>Great effort to get the design right </li></ul><ul><ul><li>Individual proposals </li></ul></ul><ul><ul><ul><li>Discussed, iterated upon </li></ul></ul></ul><ul><ul><ul><li>Detailed specifications worked out </li></ul></ul></ul><ul><ul><ul><li>Prototyped and tested </li></ul></ul></ul><ul><li>Keeping track of design documents </li></ul><ul><ul><li>Digital photographs of the board </li></ul></ul><ul><ul><li>CVS repository of the design specification proposals and documents </li></ul></ul><ul><ul><ul><li>The documents in the repository should eventually become basis for developers guide . </li></ul></ul></ul><ul><li>Some parts still dynamic. </li></ul>
  5. 5. Design Trade-offs <ul><li>Determine the scope </li></ul><ul><ul><li>How deep should the changes be </li></ul></ul><ul><ul><ul><li>Infrastructure </li></ul></ul></ul><ul><ul><ul><li>Algorithms </li></ul></ul></ul><ul><li>Design and Implementation Methodology </li></ul><ul><ul><li>Transition of modules/units from FLASH2 to FLASH3 </li></ul></ul><ul><ul><li>Reconcile the divergence </li></ul></ul>
  6. 6. The Implementation Plan <ul><li>Initially concentrate on the Framework </li></ul><ul><ul><li>Carried out independently of FLASH2 </li></ul></ul><ul><ul><li>No disruption for people using FLASH2 for science runs </li></ul></ul><ul><li>Move a minimal set of units from FLASH2 to FLASH3 </li></ul><ul><ul><li>Dependencies dictate the set </li></ul></ul><ul><ul><li>Those units frozen in FLASH2 </li></ul></ul><ul><ul><li>If unavoidable, changes in those units must be carefully synchronized </li></ul></ul><ul><ul><li>Evaluate the algorithms in the unit and redesign as needed </li></ul></ul><ul><li>Repeat until all units done </li></ul>
  7. 7. Unit Testing <ul><li>Goal: have each module be testable as independently as possible from the rest of the code </li></ul><ul><li>Tests should be very simple, light on memory. </li></ul><ul><li>Should eventually hook up with the FLASH test suite </li></ul><ul><li>What are obstacles? </li></ul><ul><ul><li>Most modules have some knowledge of other units </li></ul></ul><ul><ul><li>Most modules need some form of Grid </li></ul></ul>
  8. 8. Architecture: Lessons from FLASH 2 Example : Decentralize data management source mesh dbase amr PM data . amr PM func. UG UG UG data UG func source Grid AG PM func. UG UG data UG func PM data . FLASH2 FLASH3
  9. 9. IO UG AMR hdf5 netcdf hdf5 netcdf checkpoint() checkpoint() checkpoint() checkpoint() Lots of duplicate code! IO Common hdf5 netcdf checkpoint.F90 Calls generic methods open_file() write_data() Hdf5_open() ncmpi_open() Eliminate duplicate code
  10. 10. Infrastructure : Mesh Packages in Flash <ul><li>FLASH original design was Paramesh specific </li></ul><ul><ul><li>Block Structured </li></ul></ul><ul><ul><li>Fixed sized blocks </li></ul></ul><ul><ul><li>Specified at compile time </li></ul></ul><ul><ul><li>Every section of the code assumed knowledge of block size </li></ul></ul><ul><li>Removing fixed block size requirement opens the door for other mesh packages like a uniform grid, squishy grid and patched base grid </li></ul>Paramesh Relaxing the fixed block size restraint in Flash makes the code more flexible and robust, however it requires the code to be modified at the deepest levels
  11. 11. New Capabilities <ul><li>one block per proc </li></ul><ul><li>No AMR related overhead </li></ul>Uniform Grid Patch Based Mesh (planned) <ul><li>Adaptive mesh with variable block size </li></ul><ul><li>Allows mesh to “zoom” in on specific areas without having to refine surrounding areas </li></ul>
  12. 12. New Capabilities - Squishy Grid (Planned) <ul><li>Non-uniform, non-adaptive grid </li></ul><ul><li>No overhead from AMR </li></ul><ul><li>Useful in cases where high levels of resolution are needed in fixed areas of the domain </li></ul>Squishy Grid or
  13. 13. FLASH 3 Architecture Four Cornerstones Setup tool assemble simulation Config files Tell setup how to Assemble simulation Driver Organize interactions Between units Unit Architecture API Inheritance Data management
  14. 14. FLASH3 Units Driver I/O Runtime inputs Grid Profiling Runtime viz Simulation Infrastructure monitoring Hydro Burn Gravity MHD Physics
  15. 15. Unit Architecture Driver <ul><li>Top level: </li></ul><ul><li>API </li></ul><ul><li>Unit Test </li></ul>Grid Data Module block data time step etc Wrapper Kernel
  16. 16. Unit Hierarchy API/ Stubs Common API Impl API impl Wrapper kernel API impl API impl Wrapper kernel Wrapper kernel
  17. 17. Building an Application Grid mesh I/O Runtime Inputs Profiler Simulation/ setup Driver Physics
  18. 18. FLASH3 Framework <ul><li>How do you get the Framework out of FLASH 3? </li></ul><ul><ul><li>Include the source tree up to the top level of each Unit. </li></ul></ul><ul><ul><ul><li>Stub implementations </li></ul></ul></ul><ul><ul><ul><li>Config files </li></ul></ul></ul><ul><ul><li>Include setups/default </li></ul></ul><ul><ul><li>Include the setup tool </li></ul></ul><ul><ul><li>Include the Driver </li></ul></ul><ul><ul><li>Unit tests could be included, but have no meaning with stubs </li></ul></ul>
  19. 19. Move a Unit <ul><li>Create a Unit_data Fortran module to store data needed from other units in the code. </li></ul><ul><li>Create storage in Unit_data for all variables that kernel gets through either runtimeparameter or dbase calls. </li></ul><ul><li>Eliminate the “first call” constructs, move them to the Unit_init. </li></ul><ul><ul><li>Unit_init fills up Unit_data. </li></ul></ul><ul><li>Use wraper functions to fill up the remaining variables in Unit_data. </li></ul>
  20. 20. Moving a Module Example: PPM Hydro <ul><li>PPM Hydro </li></ul><ul><ul><li>And its dependencies (for example Eos) </li></ul></ul><ul><li>Reasons : </li></ul><ul><ul><li>Central Module </li></ul></ul><ul><ul><li>Majority of applications need it </li></ul></ul><ul><ul><li>Exercises the FLASH3 design </li></ul></ul><ul><ul><li>FLASH3 cannot be meaningfully tested without it </li></ul></ul><ul><li>Challenges : </li></ul><ul><ul><li>Legacy, with many assumptions </li></ul></ul><ul><ul><li>Has too much knowledge of the mesh </li></ul></ul><ul><ul><li>Very difficult to encapsulate </li></ul></ul><ul><ul><li>Eos even more challenging </li></ul></ul>
  21. 21. FLASH 3 Status <ul><li>The framework works, and can be separated out </li></ul><ul><li>The following units are implemented: </li></ul><ul><ul><li>Grid : can switch between PM2, PM3 and UG </li></ul></ul><ul><ul><ul><li>PM2 and UG work in parallel </li></ul></ul></ul><ul><ul><li>Driver, RuntimeInputs, Profiler </li></ul></ul><ul><ul><li>Hydro, Gamma Eos, Materials </li></ul></ul><ul><ul><li>Dummy MPI </li></ul></ul><ul><ul><li>IO </li></ul></ul><ul><li>Simple application work </li></ul><ul><li>Compiles and runs on multiple platforms </li></ul><ul><li>Test suite is up and running </li></ul><ul><li>Will be released to users within the center soon. </li></ul><ul><li>A Beta version might be released by the end of the year. </li></ul>
  22. 22. Impact on user <ul><li>We hope it won’t be painful </li></ul><ul><li>Lots of Kernels will be the same as in FLASH2 </li></ul><ul><li>A user, who has modified source files in setup: </li></ul><ul><ul><li>should be able to read Unit API documentation to clearly determine how to make the code compatible with FLASH3 or </li></ul></ul><ul><ul><li>Look at the new source file implementation for how to start on converting to FLASH3 </li></ul></ul><ul><li>If you’re in the process of developing a deep functional module for FLASH2 that you want to contribute, talk to us now </li></ul>
  23. 23. Tools <ul><li>The Diagnostic Tools </li></ul><ul><ul><li>Memory diagnostic tool </li></ul></ul><ul><ul><li>Profiling tools </li></ul></ul><ul><ul><li>Runtime viz </li></ul></ul><ul><li>Post Processing Tools </li></ul><ul><ul><li>Fidlr </li></ul></ul><ul><ul><li>FLASH view </li></ul></ul><ul><li>Documentation Tools </li></ul><ul><ul><li>Benchmark management tool </li></ul></ul><ul><ul><li>Robodoc for web interfaced documentation </li></ul></ul>
  24. 24. Tools Snapshots
  25. 25. The Future <ul><li>The next few steps </li></ul><ul><ul><li>Benchmark and profile the performance </li></ul></ul><ul><ul><li>Move efficient units in FLASH 2 to FLASH 3 </li></ul></ul><ul><ul><li>Evaluate algorithms in units that are not </li></ul></ul><ul><ul><ul><li>With some help select the appropriate ones </li></ul></ul></ul><ul><ul><ul><li>Implement them </li></ul></ul></ul><ul><li>The still open major design issues </li></ul><ul><ul><li>Gravity </li></ul></ul><ul><ul><li>Particles </li></ul></ul><ul><li>Look at load balancing with non-uniform work per block </li></ul>
  26. 26. Load Balancing <ul><li>Load balanced in number of blocks </li></ul><ul><li>Work concentrated in few blocks </li></ul><ul><li>CPU snapshot shows one processor idle for the most part </li></ul><ul><li>Work weighted load balancing needed. </li></ul>Work in blocks Distribution of blocks
  27. 27. … which brings us to <ul><li>Discussion and Questions </li></ul>