Middleware Frameworks for Adaptive Executions and Visualizations of Climate and Weather Applications on Grids<br />Sathish...
Outline<br />Parallel Simulation and Visualization<br />Resource Constraints<br />Impact on Climate Simulations<br />Adapt...
Parallel Simulation and Visualization<br />Critical climate applications like cyclone tracking require<br />High-fidelity ...
Resource Constraints<br /><ul><li>High computation rate
High I/O bandwidth
Limited network bandwidth
Limited storage space</li></ul>SIM<br />VIS<br />Simulation Process<br />Visualization Process<br />Stable Storage<br />Ne...
Impact on climate simulations<br />Rapid accumulation of data in the stable storage<br />Eventual unavailability of storag...
Adaptive Integrated Framework<br /><ul><li> Invokes a decision algorithm periodically
 Reacts to significantly low disk space</li></ul>APPLICATION<br />MANAGER<br />APPLICATION<br />CONFIG<br />Output Frequen...
 Determine near-optimal parameters
 Schedules climate simulation application
 Starts, stops, restarts simulation process</li></ul>JOB HANDLER<br /><ul><li> Simulates climate across time steps
 Outputs climate data to storage</li></ul>Application<br />Configuration<br />SIMULATION<br />PROCESS<br />VISUALIZATION<b...
Decision Algorithm<br />Objectives<br />Maximize rate of simulation<br />Maximize temporal resolution<br />Enable continuo...
Decision Algorithm<br />Input<br />Simulation resolution<br />Network bandwidth<br />Remaining disk space<br />Output<br /...
Optimization-based Approach<br />Causes of faster consumption of storage space<br /> Faster execution time<br /> Limited n...
Problem Formulation<br />Objective function: minimize t<br />Table: Decision Variables<br />Time Constraint: Time to solve...
Constraints<br />Disk Constraint: Net input to the disk ≤ Remaining disk space<br />(2)<br />(3)<br />Bound Constraints: B...
Experiments<br />Simulation: Weather Research and Forecasting Model v3.0.1<br />Visualization: VisIt v1.12.0<br />Climate ...
Experiments<br />Table: Simulation and Visualization Configurations<br />February 16, 2011<br />Yahoo! Hadoop India Summit...
Faster rate of simulation<br />Simulation stalls in Greedy-Threshold approach<br />Simulation Progress<br />Figure: For cr...
Visualization Progress<br />Faster rate of visualization<br />Lags behind in attempt to visualize every time step initiall...
Less than 50% disk space used<br />Higher rate of disk space consumption<br />Disk Space Utilization<br />Figure: For intr...
Upcoming SlideShare
Loading in...5
×

Apache Hadoop India Summit 2011 talk "Middleware Frameworks for Adaptive Executions and Visualizations of Climate and Weather Applications on Grids" by Sathish Vadhiyar

1,078

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,078
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
35
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Apache Hadoop India Summit 2011 talk "Middleware Frameworks for Adaptive Executions and Visualizations of Climate and Weather Applications on Grids" by Sathish Vadhiyar

  1. 1. Middleware Frameworks for Adaptive Executions and Visualizations of Climate and Weather Applications on Grids<br />SathishVadhiyar<br />Grid Applications Research Lab<br />Supercomputer Education and Research Centre<br />Indian Institute of Science<br />Bangalore<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  2. 2. Outline<br />Parallel Simulation and Visualization<br />Resource Constraints<br />Impact on Climate Simulations<br />Adaptive Integrated Framework<br />Framework<br />Contradictory Objectives<br />Decision Algorithm<br />Steering the Visualizations<br />Results<br />Progress of Simulation and Visualization<br />Adaptation of Parameters<br />Potential for Cloud Computing<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  3. 3. Parallel Simulation and Visualization<br />Critical climate applications like cyclone tracking require<br />High-fidelity high-resolution simulation<br />High-performance computations<br />Massive amount of output<br />On-the-fly remote visualization<br />Real-time guidance to policy and decision makers<br />Joint analysis by geographically distributed climate scientists<br />High-performance<br />simulations<br />Parallel I/O<br />Remote<br />visualization<br />DISK<br />Network<br />Figure: Simultaneous simulation and remote visualization using stable storage<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  4. 4. Resource Constraints<br /><ul><li>High computation rate
  5. 5. High I/O bandwidth
  6. 6. Limited network bandwidth
  7. 7. Limited storage space</li></ul>SIM<br />VIS<br />Simulation Process<br />Visualization Process<br />Stable Storage<br />Network<br />Figure: Illustration of resource constraints on simulation<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  8. 8. Impact on climate simulations<br />Rapid accumulation of data in the stable storage<br />Eventual unavailability of storage<br />Stalling of simulation<br />Low temporal resolution<br />Loss of visualization<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  9. 9. Adaptive Integrated Framework<br /><ul><li> Invokes a decision algorithm periodically
  10. 10. Reacts to significantly low disk space</li></ul>APPLICATION<br />MANAGER<br />APPLICATION<br />CONFIG<br />Output Frequency<br /># Processors<br />Periodic Invocation<br />DECISION<br />ALGORITHM<br /><ul><li> Adapts to resource and application dynamics
  11. 11. Determine near-optimal parameters
  12. 12. Schedules climate simulation application
  13. 13. Starts, stops, restarts simulation process</li></ul>JOB HANDLER<br /><ul><li> Simulates climate across time steps
  14. 14. Outputs climate data to storage</li></ul>Application<br />Configuration<br />SIMULATION<br />PROCESS<br />VISUALIZATION<br />PROCESS<br />FRAME SENDER<br />FRAME RECEIVER<br />Network<br />Stall if no disk space<br /><ul><li> Visualizes simulation output</li></ul>Storage<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  15. 15. Decision Algorithm<br />Objectives<br />Maximize rate of simulation<br />Maximize temporal resolution<br />Enable continuous visualization<br />Ensure availability of storage<br />Contradictory Objectives<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  16. 16. Decision Algorithm<br />Input<br />Simulation resolution<br />Network bandwidth<br />Remaining disk space<br />Output<br />Number of processors for simulation<br />Output frequency<br />Optimization Based Algorithm<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  17. 17. Optimization-based Approach<br />Causes of faster consumption of storage space<br /> Faster execution time<br /> Limited network bandwidth<br /> High frequency of output<br />Objectives<br /> Optimal processor allocation<br /> Best possible output frequency<br /> Judicious use of storage<br />Maximize simulation ratewithin the constraints related to continuous visualization, acceptable output frequency, I/O bandwidth, disk space and network bandwidth<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  18. 18. Problem Formulation<br />Objective function: minimize t<br />Table: Decision Variables<br />Time Constraint: Time to solve + Time to output ≤ Time to transfer<br />(1)<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  19. 19. Constraints<br />Disk Constraint: Net input to the disk ≤ Remaining disk space<br />(2)<br />(3)<br />Bound Constraints: Bounds for t and z<br />(4)<br />(5)<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  20. 20. Experiments<br />Simulation: Weather Research and Forecasting Model v3.0.1<br />Visualization: VisIt v1.12.0<br />Climate Application: Tracking Cyclone Aila<br />Modeled area: 32x106 sq. km. from 60ºE - 120ºE and 10ºS - 40ºN<br />Formed: 23th May 2009, Dissipated: 26th May 2009<br />Figure: Visualization of Perturbation Pressure showing the track of Aila<br />Table: Resolutions for different Pressure Values <br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  21. 21. Experiments<br />Table: Simulation and Visualization Configurations<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  22. 22. Faster rate of simulation<br />Simulation stalls in Greedy-Threshold approach<br />Simulation Progress<br />Figure: For cross-continent configuration<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  23. 23. Visualization Progress<br />Faster rate of visualization<br />Lags behind in attempt to visualize every time step initially<br />INCREASING LAG<br />Figure: For intra-country configuration<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  24. 24. Less than 50% disk space used<br />Higher rate of disk space consumption<br />Disk Space Utilization<br />Figure: For intra-country configuration<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  25. 25. Adaptivity<br />Figure: For inter-departmentconfiguration<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />
  26. 26. February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />Steering the Visualization<br />
  27. 27. February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />Steering Across the Ocean!<br />Auto-changing number of procs to maintain QoS<br />Changing Resolution of Simulation<br />Changing Visualization Frequency<br />Changing number of procs from 96 to 80<br />
  28. 28. Ship the simulations to a cloud<br />Use resource management services of clouds to find a “nearby” large storage<br />This will eliminate the storage problem/constraint<br />But new research challenges:<br />Storage can spill over; Need to maintain metadata of storage repositories<br />Simulation->Storage->Visualization will now involve multiple hops<br />Hence added benefits due to large storage-as-service in cloud will have to balanced against loss in performance<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />Potential for Clouds<br />
  29. 29. The infrastructure has to be expanded to include multiple simultaneous multi-user visualizations of multiple independent simulations<br />Such independent simulations are natural for executions on clouds.<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />Potential for Clouds<br />
  30. 30. To minimize lag between simulation and visualization site – choosing representative frames<br />Multiple visualization-simulation framework<br />Applying for other applications<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />Future Work<br />
  31. 31. PreetiMalakar (Phd student)<br />Dr. Vijay Natarajan(Co-researcher)<br />February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />Acknowledgements<br />
  32. 32. February 16, 2011<br />Yahoo! Hadoop India Summit, Indian Institute of Science<br />Thank You!<br />
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×