Your SlideShare is downloading. ×
  • Like
Apache Hadoop India Summit 2011 talk "Middleware Frameworks for Adaptive Executions and Visualizations of Climate and Weather Applications on Grids" by Sathish Vadhiyar
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Apache Hadoop India Summit 2011 talk "Middleware Frameworks for Adaptive Executions and Visualizations of Climate and Weather Applications on Grids" by Sathish Vadhiyar

  • 1,033 views
Published

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,033
On SlideShare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
31
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Middleware Frameworks for Adaptive Executions and Visualizations of Climate and Weather Applications on Grids
    SathishVadhiyar
    Grid Applications Research Lab
    Supercomputer Education and Research Centre
    Indian Institute of Science
    Bangalore
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 2. Outline
    Parallel Simulation and Visualization
    Resource Constraints
    Impact on Climate Simulations
    Adaptive Integrated Framework
    Framework
    Contradictory Objectives
    Decision Algorithm
    Steering the Visualizations
    Results
    Progress of Simulation and Visualization
    Adaptation of Parameters
    Potential for Cloud Computing
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 3. Parallel Simulation and Visualization
    Critical climate applications like cyclone tracking require
    High-fidelity high-resolution simulation
    High-performance computations
    Massive amount of output
    On-the-fly remote visualization
    Real-time guidance to policy and decision makers
    Joint analysis by geographically distributed climate scientists
    High-performance
    simulations
    Parallel I/O
    Remote
    visualization
    DISK
    Network
    Figure: Simultaneous simulation and remote visualization using stable storage
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 4. Resource Constraints
    • High computation rate
    • 5. High I/O bandwidth
    • 6. Limited network bandwidth
    • 7. Limited storage space
    SIM
    VIS
    Simulation Process
    Visualization Process
    Stable Storage
    Network
    Figure: Illustration of resource constraints on simulation
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 8. Impact on climate simulations
    Rapid accumulation of data in the stable storage
    Eventual unavailability of storage
    Stalling of simulation
    Low temporal resolution
    Loss of visualization
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 9. Adaptive Integrated Framework
    • Invokes a decision algorithm periodically
    • 10. Reacts to significantly low disk space
    APPLICATION
    MANAGER
    APPLICATION
    CONFIG
    Output Frequency
    # Processors
    Periodic Invocation
    DECISION
    ALGORITHM
    • Adapts to resource and application dynamics
    • 11. Determine near-optimal parameters
    • 12. Schedules climate simulation application
    • 13. Starts, stops, restarts simulation process
    JOB HANDLER
    • Simulates climate across time steps
    • 14. Outputs climate data to storage
    Application
    Configuration
    SIMULATION
    PROCESS
    VISUALIZATION
    PROCESS
    FRAME SENDER
    FRAME RECEIVER
    Network
    Stall if no disk space
    • Visualizes simulation output
    Storage
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 15. Decision Algorithm
    Objectives
    Maximize rate of simulation
    Maximize temporal resolution
    Enable continuous visualization
    Ensure availability of storage
    Contradictory Objectives
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 16. Decision Algorithm
    Input
    Simulation resolution
    Network bandwidth
    Remaining disk space
    Output
    Number of processors for simulation
    Output frequency
    Optimization Based Algorithm
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 17. Optimization-based Approach
    Causes of faster consumption of storage space
    Faster execution time
    Limited network bandwidth
    High frequency of output
    Objectives
    Optimal processor allocation
    Best possible output frequency
    Judicious use of storage
    Maximize simulation ratewithin the constraints related to continuous visualization, acceptable output frequency, I/O bandwidth, disk space and network bandwidth
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 18. Problem Formulation
    Objective function: minimize t
    Table: Decision Variables
    Time Constraint: Time to solve + Time to output ≤ Time to transfer
    (1)
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 19. Constraints
    Disk Constraint: Net input to the disk ≤ Remaining disk space
    (2)
    (3)
    Bound Constraints: Bounds for t and z
    (4)
    (5)
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 20. Experiments
    Simulation: Weather Research and Forecasting Model v3.0.1
    Visualization: VisIt v1.12.0
    Climate Application: Tracking Cyclone Aila
    Modeled area: 32x106 sq. km. from 60ºE - 120ºE and 10ºS - 40ºN
    Formed: 23th May 2009, Dissipated: 26th May 2009
    Figure: Visualization of Perturbation Pressure showing the track of Aila
    Table: Resolutions for different Pressure Values
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 21. Experiments
    Table: Simulation and Visualization Configurations
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 22. Faster rate of simulation
    Simulation stalls in Greedy-Threshold approach
    Simulation Progress
    Figure: For cross-continent configuration
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 23. Visualization Progress
    Faster rate of visualization
    Lags behind in attempt to visualize every time step initially
    INCREASING LAG
    Figure: For intra-country configuration
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 24. Less than 50% disk space used
    Higher rate of disk space consumption
    Disk Space Utilization
    Figure: For intra-country configuration
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 25. Adaptivity
    Figure: For inter-departmentconfiguration
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
  • 26. February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
    Steering the Visualization
  • 27. February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
    Steering Across the Ocean!
    Auto-changing number of procs to maintain QoS
    Changing Resolution of Simulation
    Changing Visualization Frequency
    Changing number of procs from 96 to 80
  • 28. Ship the simulations to a cloud
    Use resource management services of clouds to find a “nearby” large storage
    This will eliminate the storage problem/constraint
    But new research challenges:
    Storage can spill over; Need to maintain metadata of storage repositories
    Simulation->Storage->Visualization will now involve multiple hops
    Hence added benefits due to large storage-as-service in cloud will have to balanced against loss in performance
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
    Potential for Clouds
  • 29. The infrastructure has to be expanded to include multiple simultaneous multi-user visualizations of multiple independent simulations
    Such independent simulations are natural for executions on clouds.
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
    Potential for Clouds
  • 30. To minimize lag between simulation and visualization site – choosing representative frames
    Multiple visualization-simulation framework
    Applying for other applications
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
    Future Work
  • 31. PreetiMalakar (Phd student)
    Dr. Vijay Natarajan(Co-researcher)
    February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
    Acknowledgements
  • 32. February 16, 2011
    Yahoo! Hadoop India Summit, Indian Institute of Science
    Thank You!