• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Apache Hadoop India Summit 2011 talk "Middleware Frameworks for Adaptive Executions and Visualizations of Climate and Weather Applications on Grids" by Sathish Vadhiyar
 

Apache Hadoop India Summit 2011 talk "Middleware Frameworks for Adaptive Executions and Visualizations of Climate and Weather Applications on Grids" by Sathish Vadhiyar

on

  • 1,254 views

 

Statistics

Views

Total Views
1,254
Views on SlideShare
1,104
Embed Views
150

Actions

Likes
0
Downloads
31
Comments
0

2 Embeds 150

http://d.hatena.ne.jp 149
http://search.yahoo.co.jp 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Apache Hadoop India Summit 2011 talk "Middleware Frameworks for Adaptive Executions and Visualizations of Climate and Weather Applications on Grids" by Sathish Vadhiyar Apache Hadoop India Summit 2011 talk "Middleware Frameworks for Adaptive Executions and Visualizations of Climate and Weather Applications on Grids" by Sathish Vadhiyar Presentation Transcript

    • Middleware Frameworks for Adaptive Executions and Visualizations of Climate and Weather Applications on Grids
      SathishVadhiyar
      Grid Applications Research Lab
      Supercomputer Education and Research Centre
      Indian Institute of Science
      Bangalore
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Outline
      Parallel Simulation and Visualization
      Resource Constraints
      Impact on Climate Simulations
      Adaptive Integrated Framework
      Framework
      Contradictory Objectives
      Decision Algorithm
      Steering the Visualizations
      Results
      Progress of Simulation and Visualization
      Adaptation of Parameters
      Potential for Cloud Computing
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Parallel Simulation and Visualization
      Critical climate applications like cyclone tracking require
      High-fidelity high-resolution simulation
      High-performance computations
      Massive amount of output
      On-the-fly remote visualization
      Real-time guidance to policy and decision makers
      Joint analysis by geographically distributed climate scientists
      High-performance
      simulations
      Parallel I/O
      Remote
      visualization
      DISK
      Network
      Figure: Simultaneous simulation and remote visualization using stable storage
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Resource Constraints
      • High computation rate
      • High I/O bandwidth
      • Limited network bandwidth
      • Limited storage space
      SIM
      VIS
      Simulation Process
      Visualization Process
      Stable Storage
      Network
      Figure: Illustration of resource constraints on simulation
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Impact on climate simulations
      Rapid accumulation of data in the stable storage
      Eventual unavailability of storage
      Stalling of simulation
      Low temporal resolution
      Loss of visualization
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Adaptive Integrated Framework
      • Invokes a decision algorithm periodically
      • Reacts to significantly low disk space
      APPLICATION
      MANAGER
      APPLICATION
      CONFIG
      Output Frequency
      # Processors
      Periodic Invocation
      DECISION
      ALGORITHM
      • Adapts to resource and application dynamics
      • Determine near-optimal parameters
      • Schedules climate simulation application
      • Starts, stops, restarts simulation process
      JOB HANDLER
      • Simulates climate across time steps
      • Outputs climate data to storage
      Application
      Configuration
      SIMULATION
      PROCESS
      VISUALIZATION
      PROCESS
      FRAME SENDER
      FRAME RECEIVER
      Network
      Stall if no disk space
      • Visualizes simulation output
      Storage
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Decision Algorithm
      Objectives
      Maximize rate of simulation
      Maximize temporal resolution
      Enable continuous visualization
      Ensure availability of storage
      Contradictory Objectives
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Decision Algorithm
      Input
      Simulation resolution
      Network bandwidth
      Remaining disk space
      Output
      Number of processors for simulation
      Output frequency
      Optimization Based Algorithm
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Optimization-based Approach
      Causes of faster consumption of storage space
      Faster execution time
      Limited network bandwidth
      High frequency of output
      Objectives
      Optimal processor allocation
      Best possible output frequency
      Judicious use of storage
      Maximize simulation ratewithin the constraints related to continuous visualization, acceptable output frequency, I/O bandwidth, disk space and network bandwidth
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Problem Formulation
      Objective function: minimize t
      Table: Decision Variables
      Time Constraint: Time to solve + Time to output ≤ Time to transfer
      (1)
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Constraints
      Disk Constraint: Net input to the disk ≤ Remaining disk space
      (2)
      (3)
      Bound Constraints: Bounds for t and z
      (4)
      (5)
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Experiments
      Simulation: Weather Research and Forecasting Model v3.0.1
      Visualization: VisIt v1.12.0
      Climate Application: Tracking Cyclone Aila
      Modeled area: 32x106 sq. km. from 60ºE - 120ºE and 10ºS - 40ºN
      Formed: 23th May 2009, Dissipated: 26th May 2009
      Figure: Visualization of Perturbation Pressure showing the track of Aila
      Table: Resolutions for different Pressure Values
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Experiments
      Table: Simulation and Visualization Configurations
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Faster rate of simulation
      Simulation stalls in Greedy-Threshold approach
      Simulation Progress
      Figure: For cross-continent configuration
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Visualization Progress
      Faster rate of visualization
      Lags behind in attempt to visualize every time step initially
      INCREASING LAG
      Figure: For intra-country configuration
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Less than 50% disk space used
      Higher rate of disk space consumption
      Disk Space Utilization
      Figure: For intra-country configuration
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • Adaptivity
      Figure: For inter-departmentconfiguration
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
    • February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
      Steering the Visualization
    • February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
      Steering Across the Ocean!
      Auto-changing number of procs to maintain QoS
      Changing Resolution of Simulation
      Changing Visualization Frequency
      Changing number of procs from 96 to 80
    • Ship the simulations to a cloud
      Use resource management services of clouds to find a “nearby” large storage
      This will eliminate the storage problem/constraint
      But new research challenges:
      Storage can spill over; Need to maintain metadata of storage repositories
      Simulation->Storage->Visualization will now involve multiple hops
      Hence added benefits due to large storage-as-service in cloud will have to balanced against loss in performance
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
      Potential for Clouds
    • The infrastructure has to be expanded to include multiple simultaneous multi-user visualizations of multiple independent simulations
      Such independent simulations are natural for executions on clouds.
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
      Potential for Clouds
    • To minimize lag between simulation and visualization site – choosing representative frames
      Multiple visualization-simulation framework
      Applying for other applications
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
      Future Work
    • PreetiMalakar (Phd student)
      Dr. Vijay Natarajan(Co-researcher)
      February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
      Acknowledgements
    • February 16, 2011
      Yahoo! Hadoop India Summit, Indian Institute of Science
      Thank You!