• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Nimrod cloud
 

Nimrod cloud

on

  • 702 views

 

Statistics

Views

Total Views
702
Views on SlideShare
702
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Please ask questions during the talk if you have them.
  • Simple pleasingly-parallel computing for “legacy” (misnomer: just need existing app, Nimrod is the distributed glue that launches and contextualises each job). Onclusters, compute grids, and now clouds.Also support computational economy via economic scheduling.
  • Molecular docking in drug designEngineering antennae for maximum gainAirfoil optimising LD ratio
  • Original Nimrod also acted as the cluster management system, commercial spin-off to Enfuzion.
  • Nimrod/G – “G” originally stood for Globus but now more general supporting other lower level middleware, such as Condor.
  • Then AWS came along... suddenly public utility computing became a realityOn demand: start and stop machines any time, lead time of minutes.Self service: no lengthy email trail with your data centre admin, just make a web service call.PAYG: pay for what you use, tear it down when not needed.Think of it as a computational vending machine.
  • Code snippet from Nimrod EC2 actuator – bringing up your first few machines like this is cool! And incredibly easy with these APIs, and great tools like Boto.
  • Actuator model makes this integration relatively painless compared to an app highly dependent on higher Grid middleware functions.
  • Clouds provide an interesting infrastructure to supplement the usual resources available for academic computing.You can pay to get your results faster, or make them higher quality.
  • Probabilistic spatial and density distribution mapping of volcanic tephra, potentially useful in time sensitive scenarios, i.e., immediately preceding or following an eruption event.

Nimrod cloud Nimrod cloud Presentation Transcript

  • High-throughput eScience mixing Grids and Clouds:
    an experience with the Nimrod tool family
    Presenter:
    Blair Bethwaite
    MonasheScience and Grid Engineering Lab
  • MeSsAGE Lab team:
    David Abramson
    Colin Enticott
    SlavisaGaric
    and others...
    Acknowledgements
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • Agenda
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • The Nimrod tool family
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • Parametric computing with the Nimrod tools
    Vary parameters
    Execute programs
    Copy code/data in/out
    X, Y, Z could be:
    Basic data types; ints, floats, strings
    Files
    Random numbers to drive Monte Carlo modelling
    X
    Y
    Parameter
    Space
    Solution
    Space
    Z
    User Job
    EII Cloud Workshop - AWS Intro Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • Nimrod Applications
    messagelab.monash.edu.au/EScienceApplications
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • From Clusters, to Grids, to Clouds
    Jobs / Nimrod experiment
    Nimrod
    Actuator, e.g., SGE, PBS, LSF, Condor
    Local Batch System
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • From Clusters, to Grids, to Clouds
    Jobs / Nimrod experiment
    Portal
    Nimrod-O/E/K
    Nimrod/G
    Actuator, e.g., Globus
    Servers
    Upper middleware
    Lower middleware
    Pilot jobs / agents
    Agents
    Grid Middleware
    Grid Middleware
    Grid Middleware
    Agents
    Grid Middleware
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • From Clusters, to Grids, to Clouds
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
    The Grid
    Global utility computing mk.1-(beta)
    Somewhere in-between Infrastructure and Platform as-a-Service
    For Nimrod
    Increased computational scale – massively parallel
    New scheduling and data challenges
    Computational economy proposed
    Problems
    Interoperability
    Barriers to entry
  • From Clusters, to Grids, to Clouds
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • From Clusters, to Grids, to Clouds
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
    Cloud opportunities for HTC
    Virtualisation helps interoperability and scalability
    Cloud bursting
    Scale-out to supplement locally and nationally available resources
    Test computational economy and scheduling, in anger
    Deadline driven
    Budget driven
    What’s missing?
    Grids provide services above IaaS
    E.g., you can build a grid on EC2
    Grids provide job and data handling services, more like PaaS
  • From Clusters, to Grids, to Clouds
    def process_queue(self):
    """Prepare allocation of commands/agents to instances.
    This might mean requesting new instances from the web service and/or
    allocating available slots from existing instances.
    ""“
    if not self._queued_cmds and not self.proxy:
    return False
    self._update_available_instances()
    req_slots = len(self._queued_cmds)
    new_slots = req_slots - self.free_slots
    num_insts = new_slots / self.slots_per_instance
    # if we need the proxy we might have to force
    # launching an instance to host it
    if self.proxy and num_insts < 1
    and len(self.instances) < 1:
    num_insts = 1
    rsv = None
    ...
    ...
    if num_insts > 0:
    try:
    rsv = self.ec2conn.run_instances(self.ami_id,
    min_count=1, max_count=num_insts,
    key_name=self.ws_label,
    security_groups=[self.secgroup.name],
    instance_type=self.ec2InstanceType)
    except EC2ResponseError,e:
    if ec2.parse_response_error(e, 'Code') ==
    u'InstanceLimitExceeded':
    self.at_instance_limit = True
    print "[%s] Instance limit exceeded" % self.label
    else:
    print "[%s] Error running instances:n%s" %
    (self.label, t5exc.exception())
    raise
    if rsv:
    self._pending_reservations.append(rsv)
    ...
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • Integrating with IaaS
    Jobs / Nimrod experiment
    Portal
    Nimrod-O/E/K
    Nimrod/G
    Actuator: Globus,...
    Services
    New actuators: EC2, IBM, Azure, OCCI?,...?
    RESTfulIaaS API
    Grid Middleware
    VM
    Agents
    Agents
    VM
    VM
    Agents
    Agents
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • Integrating with IaaS
    Advantage: Nimrod is already a meta-scheduler
    Creates an ad-hoc grid dynamically overlaying the available resource pool
    Don’t need Grid-like job processing services to stand-up resource pool
    Requires explicit management of infrastructure
    Extra level of scheduling – when to initialise infrastructure?
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • Integrating with IaaS
    1
    2
    3
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • Application Examples
    A lot of existing grid based infrastructure
    So, mix it together
    “Mixing Grids and Clouds: High-Throughput Science Using the Nimrod Tool Family,” in Cloud Computing, vol. 0 (Springer London, 2010)
    Markov Chain Monte Carlo methods for recommender systems
    For better results, insert coins here...
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • Application Examples
    Modelling ash dispersion – NG-TEPHRA
    IEEE e-Science 2010
    Supplement local infrastructure for deadline sensitive analysis
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • Work-in-progress
    What’s keeping me awake...
    Spot-price scheduling
    Smarter data handling
    Windows support
    On EC2
    And integrating with Azure
    Rose
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • Nimrod utilising NeCTAR RC
    Host MeSsAGE Lab tools
    Dev and test environment
    Excess capacity
    supporting HTC
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
  • Thank you!
    Presentation by:
    Blair Bethwaite
    Researcher, Developer, SysAdmin
    Monash eScience and Grid Engineering Lab
    Feedback/queries:
    blair.bethwaite@monash.edu
    david.abramson@monash.edu
    NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni