High-throughput eScience mixing Grids and Clouds:<br />an experience with the Nimrod tool family<br />Presenter:<br />Blai...
MeSsAGE Lab team:<br />David Abramson<br />Colin Enticott<br />SlavisaGaric<br />and others...<br />Acknowledgements<br />...
Agenda<br />NeCTAR Research Cloud Workshop		     Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
The Nimrod tool family<br />NeCTAR Research Cloud Workshop		     Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
Parametric computing with the Nimrod tools<br />Vary parameters<br />Execute programs<br />Copy code/data in/out<br />X, Y...
Nimrod Applications<br />messagelab.monash.edu.au/EScienceApplications<br />NeCTAR Research Cloud Workshop		     Blair Bet...
From Clusters, to Grids, to Clouds<br />Jobs / Nimrod experiment<br />Nimrod<br />Actuator, e.g., SGE, PBS, LSF, Condor<br...
From Clusters, to Grids, to Clouds<br />Jobs / Nimrod experiment<br />Portal<br />Nimrod-O/E/K<br />Nimrod/G<br />Actuator...
From Clusters, to Grids, to Clouds<br />NeCTAR Research Cloud Workshop		     Blair Bethwaite - MeSsAGE Lab, Monash Uni<br ...
From Clusters, to Grids, to Clouds<br />NeCTAR Research Cloud Workshop		     Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
From Clusters, to Grids, to Clouds<br />NeCTAR Research Cloud Workshop		     Blair Bethwaite - MeSsAGE Lab, Monash Uni<br ...
From Clusters, to Grids, to Clouds<br /> def process_queue(self):<br />        """Prepare allocation of commands/agents to...
Integrating with IaaS<br />Jobs / Nimrod experiment<br />Portal<br />Nimrod-O/E/K<br />Nimrod/G<br />Actuator: Globus,...<...
Integrating with IaaS<br />Advantage: Nimrod is already a meta-scheduler<br />Creates an ad-hoc grid dynamically overlayin...
Integrating with IaaS<br />1<br />2<br />3<br />NeCTAR Research Cloud Workshop		     Blair Bethwaite - MeSsAGE Lab, Monash...
Application Examples<br />A lot of existing grid based infrastructure<br />So, mix it together<br />“Mixing Grids and Clou...
Application Examples<br />Modelling ash dispersion – NG-TEPHRA<br />IEEE e-Science 2010<br />Supplement local infrastructu...
Work-in-progress<br />What’s keeping me awake...<br />Spot-price scheduling<br />Smarter data handling<br />Windows suppor...
Nimrod utilising NeCTAR RC<br />Host MeSsAGE Lab tools<br />Dev and test environment<br />Excess capacity<br />		supportin...
Thank you!<br />Presentation by:<br />Blair Bethwaite<br />Researcher, Developer, SysAdmin<br />Monash eScience and Grid E...
Upcoming SlideShare
Loading in...5
×

Nimrod cloud

578

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
578
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Please ask questions during the talk if you have them.
  • Simple pleasingly-parallel computing for “legacy” (misnomer: just need existing app, Nimrod is the distributed glue that launches and contextualises each job). Onclusters, compute grids, and now clouds.Also support computational economy via economic scheduling.
  • Molecular docking in drug designEngineering antennae for maximum gainAirfoil optimising LD ratio
  • Original Nimrod also acted as the cluster management system, commercial spin-off to Enfuzion.
  • Nimrod/G – “G” originally stood for Globus but now more general supporting other lower level middleware, such as Condor.
  • Then AWS came along... suddenly public utility computing became a realityOn demand: start and stop machines any time, lead time of minutes.Self service: no lengthy email trail with your data centre admin, just make a web service call.PAYG: pay for what you use, tear it down when not needed.Think of it as a computational vending machine.
  • Code snippet from Nimrod EC2 actuator – bringing up your first few machines like this is cool! And incredibly easy with these APIs, and great tools like Boto.
  • Actuator model makes this integration relatively painless compared to an app highly dependent on higher Grid middleware functions.
  • Clouds provide an interesting infrastructure to supplement the usual resources available for academic computing.You can pay to get your results faster, or make them higher quality.
  • Probabilistic spatial and density distribution mapping of volcanic tephra, potentially useful in time sensitive scenarios, i.e., immediately preceding or following an eruption event.
  • Transcript of "Nimrod cloud"

    1. 1. High-throughput eScience mixing Grids and Clouds:<br />an experience with the Nimrod tool family<br />Presenter:<br />Blair Bethwaite<br />MonasheScience and Grid Engineering Lab<br />
    2. 2. MeSsAGE Lab team:<br />David Abramson<br />Colin Enticott<br />SlavisaGaric<br />and others...<br />Acknowledgements<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    3. 3. Agenda<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    4. 4. The Nimrod tool family<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    5. 5. Parametric computing with the Nimrod tools<br />Vary parameters<br />Execute programs<br />Copy code/data in/out<br />X, Y, Z could be:<br />Basic data types; ints, floats, strings<br />Files<br />Random numbers to drive Monte Carlo modelling<br />X<br />Y<br />Parameter<br />Space<br />Solution<br />Space<br />Z<br />User Job<br />EII Cloud Workshop - AWS Intro Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    6. 6. Nimrod Applications<br />messagelab.monash.edu.au/EScienceApplications<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    7. 7. From Clusters, to Grids, to Clouds<br />Jobs / Nimrod experiment<br />Nimrod<br />Actuator, e.g., SGE, PBS, LSF, Condor<br />Local Batch System<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    8. 8. From Clusters, to Grids, to Clouds<br />Jobs / Nimrod experiment<br />Portal<br />Nimrod-O/E/K<br />Nimrod/G<br />Actuator, e.g., Globus<br />Servers<br />Upper middleware<br />Lower middleware<br />Pilot jobs / agents<br />Agents<br />Grid Middleware<br />Grid Middleware<br />Grid Middleware<br />Agents<br />Grid Middleware<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    9. 9. From Clusters, to Grids, to Clouds<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />The Grid<br />Global utility computing mk.1-(beta)<br />Somewhere in-between Infrastructure and Platform as-a-Service<br />For Nimrod<br />Increased computational scale – massively parallel<br />New scheduling and data challenges<br />Computational economy proposed<br />Problems<br />Interoperability<br />Barriers to entry<br />
    10. 10. From Clusters, to Grids, to Clouds<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    11. 11. From Clusters, to Grids, to Clouds<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />Cloud opportunities for HTC<br />Virtualisation helps interoperability and scalability<br />Cloud bursting<br />Scale-out to supplement locally and nationally available resources<br />Test computational economy and scheduling, in anger<br />Deadline driven<br />Budget driven<br />What’s missing?<br />Grids provide services above IaaS<br />E.g., you can build a grid on EC2<br />Grids provide job and data handling services, more like PaaS<br />
    12. 12. From Clusters, to Grids, to Clouds<br /> def process_queue(self):<br /> """Prepare allocation of commands/agents to instances.<br /> This might mean requesting new instances from the web service and/or<br /> allocating available slots from existing instances.<br /> ""“<br /> if not self._queued_cmds and not self.proxy:<br /> return False<br />self._update_available_instances()<br />req_slots = len(self._queued_cmds)<br />new_slots = req_slots - self.free_slots<br />num_insts = new_slots / self.slots_per_instance<br /> # if we need the proxy we might have to force<br /> # launching an instance to host it<br /> if self.proxy and num_insts < 1 <br /> and len(self.instances) < 1:<br />num_insts = 1<br />rsv = None<br /> ...<br /> ...<br /> if num_insts > 0:<br /> try:<br />rsv = self.ec2conn.run_instances(self.ami_id,<br />min_count=1, max_count=num_insts,<br />key_name=self.ws_label,<br />security_groups=[self.secgroup.name],<br />instance_type=self.ec2InstanceType)<br /> except EC2ResponseError,e:<br /> if ec2.parse_response_error(e, 'Code') == <br />u'InstanceLimitExceeded':<br />self.at_instance_limit = True<br /> print "[%s] Instance limit exceeded" % self.label<br /> else:<br /> print "[%s] Error running instances:n%s" % <br /> (self.label, t5exc.exception())<br /> raise<br /> if rsv:<br />self._pending_reservations.append(rsv)<br /> ...<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    13. 13. Integrating with IaaS<br />Jobs / Nimrod experiment<br />Portal<br />Nimrod-O/E/K<br />Nimrod/G<br />Actuator: Globus,...<br />Services<br />New actuators: EC2, IBM, Azure, OCCI?,...?<br />RESTfulIaaS API<br />Grid Middleware<br />VM<br />Agents<br />Agents<br />VM<br />VM<br />Agents<br />Agents<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    14. 14. Integrating with IaaS<br />Advantage: Nimrod is already a meta-scheduler<br />Creates an ad-hoc grid dynamically overlaying the available resource pool<br />Don’t need Grid-like job processing services to stand-up resource pool<br />Requires explicit management of infrastructure<br />Extra level of scheduling – when to initialise infrastructure?<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    15. 15. Integrating with IaaS<br />1<br />2<br />3<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    16. 16. Application Examples<br />A lot of existing grid based infrastructure<br />So, mix it together<br />“Mixing Grids and Clouds: High-Throughput Science Using the Nimrod Tool Family,” in Cloud Computing, vol. 0 (Springer London, 2010)<br />Markov Chain Monte Carlo methods for recommender systems<br />For better results, insert coins here...<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    17. 17. Application Examples<br />Modelling ash dispersion – NG-TEPHRA<br />IEEE e-Science 2010<br />Supplement local infrastructure for deadline sensitive analysis<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    18. 18. Work-in-progress<br />What’s keeping me awake...<br />Spot-price scheduling<br />Smarter data handling<br />Windows support<br />On EC2<br />And integrating with Azure<br />Rose<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    19. 19. Nimrod utilising NeCTAR RC<br />Host MeSsAGE Lab tools<br />Dev and test environment<br />Excess capacity<br /> supporting HTC<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />
    20. 20. Thank you!<br />Presentation by:<br />Blair Bethwaite<br />Researcher, Developer, SysAdmin<br />Monash eScience and Grid Engineering Lab<br />Feedback/queries:<br />blair.bethwaite@monash.edu<br />david.abramson@monash.edu<br />NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni<br />

    ×