"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
Nimrod cloud
1. High-throughput eScience mixing Grids and Clouds: an experience with the Nimrod tool family Presenter: Blair Bethwaite MonasheScience and Grid Engineering Lab
2. MeSsAGE Lab team: David Abramson Colin Enticott SlavisaGaric and others... Acknowledgements NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
4. The Nimrod tool family NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
5. Parametric computing with the Nimrod tools Vary parameters Execute programs Copy code/data in/out X, Y, Z could be: Basic data types; ints, floats, strings Files Random numbers to drive Monte Carlo modelling X Y Parameter Space Solution Space Z User Job EII Cloud Workshop - AWS Intro Blair Bethwaite - MeSsAGE Lab, Monash Uni
7. From Clusters, to Grids, to Clouds Jobs / Nimrod experiment Nimrod Actuator, e.g., SGE, PBS, LSF, Condor Local Batch System NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
8. From Clusters, to Grids, to Clouds Jobs / Nimrod experiment Portal Nimrod-O/E/K Nimrod/G Actuator, e.g., Globus Servers Upper middleware Lower middleware Pilot jobs / agents Agents Grid Middleware Grid Middleware Grid Middleware Agents Grid Middleware NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
9. From Clusters, to Grids, to Clouds NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni The Grid Global utility computing mk.1-(beta) Somewhere in-between Infrastructure and Platform as-a-Service For Nimrod Increased computational scale – massively parallel New scheduling and data challenges Computational economy proposed Problems Interoperability Barriers to entry
10. From Clusters, to Grids, to Clouds NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
11. From Clusters, to Grids, to Clouds NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni Cloud opportunities for HTC Virtualisation helps interoperability and scalability Cloud bursting Scale-out to supplement locally and nationally available resources Test computational economy and scheduling, in anger Deadline driven Budget driven What’s missing? Grids provide services above IaaS E.g., you can build a grid on EC2 Grids provide job and data handling services, more like PaaS
12. From Clusters, to Grids, to Clouds def process_queue(self): """Prepare allocation of commands/agents to instances. This might mean requesting new instances from the web service and/or allocating available slots from existing instances. ""“ if not self._queued_cmds and not self.proxy: return False self._update_available_instances() req_slots = len(self._queued_cmds) new_slots = req_slots - self.free_slots num_insts = new_slots / self.slots_per_instance # if we need the proxy we might have to force # launching an instance to host it if self.proxy and num_insts < 1 br /> and len(self.instances) < 1: num_insts = 1 rsv = None ... ... if num_insts > 0: try: rsv = self.ec2conn.run_instances(self.ami_id, min_count=1, max_count=num_insts, key_name=self.ws_label, security_groups=[self.secgroup.name], instance_type=self.ec2InstanceType) except EC2ResponseError,e: if ec2.parse_response_error(e, 'Code') == br />u'InstanceLimitExceeded': self.at_instance_limit = True print "[%s] Instance limit exceeded" % self.label else: print "[%s] Error running instances:%s" % br /> (self.label, t5exc.exception()) raise if rsv: self._pending_reservations.append(rsv) ... NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
13. Integrating with IaaS Jobs / Nimrod experiment Portal Nimrod-O/E/K Nimrod/G Actuator: Globus,... Services New actuators: EC2, IBM, Azure, OCCI?,...? RESTfulIaaS API Grid Middleware VM Agents Agents VM VM Agents Agents NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
14. Integrating with IaaS Advantage: Nimrod is already a meta-scheduler Creates an ad-hoc grid dynamically overlaying the available resource pool Don’t need Grid-like job processing services to stand-up resource pool Requires explicit management of infrastructure Extra level of scheduling – when to initialise infrastructure? NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
15. Integrating with IaaS 1 2 3 NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
16. Application Examples A lot of existing grid based infrastructure So, mix it together “Mixing Grids and Clouds: High-Throughput Science Using the Nimrod Tool Family,” in Cloud Computing, vol. 0 (Springer London, 2010) Markov Chain Monte Carlo methods for recommender systems For better results, insert coins here... NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
17. Application Examples Modelling ash dispersion – NG-TEPHRA IEEE e-Science 2010 Supplement local infrastructure for deadline sensitive analysis NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
18. Work-in-progress What’s keeping me awake... Spot-price scheduling Smarter data handling Windows support On EC2 And integrating with Azure Rose NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
19. Nimrod utilising NeCTAR RC Host MeSsAGE Lab tools Dev and test environment Excess capacity supporting HTC NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
20. Thank you! Presentation by: Blair Bethwaite Researcher, Developer, SysAdmin Monash eScience and Grid Engineering Lab Feedback/queries: blair.bethwaite@monash.edu david.abramson@monash.edu NeCTAR Research Cloud Workshop Blair Bethwaite - MeSsAGE Lab, Monash Uni
Editor's Notes
Please ask questions during the talk if you have them.
Simple pleasingly-parallel computing for “legacy” (misnomer: just need existing app, Nimrod is the distributed glue that launches and contextualises each job). Onclusters, compute grids, and now clouds.Also support computational economy via economic scheduling.
Molecular docking in drug designEngineering antennae for maximum gainAirfoil optimising LD ratio
Original Nimrod also acted as the cluster management system, commercial spin-off to Enfuzion.
Nimrod/G – “G” originally stood for Globus but now more general supporting other lower level middleware, such as Condor.
Then AWS came along... suddenly public utility computing became a realityOn demand: start and stop machines any time, lead time of minutes.Self service: no lengthy email trail with your data centre admin, just make a web service call.PAYG: pay for what you use, tear it down when not needed.Think of it as a computational vending machine.
Code snippet from Nimrod EC2 actuator – bringing up your first few machines like this is cool! And incredibly easy with these APIs, and great tools like Boto.
Actuator model makes this integration relatively painless compared to an app highly dependent on higher Grid middleware functions.
Clouds provide an interesting infrastructure to supplement the usual resources available for academic computing.You can pay to get your results faster, or make them higher quality.
Probabilistic spatial and density distribution mapping of volcanic tephra, potentially useful in time sensitive scenarios, i.e., immediately preceding or following an eruption event.
