Grid on Demand

805 views

Published on

Slides of the Presentation (30 july 2010) of Grid on Demand

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
805
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Grid on Demand

  1. 1. Welcome Grid on Demand Willem Toorop and Alain van Hoof {wtoorop,ahoof}@os3.nl June 30, 2010Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 1 / 39
  2. 2. Introduction Research Question Research Question Can Grid computing be offered as a Cloud service and what type of applications benefit from this?Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 2 / 39
  3. 3. Introduction The Grid and The Cloud The Grid and The Cloud: Sharing Resources Sharing Resources to minimizing idle resources Resource Capacity Goods Processor Flops Compute Hard-disk memory Bytes Storage Network Bytes/sec Communication Bandwidth Grid Cloud (IaaS) Distribution By Collaboration By Cloud Provider Resource Type Heterogeneous Homogeneous Abstraction Middleware (open standards) Virtualization Sharing Guarantee Security infrastructure Financial infraWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 3 / 39
  4. 4. Introduction The Grid and The Cloud The Grid and The Cloud: Dividing the Resources VO Grid User User Cloud Grid Grid User User User User User VOWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 4 / 39
  5. 5. A Grid in the Cloud Urgent Computing A Grid in The Cloud: Urgent Computing GRID on Demand Using the elasticity and scalability of the Cloud and providing a Grid interface, current e-Science (computationally intensive science) applications can use a Grid on Demand when there is a high demand for resources for a short period of time. Example UrbanFlood (http://urbanflood.eu) wants to calculate flooding predictions when actual flooding happens, Grid on Demand can provide this.Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 5 / 39
  6. 6. A Grid in the Cloud Grid on Demand Grid on Demand Job Submission Client OS: Ubuntu 10.04 Grid: Globus 4.2.1 PBS: Torque 2.6.8 Compute Nodes Head Node Grid Interface: Globus Toolkit 4.2 (open standard, widely used) Queue Manager (Distribution of jobs to nodes): Torque 2.6.2 Amazon EC2 Cloud Provider (de-facto standard with open API)Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 6 / 39
  7. 7. Implementation What makes the Grid, the Grid What makes the Grid, the Grid Typical Grid usage example spanning multiple organizations Organization B ta Compute st 1 o f da acc 1 2 te 12X e dat ss use mpu a o Organization C to c Organization A use my data to compute X Grid store data Scheduler Storage User use 2 nd 1 ata to co 2 of da ss d mpu 1 ta te X acce 2 Compute Organization DWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 7 / 39
  8. 8. Implementation What makes the Grid, the Grid Proxy certificates for delegation The Grid is a collaboration between multiple organizations. Grid users need to be authorized to use remote resources Those resources should be authorized to perform actions on behalf of the Grid user Proxy certificates enable delegation of authorityWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 8 / 39
  9. 9. Implementation What makes the Grid, the Grid Grid user create proxy Scheduler compute X create cert. request sign this Compute Okay compute XWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39
  10. 10. Implementation What makes the Grid, the Grid Grid user create proxy Scheduler compute X create cert. request sign this Compute Okay compute X Virtual Organizations are based on delegation by proxy certificates.Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39
  11. 11. Implementation What makes the Grid, the Grid Grid user Proxy certificates allow for dynamic resource allocation, create proxy Scheduler But the hosts still need a CA signed certificate. compute X This is a problem when hostname’s create cert. are assigned dynamically! request sign this Compute Okay compute XWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39
  12. 12. Implementation What makes the Grid, the Grid Grid user Proxy certificates allow for dynamic resource allocation, create proxy Scheduler But the hosts still need a CA signed certificate. compute X This is a problem when hostname’s create cert. are assigned dynamically! request sign this Compute Fixed CA in image? Okay compute XWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39
  13. 13. Implementation What makes the Grid, the Grid Grid user Proxy certificates allow for dynamic resource allocation, create proxy Scheduler But the hosts still need a CA signed certificate. compute X This is a problem when hostname’s create cert. are assigned dynamically! request sign this Compute Fixed CA in image? Okay compute X Dynamic CA on every Launch?Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39
  14. 14. Implementation Amazon Elastic Block Store Easy Image management with Amazon Elastic Block Store configure launch instance IA Running (A, S) Pending VS ← S (IA , VS , H) term inate p sto Shutting-down Stopping Pending Terminated rt sta IA inate term create AMI A Stopped (A , S ) S ← VS (IA , VS ) Life cycle of Amazon Machine Images using Elastic Block Storage A = Amazon Machine Image, S = Snapshot, I = Instance, V = Volume, H = HostnameWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 10 / 39
  15. 15. Implementation EBS for configuration stages EBS for configuration stages configure configure launch provide Amazon Access Credentials instance IA Running (configure cluster) Running (A, S) VS ← S (IA , VS , H) (IA , VS , H) sh shar sh wn for ng ut for utdo shutdown shutdown for reuse for reuse do ing ari start start wn sh launch IA launch IA VS ← S VS ← S Stopped Stopped Stopped (IA , VS ) (IA , VS ) (IA , VS ) create AMI A create AMI A create AMI A S ← VS S ← VS S ← VS (A , S ) (A , S ) (A , S )Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 11 / 39
  16. 16. Web Interface Launching Grid on DemandWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 12 / 39
  17. 17. Web Interface Launching Grid on DemandWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 13 / 39
  18. 18. Web Interface Launching Grid on DemandWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 14 / 39
  19. 19. Web Interface Launching Grid on DemandWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 15 / 39
  20. 20. Web Interface Launching Grid on DemandWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 16 / 39
  21. 21. Web Interface Launching Grid on DemandWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 17 / 39
  22. 22. Web Interface Launching Grid on DemandWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 18 / 39
  23. 23. Web Interface Launching Grid on DemandWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 19 / 39
  24. 24. Web Interface Configuring the clusterWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 20 / 39
  25. 25. Web Interface Configuring the clusterWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 21 / 39
  26. 26. Web Interface Configuring the clusterWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 22 / 39
  27. 27. Web Interface Configuring the clusterWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 23 / 39
  28. 28. Web Interface Real Grid certificate usageWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 24 / 39
  29. 29. Web Interface Real Grid certificate usageWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 25 / 39
  30. 30. Web Interface Real Grid certificate usageWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 26 / 39
  31. 31. Web Interface Elasticity of the clusterWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 27 / 39
  32. 32. Web Interface Elasticity of the clusterWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 28 / 39
  33. 33. Web Interface Elasticity of the clusterWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 29 / 39
  34. 34. Web Interface Saving your settingsWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 30 / 39
  35. 35. Web Interface Saving your settingsWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 31 / 39
  36. 36. Web Interface Saving your settingsWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 32 / 39
  37. 37. “Real use” and Performance WS-VLAM and WAVE “Real use” and Performance test To test the possibilities and performance of Grid on Demand a Biomedical Application and a workflow manager are usedWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39
  38. 38. “Real use” and Performance WS-VLAM and WAVE “Real use” and Performance test To test the possibilities and performance of Grid on Demand a Biomedical Application and a workflow manager are used Biomedical Application: WAVE Parallel (MonteCarlo simulation) applicationWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39
  39. 39. “Real use” and Performance WS-VLAM and WAVE “Real use” and Performance test To test the possibilities and performance of Grid on Demand a Biomedical Application and a workflow manager are used Biomedical Application: WAVE Parallel (MonteCarlo simulation) application Workflow Manager A (graphical) tool to assist complex e-Science application creation Creates a series of jobsubmissions WS-VLAM created by UvA SNE Group Connects to Globus Grid InterfaceWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39
  40. 40. “Real use” and Performance Connecting the Grid on Demand to a Workflow Manager Connecting the Grid on Demand to a Workflow Manager WAVE Biomedical Compute OS: Ubuntu 10.04 Application Grid: Globus 4.2.1 PBS: Torque 2.6.8 Compute Nodes Head NodeWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 34 / 39
  41. 41. “Real use” and Performance Preliminary performance results: total execution time Preliminary performance results: total execution time Grid on demand compared to a 32 node physical UvA cluster (DAS3) with Globus Grid interface Grid Jobs Total execution time DAS3 28 00:42:58 Grid on Demand 28 00:46:22 DAS3 98 02:46:36 Grid on Demand 98 00:52:26Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 35 / 39
  42. 42. “Real use” and Performance Preliminary performance results: 30 Jobs Preliminary performance results of job pending and running time: 30 jobs Performance on DAS3 - 30 Jobs Performance on Grid on Demand 30 Jobs 01:20:00 01:20:00 Running time Running time Pending time Pending time 01:10:00 01:10:00 01:00:00 01:00:00 00:50:00 00:50:00 Time (HH:MM:SS) Time (HH:MM:SS) 00:40:00 00:40:00 00:30:00 00:30:00 00:20:00 00:20:00 00:10:00 00:10:00 00:00:00 00:00:00 0 5 10 15 20 25 30 0 5 10 15 20 25 30 Job Number Job NumberWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 36 / 39
  43. 43. “Real use” and Performance Preliminary performance results: 100 Jobs Preliminary performance results of job pending and running time: 100 jobs Performance on DAS3 - 100 Jobs Performance on Grid on Demand - 100 Jobs Running time Running time 02:30:00 Pending time 02:30:00 Pending time 02:15:00 02:15:00 02:00:00 02:00:00 01:45:00 01:45:00 Time (HH:MM:SS) Time (HH:MM:SS) 01:30:00 01:30:00 01:15:00 01:15:00 01:00:00 01:00:00 00:45:00 00:45:00 00:30:00 00:30:00 00:15:00 00:15:00 00:00:00 00:00:00 0 10 20 30 40 50 60 70 80 90 100 0 20 40 60 80 100 Job Number Job NumberWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 37 / 39
  44. 44. Conclusion Conclusion Grid on Demand can offer Grid as a Cloud service Parallel applications run/scale very well on Grid on Demand Grid on Demand can be a solution for Urgent Computing demandsWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 38 / 39
  45. 45. The End QuestionsWillem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 39 / 39

×