Building service testbeds on FIREExperimenting Virtual Clusters on distributed     Cloud environments using BonFIREA. Góme...
Overview       Objective of experiment       Experiment Set-up       Virtual Cluster Architecture       Experiments ab...
Objective of Experiment    Research the feasibility of using multiple Cloud environments      for the provision of Service...
Objective of Experiment (II)     Experiments about time-to-deploy the virtual      infrastructure.     Experiments about...
EXPERIMENT SET-UP5      VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012   BonFIRE 5
BonFIRE InfrastructureINRIA:   Vendor   Freq.                  (GHz)                          Cores          RAM          ...
VCOC Experiment Agent                                             Manages multiple                                       ...
Experiment Agent Workflow                                   Cluster ready8   VCOC, FIRE Engineering Workshop, Ghent, Nov. ...
Elasticity Engine                                   Takes information about                                    applicatio...
VCOC Interfaces                                                           Monitoring                                      ...
VIRTUAL CLUSTER     ARCHITECTURE11      VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012   BonFIRE 11
VC Configuration on BonFIRE12   VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012   BonFIRE 12
Simple VC Configuration13   VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012   BonFIRE 13
Distributed VC14   VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012   BonFIRE 14
Fault-tolerance VC configuration15   VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012   BonFIRE 15
Name of the Managed                                              Experiment                                        JSON VC...
EXPERIMENTS ABOUT     TIME-TO-DEPLOY17      VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012   BonFIRE 17
Experiment data  5 repetitions                                                        Maximum  Representation: Box plot ...
Deployment time vs VC sizeBonFIRE site: INRIA                    Experiment Manager          Start VMsReserved: 2 physical...
Deployment time vs storage size                                                       Without cacheBonFIRE site: INRIARese...
Two clusters in one site                        Second cluster delayed because the deployment of first VC. BonFIRE site: ...
Application execution. One vs Two                                                        sites VC Conf.: Distributed VC (...
Adding a new VM Client VC Conf.: Simple VC BonFIRE sites:    INRIA: Master + CEs Using BonFIRE Resource  Manager Very...
ELASTICITY EXPERIMENT24      VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012   BonFIRE 24
Horizontal elasticity Monitoring application  performance works. We have modified software  to produce information more ...
FAULT TOLERANCE     EXPERIMENT WITH ELASTICITY26      VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012   BonFIR...
Virtual Cluster                               SYNC27   VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012   BonFI...
Fault tolerance  BonFIRE sites:      HLRS (Master + 4 CEs)      INRIA (Shadow + 4 CEs)  Demanded performance   (500H/s...
CONCLUSIONS29      VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012   BonFIRE 29
Conclusions      BonFIRE provides infrastructure for experiments about       new concepts and services on Cloud.      Di...
Acknowledgment     To all BonFIRE team, specially to:        Maxence Dunnewind (INRIA)        Michael Gienger (HLRS)    ...
Building service testbeds on FIREThank you for your attention Project contact: bonfire@bonfire-project.eu   Experiment Con...
Upcoming SlideShare
Loading in …5
×

VCOC BonFIRE presentation at FIRE Engineering Workshop 2012

511 views

Published on

VCOC experiment in BonFIRE European Project (http://www.bonfire-project.eu) results. It shows a general architecture with fault-tolerance to use in fistributed Cloud environments and the usage of application performance indicators to trigger cluster elasticity. More information at www.cesga.es.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
511
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

VCOC BonFIRE presentation at FIRE Engineering Workshop 2012

  1. 1. Building service testbeds on FIREExperimenting Virtual Clusters on distributed Cloud environments using BonFIREA. Gómez, L.M. Carril, R. Valin, J.C. Mouriño, C. Cotelo FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012
  2. 2. Overview  Objective of experiment  Experiment Set-up  Virtual Cluster Architecture  Experiments about time-to-deploy  Elasticity experiment  Fault-tolerance experiment  Conclusions2 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 2
  3. 3. Objective of Experiment Research the feasibility of using multiple Cloud environments for the provision of Services which need a large pool of CPUs (as High Throughput Computing). Why3 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 3
  4. 4. Objective of Experiment (II)  Experiments about time-to-deploy the virtual infrastructure.  Experiments about elasticity triggered by application performance.  Experiments about fault-tolerant virtual infrastructure.4 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 4
  5. 5. EXPERIMENT SET-UP5 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 5
  6. 6. BonFIRE InfrastructureINRIA: Vendor Freq. (GHz) Cores RAM (GB) HLRS: Vendor Freq. (GHz) Cores RAM (GB) Intel 2.33 2*2 4 Intel 3.2 2*2 2 AMD 1,7 2*12 48 Intel 2.66 2*2 8 Intel 2,5 2*4 32 AMD 2.6 4*12 196 Intel 2.93 2*4 24 AMD 2 2 4 Intel I7 2.53 2 4 Intel I7 2.1 4 8 Intel Atom 1 2 AMD 1.65 2 2 T56N Cloud Manager: OpenNebula 3.0 Source: http://www.bonfire-project.eu/infrastructure (Nov. 2012) 6 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 6
  7. 7. VCOC Experiment Agent  Manages multiple experiment executions.  Works with the Experiment Manager.  Controls deployed resources.  Stores Experiment data.7 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 7
  8. 8. Experiment Agent Workflow Cluster ready8 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 8
  9. 9. Elasticity Engine  Takes information about application performance  Get data about queues  Calculates Nr. VMs to fulfill demanded performance  Adds new VMs to Experiment9 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 9
  10. 10. VCOC Interfaces Monitoring GUI dashboard Identity LDAP Portal Server (Used by Portal,VCOC Experiment Agent Experiment Manager, API Resource Manageruses Experiment Manager and Testbeds) Experiment Manager Portal UsageElasticity Engine uses OCCI Monitoring Reservation Accounting API AuthZResource Manager (Used by RM to authorize resource Resource Manager creation) OCCI Reservation Message Read/ Write Queue Enactor OCCI Reservation SSH Testbed SSH Monitoring Monitoring SSH Gateway GUI API VM Gateway (Monitoring VM VM Aggregator) 10 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 10
  11. 11. VIRTUAL CLUSTER ARCHITECTURE11 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 11
  12. 12. VC Configuration on BonFIRE12 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 12
  13. 13. Simple VC Configuration13 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 13
  14. 14. Distributed VC14 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 14
  15. 15. Fault-tolerance VC configuration15 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 15
  16. 16. Name of the Managed Experiment JSON VC Example { "name": "Virtual Cluster Experiment", Maximum elapsed time for the Experiment in minutes "description": "Virtual Cluster", "duration": 180, Description of the experiment "resources": [ { "compute": { "name": "master", Location for deployment "description": "Virtual Cluster Master", "instanceType": "small", "locations": ["fr-inria"], Main Storage Resource "resources": [ { "storage": "@VirtualClusterMaster-vcocdemo-v0.1"}, { "storage": "@vcocdemo_volume-1GB-empty"}, Master Virtual Machine Configuration { "network": "@BonFIRE WAN"} ], "contexts": [ Additional Storage Resource { } ] Network } }, { "compute": { "name": "client-1", "description": "Virtual Cluster Client", "instanceType": "small", "locations": ["fr-inria"], "resources": [ { "storage": "@VirtualClusterClient-vcocdemo-v0.1"}, { "network": "@BonFIRE WAN"} Client Virtual Machine Description ], "contexts": [ { } ] } }16 ] } VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 16
  17. 17. EXPERIMENTS ABOUT TIME-TO-DEPLOY17 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 17
  18. 18. Experiment data  5 repetitions Maximum  Representation: Box plot Average Median Minimum18 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 18
  19. 19. Deployment time vs VC sizeBonFIRE site: INRIA Experiment Manager Start VMsReserved: 2 physical nodes with2x12 cores AMD Opteron 1.7 GHzand 48 GB RAMVC Configuration:• Simple VC• One master (small + 4GB storage)• n-1 CEs (small) Conf. OGS Undeployment19 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 19
  20. 20. Deployment time vs storage size Without cacheBonFIRE site: INRIAReserved: 2 physical nodes with2x12 cores AMD Opteron 1.7 GHzand 48 GB RAMVC Configuration: Simple VC 16VM One master (small + x GB storage) 15 CEs (small) With Cache 20 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 20
  21. 21. Two clusters in one site Second cluster delayed because the deployment of first VC. BonFIRE site: INRIA Reserved: 2 physical nodes with 2x12 cores AMD Opteron First Virtual Cluster 1.7 GHz and 48 GB RAM VC Configuration: 4 VM 8 VM  Simple VC  One master (small + 4GB storage)  n-1 CEs (small) Second Virtual Cluster21 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 21
  22. 22. Application execution. One vs Two sites VC Conf.: Distributed VC (_dist) BonFIRE sites:  INRIA: Master + CEs  HLRS: CEs Deployment time decreases. Two sites faster than one site. But because second site has better CPUs. Impact of deployment ~ 10% total time. 22 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 22
  23. 23. Adding a new VM Client VC Conf.: Simple VC BonFIRE sites:  INRIA: Master + CEs Using BonFIRE Resource Manager Very fast due to cache 23 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 23
  24. 24. ELASTICITY EXPERIMENT24 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 24
  25. 25. Horizontal elasticity Monitoring application performance works. We have modified software to produce information more frequently. Execution with deadline. Elasticity works.25 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 25
  26. 26. FAULT TOLERANCE EXPERIMENT WITH ELASTICITY26 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 26
  27. 27. Virtual Cluster SYNC27 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 27
  28. 28. Fault tolerance  BonFIRE sites:  HLRS (Master + 4 CEs)  INRIA (Shadow + 4 CEs)  Demanded performance (500H/s)  Fault simulated putting HLRS VMs in CANCEL.  INRIA Shadow took control of cluster.  Elasticity worked, demanding more CEs to INRIA.28 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 28
  29. 29. CONCLUSIONS29 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 29
  30. 30. Conclusions  BonFIRE provides infrastructure for experiments about new concepts and services on Cloud.  Distributed VC can be used to speed up HTC applications.  Elasticity engine based on application performance for HTC works.  High QoS can be provided in VC using distributed VC + elasticity.30 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 30
  31. 31. Acknowledgment To all BonFIRE team, specially to:  Maxence Dunnewind (INRIA)  Michael Gienger (HLRS)  Ally Hume (EPCC)  Kostas Kavoussanakis (EPCC)  David García Pérez (Atos Research)31 VCOC, FIRE Engineering Workshop, Ghent, Nov. 6th – 7th 2012 BonFIRE 31
  32. 32. Building service testbeds on FIREThank you for your attention Project contact: bonfire@bonfire-project.eu Experiment Contact: agomez@cesga.es

×