Downloaded 11 times
![BonFIRE38 38
Experiment Descriptor
{
"name": "myExperiment", "description": “Controlled Network", "duration": 60,
"resources": [
{ "network": { "name": "myNetwork", "locations": ["be-ibbt"],
"address": "192.168.0.0", "size": "C",
"lossrate": 0, "bandwidth": 700, "latency": 0 }
},
{ "compute": { "name": "server", "locations": ["be-ibbt"],
"instanceType": "Large-EN", "min": 1,
"resources": [
{"storage": "@iperf-demo2"},
{"network": "@BonFIRE WAN"},
{"network": "myNetwork"}
],
"contexts": [] }
},
{ "compute": { "name": "client", "locations": ["be-ibbt"],
"instanceType": "Large-EN", "min": 1,
"resources": [
{"storage": "@iperf-demo2"},
{"network": "@BonFIRE WAN"},
{"network": "myNetwork"}
],
"contexts": [ {"IPERF_SERVER": ["server","myNetwork"]} ] }
}
]
}](https://image.slidesharecdn.com/bonfire-features-sites-tools-130529061555-phpapp02/85/BonFIRE-features-sites-and-tools-38-320.jpg)
![BonFIRE56 56
JSON experiment description snippet
{
"name": "An experiment",
"description": "An experiment description ",
"duration": 60,
"resources": [
{
"compute": {
"name": "VM",
"locations": [
"fr-inria"
],
"instanceType": "small",
"min": 1,
...
20,](https://image.slidesharecdn.com/bonfire-features-sites-tools-130529061555-phpapp02/85/BonFIRE-features-sites-and-tools-56-320.jpg)
![BonFIRE60 60
Ruby script example
session = Restfully::Session.new(
:configuration_file => "~/.restfully/api.bonfire-project.eu.yml",
:gateway => "ssh.bonfire.grid5000.fr",
:keys => ["~/.ssh/id_rsa"])
begin
experiment = session.root.experiments.find{|e|
e['name'] == "Demo SSH" && e['status'] == "running"
} || session.root.experiments.submit(
:name => "Demo SSH",
:description => "SSH demo using Restfully - #{Time.now.to_s}",
:walltime => 8*3600 # 8 hours
)](https://image.slidesharecdn.com/bonfire-features-sites-tools-130529061555-phpapp02/85/BonFIRE-features-sites-and-tools-60-320.jpg)
![BonFIRE63 63
Command line tools examples
Create experiment
• bfexperiment create <name> [-D <description>] [-W <walltime>]
[-G <group> ...]
• bfexperiment create my-experiment -D description -W 60 -G
bonfire
Create storage resource
• bfstorage create <name> <location> [<experiment>] [-D
<description>] [-S <size>] [-T {datablock|shared}] [-F <fstype>] [-
G <group> ...] [-P] [-R]
• bfstorage create my-storage fr-inria 42 -D storage-desc -S 1024
-T datablock -F ext3 -G bonfire](https://image.slidesharecdn.com/bonfire-features-sites-tools-130529061555-phpapp02/85/BonFIRE-features-sites-and-tools-63-320.jpg)
Uploaded byBonFIRE
BonFIRE: features, sites and tools
The document outlines the procedures and functionalities related to building service testbeds on the Bonfire platform, delivered in a webinar by Kostas Kavoussanakis. It includes information on the experiment lifecycle, prerequisites, resource management, and the use of client tools for creating and managing experiments. Additionally, it covers aspects such as monitoring, configuration, and the various testbed sites available within the Bonfire initiative.
More Related Content
Similar to BonFIRE: features, sites and tools
BonFIRE: features, sites and tools
- 1. Building service testbedson FIRE BonFIRE Webinar 29th May 2013 Delivered by Kostas Kavoussanakis, EPCC
- 2. BonFIRE2 2 Overview • Basicexperiment lifecycle • BonFIRE key functionalities • Testbed sites • Client tools • Experiment support mechanism
- 3. Building service testbedson FIRE BonFIRE Experiment Lifecycle
- 4. BonFIRE4 4 Prerequisites • Youmust have an account in BonFIRE. – If you don’t, please go to: • http://www.bonfire-project.eu/involved • https://portal.bonfire-project.eu/register/ • Your public key must be uploaded. – If not, you can find more information here: • http://doc.bonfire-project.eu/R3.1/getting-started/upload-ssh- key.html • You must have access to a SSH environment able to connect through one the BonFIRE SSH gateways – If not, you can find more information here: • http://doc.bonfire-project.eu/R3.1/getting-started/ssh-gateway- config.html
- 5. BonFIRE5 5 1. Createthe experiment You can choose between different tools Portal Command Line Experiment Manager Restfully CURL with OCCI • Don’t forget to set walltime properly. After it expires, the experiment (and its resources) will be SHUTDOWN and DELETED.
- 6. BonFIRE6 6 Experiments Resource Manager Experiment: 356 Walltime:3600 Experiment with Walltime 3600
- 7. BonFIRE7 7 2. Setup the experiment • Create networks, VMs and Storages • If you want monitoring, don’t forget to add an Aggregator. – http://doc.bonfire-project.eu/R3.1/monitoring/howto.html
- 8. BonFIRE8 8 Experiments Resource Manager Experiment 356 Experiment357 My Data Block Data Block SAVE
- 9. BonFIRE9 9 Managed Experimentsfor initial deployment Resource Manager Experiment 356 Experiment name: myExperiment compute: Name: compute1 Location: uk-eppc Disk: OS Network: BonFIRE WAN Compute: Name: compute2 Location: fr-inria Disk: OS Network: BonFIRE WAN Experiment Manager Managed Experiment 104
- 10. BonFIRE10 10 3. Configurethe computes • Log in to the computes and configure the, • Bear in mind that you can SAVE an image and it can be reused as many times as you want in the same location. • Images can’t be moved between locations, think about scripts for repeating the same commands.
- 11. BonFIRE11 11 Creating andconfiguring VMs OS OS MySQL OS OS >> apt_get mysql OS MySQL OS MySQL > OS > apt_get mysql OS MySQL OS MySQL SAVE SAVE
- 12. BonFIRE12 12 Contextualisation, elasticityand connectivity <context> <role>server</role> … </context> <context> <role>client</role> <serverIP> 192.168.2.2 </serverIP> … </context> <context> <role>client</role> <serverIP> 192.168.2.2 </serverIP> … </context> <context> <role>client</role> <serverIP> 192.168.2.2 </serverIP> … </context> BonFIRE WAN BonFIRE WAN
- 13.
- 14. BonFIRE14 14 SSH Gatewaysand VPN BonFIRE WAN JohnSmith/***** LDAP ServerLDAP Server VPN serverVPN server SSH GatewaySSH Gateway SSH GatewaySSH Gateway SSH GatewaySSH Gateway JohnSmith/***** JohnSmith/***** > ssh 192.168.4.2
- 15. BonFIRE15 15 4. Performyour experiment • Here you decide what to do! • If the experiment contains Virtual Wall (iMinds) resources, don’t forget to put in RUNNING state the experiment.
- 16. BonFIRE16 16 5. Monitoring •Access the Zabbix GUI, tunneled through the Portal. • Monitor Zabbix parameters, or configure your own. – BonFIRE offers application, VM and infrastructure monitoring from the same API • Use the monitoring API to make elasticity and application decisions on the fly
- 17. BonFIRE17 17 6. Wrapup your experiment • The experiment can finish in two different ways: – Expiry time (Walltime) ends. – You have finished before walltime arrives, and you decide to stop or delete the experiment. • Before this time arrives, save the images and datablocks that you want to use again. Then, shutdown the VM where the image/datablock is being used. – http://doc.bonfire-project.eu/R3.1/reference/experiment- lifecycle.html
- 18. BonFIRE18 18 7. Afterthe experiment • Access the monitoring data that you have saved – http://doc.bonfire-project.eu/R3.1/monitoring/getting-data/export- csv.html • Reuse the images that you have configured and saved
- 19. Building service testbedson FIRE BonFIRE Key Functionalities
- 20. BonFIRE20 20 Four pillarsof BonFIRE
- 21.
- 22. BonFIRE22 22 Three layersof monitoring Physical Machine Virtual Machine Virtual Machine Virtual Machine
- 23. BonFIRE23 23 Agents andAggregators VM Host 1 VM Host 2 VM Host 2 VM Host 1 VMVM VMVM Experiment Aggregator Experiment Aggregator Data VM data Application Data Intrastructure data VM Host 1 Agent Agent Site Aggregator Site Aggregator Site Aggregator Site AggregatorAgent Agent Agent Agent VM Host 2
- 24.
- 25.
- 26.
- 27. BonFIRE27 27 Emulated networksusing the Virtual Wall Client 1Client 1 Client 2Client 2 Client 3Client 3 Client 4Client 4 10Mbps10Mbps 1000Mbps1000MbpsServerServer1000Mbps1000Mbps 100Mbps 50ms latency 3% loss rate 100Mbps 50ms latency 3% loss rate 100Mbps100Mbps 100Mbps 50ms latency 5% loss rate 100Mbps 50ms latency 5% loss rate 100Mbps 50ms latency 10% loss rate 100Mbps 50ms latency 10% loss rate 100Mbps 50ms latency 15% loss rate 100Mbps 50ms latency 15% loss rate 100Mbps 50ms latency 20% loss rate 100Mbps 50ms latency 20% loss rate 100Mbps Traffic: TCP Packet size: 200 bytes Throughput: 20 packet/s 100Mbps Traffic: TCP Packet size: 200 bytes Throughput: 20 packet/s 100Mbps Traffic: TCP Packet size: 200 bytes Throughput: 30 packet/s 100Mbps Traffic: TCP Packet size: 200 bytes Throughput: 30 packet/s 100Mbps Traffic: TCP Packet size: 200 bytes Throughput: 40 packet/s 100Mbps Traffic: TCP Packet size: 200 bytes Throughput: 40 packet/s
- 28. BonFIRE28 28 epcc vmhost0 vmhost1 Resource Manager Controlled placement Location:epcc Instance type: lite Open Nebula Location: epcc Instance type: lite Host: vmhost1 Location: epcc Instance type: large Location: epcc Instance type: large Host: vmhost1
- 29. BonFIRE29 29 Exclusive physicalmachines INRIA node1 node2 node3 node7node6 node8 node9 Open Nebula Resource Manager Reservation System Give me 3 physical machines from 26/4/12 10:00 to 28/4/12 20:00 Cluster: 34563 P23 P40 p92 Location: inria Cluster: 34563 p23 p40 p92 Cluster: 34563 Location: inria Cluster: 34563 Host: p40 node5
- 30. BonFIRE30 30 Controlled Contention VMHost 1 VM Host 2 ExclusivePhysicalMachine VM under testVM under test VM under testVM under test VM under testVM under test CoCoMaCoCoMa Memory Use IO Use
- 31.
- 32.
- 33. BonFIRE33 33 epcc vmhost0 vmhost1 Custom instancetype Resource Manager Open NebulaLocation: epcc Instance type: custom CPU = 16 VPCU = 16 Memory = 16GB Host = vmhost1
- 34. BonFIRE34 34 Elasticity asa Service (EaaS) Elasticity Manager Elasticity Manager AggregatorAggregator Aggregator IP: 192.168.4.233 Upscale: {system.cpu.usage.last(0)}>70 Min: 1 Max: 5 Type: lite Disk: disk_name Location: uk-epcc Load Balancer: HAProxy, port 80, uk-epcc Load Balancer Load Balancer VMVM VMVM VMVM
- 35.
- 36. BonFIRE36 36 Ease ofUse • If it’s not easy, it won’t be used • But what is easy? • BonFIRE interpretations:
- 37.
- 38. BonFIRE38 38 Experiment Descriptor { "name":"myExperiment", "description": “Controlled Network", "duration": 60, "resources": [ { "network": { "name": "myNetwork", "locations": ["be-ibbt"], "address": "192.168.0.0", "size": "C", "lossrate": 0, "bandwidth": 700, "latency": 0 } }, { "compute": { "name": "server", "locations": ["be-ibbt"], "instanceType": "Large-EN", "min": 1, "resources": [ {"storage": "@iperf-demo2"}, {"network": "@BonFIRE WAN"}, {"network": "myNetwork"} ], "contexts": [] } }, { "compute": { "name": "client", "locations": ["be-ibbt"], "instanceType": "Large-EN", "min": 1, "resources": [ {"storage": "@iperf-demo2"}, {"network": "@BonFIRE WAN"}, {"network": "myNetwork"} ], "contexts": [ {"IPERF_SERVER": ["server","myNetwork"]} ] } } ] }
- 39. BonFIRE39 39 300 pagesof User Doc
- 40. Building service testbedson FIRE BonFIRE Testbed Sites
- 41. BonFIRE41 41 Infrastructure ingeneral • Open Access Sites – Virtual Wall: iMinds (IBBT) – OpenNebula: EPCC / INRIA / USTUTT – HP Cells: HPLabs • Side wide interconnects – Private: BonFIRE WAN (VPN network) – Public: GEANT • Special BonFIRE flavours – Heterogeneous resources – Permanent and on-request infrastructure – Virtual Wall: controlled networks (emulated) – HP Cells: securely isolated cells contain storages, computes, …
- 42. BonFIRE42 42 Site Capacities PermanentCores Memory Storage Nodes HPLabs 128 136 GB 5 TB 32 iMinds 400 400 GB 32 TB 100 HLRS 154 452 GB 12 TB 17 EPCC 96 256 GB 3.6 TB 2 INRIA 48 256 GB 2.4 TB 4 On-Request Cores Memory Storage Nodes HPLabs 384 144 GB 32 TB 96 iMinds 64 64 GB 32 TB 16 HLRS EPCC INRIA 1672 2.7 TB 42 TB 160
- 43.
- 44. BonFIRE44 44 • Siteshave different computing architectures • Different varieties of VMs are described through instance types • Sites support different instance types Name VCPU cores Memory Features Lite 0.5 256MB CPU may be < 1 Small 1 1GB Medium 2 2GB Large 2 4GB Large+ 2 4GB Higher CPU clock speed (over 3GHz) Large-EN 4 4GB Emulated network Xlarge 4 8GB XLarge+ 4 8GB Higher CPU clock speed (over 3GHz) Custom Free Free Integer values Which site to use? Site Lite Small Medium Large Large+ Large-EN XLarge Xlarge+ Custom HP + + + + + iMinds ++ HLRS + + + + ++ + ++ + EPCC + + + ++ ++ + INRIA + + + +
- 45. BonFIRE45 45 Which siteto use? • Depends on your requirements • Special characteristics – Controlled networks only available at iMinds – Public IPs are not available at iMinds and HLRS – Reservation of nodes only available at INRIA • Reservation at Inria Exclusive access to node • https://api.bonfire-project.eu/locations/fr-inria/reservations • Access methods – SSH & VPN – Each gateway serves for every BonFIRE WAN IP address
- 46. BonFIRE46 46 Information aboutsite status • BonFIRE health map indicates general availability – Nagios system tests – http://nebulosus.rus.uni-stuttgart.de/nagvis/frontend/nagvis- js/index.php?mod=Map&act=view&show=bonfire-full • Log files – OpenNebula status – OpenNebula Virtual machines – Experiment Manager log • Mailing list – bonfire-announcements@lists.atosresearch.eu
- 47. Building service testbedson FIRE BonFIRE Client Tools
- 48. BonFIRE48 48 Interacting withBonFIRE Resource Manager Site Z Site Y Site X OCCI EnactorcURL Experimenter Create compute XML All communication via an open RESTful interface Don’t need to write XML and use OCCI directly - we have several client tools available!
- 49. BonFIRE49 49 Client toolsoverview BonFIRE Portal • Web interface to the BonFIRE API • Very simple to use and get started in BonFIRE Experiment descriptors • Write the initial resource deployment in JSON or OVF • Create descriptor step-by-step on Portal and edit later Restfully • General-purpose RESTful client • Ruby scripting or Ruby shell interactions with the BonFIRE API Command line tools • Interactive, manual or scripted interaction with the BonFIRE API
- 50. BonFIRE50 50 BonFIRE Portal Step-by-step,wizard-like creation of resources
- 51.
- 52. BonFIRE52 52 BonFIRE Portal Monitoring
- 53. BonFIRE53 53 BonFIRE Portalpros and cons Pros • Very easy to use • Great starting point for understanding how things work in BonFIRE • Gives you the overview of your experiments and resources Cons • Slow to set up large-scale experiments • Manual creation of each resource • No scripted events possible for interacting with services
- 54. BonFIRE54 54 Experiment descriptors Experimentdescriptors can specify the initial deployment of resources. Additional features include: • VM contextualisation • IP dependency resolution • Configuration of monitoring metrics Two representation languages available • JSON • OVF
- 55.
- 56. BonFIRE56 56 JSON experimentdescription snippet { "name": "An experiment", "description": "An experiment description ", "duration": 60, "resources": [ { "compute": { "name": "VM", "locations": [ "fr-inria" ], "instanceType": "small", "min": 1, ... 20,
- 57. BonFIRE57 57 Experiment descriptorpros and cons Pros • Good for large-scale deployment of resources • Easy to create via the Portal and edit offline • Intuitive and purposefully created for BonFIRE • Submit via Portal or command line Cons • Cannot script other elements of experiment set-ups like interacting with experiment services • Cannot get connection to resources when created to perform actions
- 58. BonFIRE58 58 Restfully • Ageneral-purpose client tool for RESTful APIs • Abstracts the details of exchanging HTTP requests • Discovers resources at run-time • Can be used in a Ruby shell to interactively query the BonFIRE API • Can be used in Ruby scripts to automate deployment of resources and interactions with the resources Written in Ruby
- 59. BonFIRE59 59 Ruby shellexample ruby-1.8.7-p249 > pp root.locations #<Collection:0x813dd534 uri=https://api.bonfire-project.eu/locations RELATIONSHIPS parent, self ITEMS (0..5)/5 #<Resource:0x813c6b04 uri=https://api.bonfire-project.eu/locations/be- ibbt> #<Resource:0x813b5868 uri=https://api.bonfire-project.eu/locations/de- hlrs> #<Resource:0x813a45cc uri=https://api.bonfire-project.eu/locations/fr- inria> #<Resource:0x81393330 uri=https://api.bonfire-project.eu/locations/uk- epcc> #<Resource:0x81382094 uri=https://api.bonfire-project.eu/locations/uk- hplabs>> => nil
- 60. BonFIRE60 60 Ruby scriptexample session = Restfully::Session.new( :configuration_file => "~/.restfully/api.bonfire-project.eu.yml", :gateway => "ssh.bonfire.grid5000.fr", :keys => ["~/.ssh/id_rsa"]) begin experiment = session.root.experiments.find{|e| e['name'] == "Demo SSH" && e['status'] == "running" } || session.root.experiments.submit( :name => "Demo SSH", :description => "SSH demo using Restfully - #{Time.now.to_s}", :walltime => 8*3600 # 8 hours )
- 61. BonFIRE61 61 Restfully prosand cons Pros • Good for large-scale experiments • One script for deployment of resources and gives you connection to resources to interact with, e.g., to install software, start services, etc. • Interactive scripting possible (user input on command line) Cons • You have to create the scripts from scratch • Learning curve for Ruby can be steep, but we provide examples online • Manually need to resolve deployment constraints
- 62. BonFIRE62 62 Command linetools • Purposefully built for BonFIRE • Written in Python – Bindings are easy to get by reading our online documentation • Support for BonFIRE experiment lifecycle: – bfexperiment – bfcompute – bfstorage – … • Abbreviations possible, instead of long URIs – /locations/uk-epcc/networks/42 42
- 63. BonFIRE63 63 Command linetools examples Create experiment • bfexperiment create <name> [-D <description>] [-W <walltime>] [-G <group> ...] • bfexperiment create my-experiment -D description -W 60 -G bonfire Create storage resource • bfstorage create <name> <location> [<experiment>] [-D <description>] [-S <size>] [-T {datablock|shared}] [-F <fstype>] [- G <group> ...] [-P] [-R] • bfstorage create my-storage fr-inria 42 -D storage-desc -S 1024 -T datablock -F ext3 -G bonfire
- 64. BonFIRE64 64 Command linetools pros and cons Pros • Intuitive to use • Scripting possible, e.g., Bash on Linux or Batch (MSDOS) on Windows Cons • Scripting is more limited than Restfully – Connecting to interact with resources possible but not as easy as with Ruby • Less declarative than experiment descriptor – Manually need to resolve deployment constraints
- 65. BonFIRE65 65 Resources • Clienttools: http://doc.bonfire-project.eu/integration/client- tools/overview.html
- 66. Building service testbedson FIRE Experiment Support
- 67. BonFIRE67 67 Support Mechanism Documentation • General Information: http://bonfire-project.eu/ • User Documentation: http://www.bonfire-project.eu/docs User Forum https://forum.bonfire-project.eu/ Support Ticketing System support@bonfire-project.eu
- 68. Building service testbedson FIRE Thank you for your attention
- 69. BonFIRE69 69 Licence andAcknowledgements Copyright © 2013, EPCC, The University of Edinburgh, on behalf of the BonFIRE Consortium. Licensed under Creative Commons “Attribution-NoDerivs”. BonFIRE is funded by the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement numbers 257386 and 287938. www.bonfire-project.eu
Editor's Notes
- #64 The following positional argument must be specified when creating experiments:<name> - The name for this experiment.The following options are applicable when creating experiments:-D <description>, --description <description> - description of the experiment. Defaults to “<no description>”.-W <walltime>, --walltime <walltime> - lifetime of the experiment in seconds. Defaults to one day.-G <group>, --group <group> - A user group this experiment will be accessible by. This option can be specified multiple times.The following positional arguments must be specified when creating storages (in that order):<name> - The name for this resource.<location> - The BonFIRE site where this resource will be created.Optionally, an experiment this resource will be part of can be specified as a third positional argument. The following options are applicable when creating storages:-D <description>, --description - The description of this storage resource. Defaults to “<no description>”.-S <size>, --size - Size of this storage resource in MiB. This is only applicable to datablock resources. Defaults to 1024 MiB.-T {datablock|shared} - The type of this storage resource. Note that “shared” storages are only available on be-ibbt. Defaults to “datablock”.-F <filesystem>, --fstype <filesystem> - The filesystem the storage resource will be formatted with. This is only applicable to datablock resources. Defaults to “ext3”.-G <group>, --group <group> - A user group this resource will be accessible by. This option can be specified multiple times.-P, --public - Indicates that this storage resource should be publicly available. Defaults to <false>.-R, --persistent - Indicates that a persistent storage resource should be created. Defaults to <false>.