Nubilum: Resource Management System for Distributed Clouds
Upcoming SlideShare
Loading in...5
×
 

Nubilum: Resource Management System for Distributed Clouds

on

  • 705 views

My PhD thesis about resource management on Distributed Cloud environments.

My PhD thesis about resource management on Distributed Cloud environments.

Statistics

Views

Total Views
705
Views on SlideShare
697
Embed Views
8

Actions

Likes
0
Downloads
11
Comments
0

2 Embeds 8

https://mj89sp3sau2k7lj1eg3k40hkeppguj6j-a-sites-opensocial.googleusercontent.com 5
http://mj89sp3sau2k7lj1eg3k40hkeppguj6j-a-sites-opensocial.googleusercontent.com 3

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Nubilum: Resource Management System for Distributed Clouds Document Transcript

  • 1. Pós-Graduação em Ciência da Computação“Nubilum: Resource Management System forDistributed Clouds”PorGlauco Estácio GonçalvesTese de DoutoradoUniversidade Federal de Pernambucoposgraduacao@cin.ufpe.brwww.cin.ufpe.br/~posgraduacaoRECIFE, 03/2012
  • 2. UNIVERSIDADE FEDERAL DE PERNAMBUCOCENTRO DE INFORMÁTICAPÓS-GRADUAÇÃO EM CIÊNCIA DA COMPUTAÇÃOGLAUCO ESTÁCIO GONÇALVES“Nubilum: Resource Management System for Distributed Clouds"ESTE TRABALHO FOI APRESENTADO À PÓS-GRADUAÇÃO EMCIÊNCIA DA COMPUTAÇÃO DO CENTRO DE INFORMÁTICA DAUNIVERSIDADE FEDERAL DE PERNAMBUCO COMO REQUISITOPARCIAL PARA OBTENÇÃO DO GRAU DE DOUTOR EM CIÊNCIADA COMPUTAÇÃO.ORIENTADORA: Dra. JUDITH KELNERCO-ORIENTADOR: Dr. DJAMEL SADOKRECIFE, MARÇO/2012
  • 3. Tese de Doutorado apresentada por Glauco Estácio Gonçalves à Pós- Graduação emCiência da Computação do Centro de Informática da Universidade Federal dePernambuco, sob o título “Nubilum: Resource Management System for DistributedClouds” orientada pela Profa. Judith Kelner e aprovada pela Banca Examinadoraformada pelos professores:___________________________________________________________Prof. Paulo Romero Martins MacielCentro de Informática / UFPE__________________________________________________________Prof. Stênio Flávio de Lacerda FernandesCentro de Informática / UFPE____________________________________________________________Prof. Kelvin Lopes DiasCentro de Informática / UFPE_________________________________________________________Prof. José Neuman de SouzaDepartamento de Computação / UFC___________________________________________________________Profa. Rossana Maria de Castro AndradeDepartamento de Computação / UFCVisto e permitida a impressão.Recife, 12 de março de 2012.___________________________________________________Prof. Nelson Souto RosaCoordenador da Pós-Graduação em Ciência da Computação doCentro de Informática da Universidade Federal de Pernambuco.
  • 4. To my family Danielle, JoãoLucas, and Catarina.
  • 5. ivAcknowledgmentsI would like to express my gratitude to God, cause of all the things and also myexistence; and to the Blessed Virgin Mary to whom I appealed many times in prayer, beingattended always.I would like to thank my advisor Dr. Judith Kelner and my co-advisor Dr. DjamelSadok, whose expertise and patience added considerably to my doctoral experience. Thanksfor the trust in my capacity to conduct my doctorate at GPRT (Networks andTelecommunications Research Group).I am indebted to all the people from GPRT for their invaluable help for this work. Avery special thanks goes out to Patrícia, Marcelo, and André Vítor, which have givenvaluable comments over the course of my PhD.I must also acknowledge my committee members, Dr. Jose Neuman, Dr. Otto Duarte,Dr. Rossana Andrade, Dr. Stênio Fernandes, Dr. Kelvin Lopes, and Dr. Paulo Maciel forreviewing my proposal and dissertation, offering helpful comments to improve my work.I would like to thank my wife Danielle for her prayer, patience, and love which gaveme the necessary strength to finish this work. A special thanks to my children, João Lucasand Catarina. They are gifts of God that make life delightful.Finally, I would like to thank my parents, João and Fátima, and my sisters, Cynara andKarine, for their love. Their blessings have always been with me as I followed in my doctoralresearch.
  • 6. vAbstractThe current infrastructure of Cloud Computing providers is composed of networking andcomputational resources that are located in large datacenters supporting as many ashundreds of thousands of diverse IT equipment. In such scenario, there are severalmanagement challenges related to the energy, failure and operational management andtemperature control. Moreover, the geographical distance between resources and final usersis a source of delay when accessing the services. An alternative to such challenges is thecreation of Distributed Clouds (D-Clouds) with geographically distributed resources along toa network infrastructure with broad coverage.Providing resources in such a distributed scenario is not a trivial task, since, beyond theprocessing and storage resources, network resources must be taken in consideration offeringusers a connectivity service for data transportation (also called Network as a Service – NaaS).Thereby, the allocation of resources must consider the virtualization of servers and thenetwork devices. Furthermore, the resource management must consider all steps from theinitial discovery of the adequate resource for attending developers’ demand to its finaldelivery to the users.Considering those challenges in resource management in D-Clouds, this Thesisproposes then Nubilum, a system for resource management on D-Clouds considering geo-locality of resources and NaaS aspects. Through its processes and algorithms, Nubilumoffers solutions for discovery, monitoring, control, and allocation of resources in D-Cloudsin order to ensure the adequate functioning of the D-Cloud while meeting developers’requirements. Nubilum and its underlying technologies and building blocks are describedand their allocation algorithms are also evaluated to verify their efficacy and efficiency.Keywords: cloud computing, resource management mechamisms, network virtualization.
  • 7. viResumoAtualmente, a infraestrutura dos provedores de computação em Nuvem é composta porrecursos de rede e de computação, que são armazenados em datacenters de centenas demilhares de equipamentos. Neste cenário, encontram-se diversos desafios quanto à gerênciade energia e controle de temperatura, além de, devido à distância geográfica entre os recursose os usuários, ser fonte de atraso no acesso aos serviços. Uma alternativa a tais desafios é ouso de Nuvens Distribuídas (Distributed Clouds – D-Clouds) com recursos distribuídosgeograficamente ao longo de uma infraestrutura de rede com cobertura abrangente.Prover recursos em tal cenário distribuído não é uma tarefa trivial, pois, além derecursos computacionais e de armazenamento, deve-se considerar recursos de rede os quaissão oferecidos aos usuários da nuvem como um serviço de conectividade para transporte dedados (também chamado Network as a Service – NaaS). Desse modo, o processo de alocaçãodeve considerar a virtualização de ambos, servidores e elementos de rede. Além disso, agerência de recursos deve considerar desde a descoberta dos recursos adequados paraatender as demandas dos usuários até a manutenção da qualidade de serviço na sua entregafinal.Considerando estes desafios em gerência de recursos em D-Clouds, este trabalhopropõe Nubilum: um sistema para gerência de recursos em D-Cloud que considera aspectosde geo-localidade e NaaS. Por meio de seus processos e algoritmos, Nubilum oferecesoluções para descoberta, monitoramento, controle e alocação de recursos em D-Clouds deforma a garantir o bom funcionamento da D-Cloud, além de atender os requisitos dosdesenvolvedores. As diversas partes e tecnologias de Nubilum são descritos em detalhes esuas funções delineadas. Ao final, os algoritmos de alocação do sistema são tambémavaliadas de modo a verificar sua eficácia e eficiência.Palavras-chave: computação em nuvem, mecanismos de alocação de recursos, virtualizaçãode redes.
  • 8. viiContentsAbstract vResumo viAbbreviations and Acronyms xii1 Introduction 11.1 Motivation............................................................................................................................................. 21.2 Objectives ............................................................................................................................................. 41.3 Organization of the Thesis................................................................................................................. 42 Cloud Computing 62.1 What is Cloud Computing?................................................................................................................ 62.2 Agents involved in Cloud Computing.............................................................................................. 72.3 Classification of Cloud Providers...................................................................................................... 82.3.1 Classification according to the intended audience..................................................................................82.3.2 Classification according to the service type.............................................................................................82.3.3 Classification according to programmability.........................................................................................102.4 Mediation System............................................................................................................................... 112.5 Groundwork Technologies.............................................................................................................. 122.5.1 Service-Oriented Computing...................................................................................................................122.5.2 Server Virtualization..................................................................................................................................122.5.3 MapReduce Framework............................................................................................................................132.5.4 Datacenters.................................................................................................................................................143 Distributed Cloud Computing 153.1 Definitions.......................................................................................................................................... 153.2 Research Challenges inherent to Resource Management ............................................................ 183.2.1 Resource Modeling....................................................................................................................................183.2.2 Resource Offering and Treatment..........................................................................................................203.2.3 Resource Discovery and Monitoring......................................................................................................223.2.4 Resource Selection and Optimization....................................................................................................233.2.5 Summary......................................................................................................................................................274 The Nubilum System 284.1 Design Rationale................................................................................................................................ 284.1.1 Programmability.........................................................................................................................................284.1.2 Self-optimization........................................................................................................................................294.1.3 Existing standards adoption.....................................................................................................................294.2 Nubilum’s conceptual view.............................................................................................................. 294.2.1 Decision plane............................................................................................................................................304.2.2 Management plane.....................................................................................................................................314.2.3 Infrastructure plane...................................................................................................................................324.3 Nubilum’s functional components.................................................................................................. 324.3.1 Allocator......................................................................................................................................................334.3.2 Manager.......................................................................................................................................................34
  • 9. viii4.3.3 Worker.........................................................................................................................................................354.3.4 Network Devices.......................................................................................................................................364.3.5 Storage System ...........................................................................................................................................374.4 Processes............................................................................................................................................. 374.4.1 Initialization processes..............................................................................................................................374.4.2 Discovery and monitoring processes......................................................................................................384.4.3 Resource allocation processes..................................................................................................................394.5 Related projects.................................................................................................................................. 405 Control Plane 435.1 The Cloud Modeling Language ....................................................................................................... 435.1.1 CloudML Schemas.....................................................................................................................................455.1.2 A CloudML usage example......................................................................................................................525.1.3 Comparison and discussion .....................................................................................................................565.2 Communication interfaces and protocols...................................................................................... 575.2.1 REST Interfaces.........................................................................................................................................575.2.2 Network Virtualization with Openflow.................................................................................................635.3 Control Plane Evaluation ................................................................................................................. 656 Resource Allocation Strategies 686.1 Manager Positioning Problem ......................................................................................................... 686.2 Virtual Network Allocation.............................................................................................................. 706.2.1 Problem definition and modeling ...........................................................................................................726.2.2 Allocating virtual nodes............................................................................................................................746.2.3 Allocating virtual links...............................................................................................................................756.2.4 Evaluation...................................................................................................................................................766.3 Virtual Network Creation................................................................................................................. 816.3.1 Minimum length Steiner tree algorithms ...............................................................................................826.3.2 Evaluation...................................................................................................................................................866.4 Discussion........................................................................................................................................... 897 Conclusion 917.1 Contributions ..................................................................................................................................... 927.2 Publications ........................................................................................................................................ 937.3 Future Work ....................................................................................................................................... 94References 96
  • 10. ixList of FiguresFigure 1 Agents in a typical Cloud Computing scenario (from [24]) ..................................................7Figure 2 Classification of Cloud types (from [71]).................................................................................9Figure 3 Components of an Archetypal Cloud Mediation System (adapted from [24]) ................11Figure 4 Comparison between (a) a current Cloud and (b) a D-Cloud............................................16Figure 5 ISP-based D-Cloud example ...................................................................................................17Figure 6 Nubilum’s planes and modules...............................................................................................30Figure 7 Functional components of Nubilum......................................................................................33Figure 8 Schematic diagram of Allocator’s modules and relationships with other components..33Figure 9 Schematic diagram of Manager’s modules and relationships with other components...34Figure 10 Schematic diagram of Worker modules and relationships with the server system........35Figure 11 Link discovery process using LLDP and Openflow ..........................................................38Figure 12 Sequence diagram of the Resource Request process for a developer..............................39Figure 13 Integration of different descriptions using CloudML........................................................44Figure 14 Basic status type used in the composition of other types..................................................45Figure 15 Type for reporting status of the virtual nodes ....................................................................46Figure 16 XML Schema used to report the status of the physical node...........................................46Figure 17 Type for reporting complete description of the physical nodes.......................................46Figure 18 Type for reporting the specific parameters of any node ...................................................47Figure 19 Type for reporting information about the physical interface ...........................................48Figure 20 Type for reporting information about a virtual machine..................................................48Figure 21 Type for reporting information about the whole infrastructure ......................................49Figure 22 Type for reporting information about the physical infrastructure...................................49Figure 23 Type for reporting information about a physical link .......................................................50Figure 24 Type for reporting information about the virtual infrastructure .....................................50Figure 25 Type describing the service offered by the provider .........................................................51Figure 26 Type describing the requirements that can be requested by a developer .......................52Figure 27 Example of a typical Service description XML ..................................................................53Figure 28 Example of a Request XML..................................................................................................53Figure 29 Physical infrastructure description........................................................................................54Figure 30 Virtual infrastructure description..........................................................................................55Figure 31 Communication protocols employed in Nubilum..............................................................57Figure 32 REST operation for the retrieval of service information..................................................59Figure 33 REST operation for updating information of a service ....................................................59Figure 34 REST operation for requesting resources for a new application.....................................59Figure 35 REST operation for changing resources of a previous request .......................................60Figure 36 REST operation for releasing resources of an application ...............................................60Figure 37 REST operation for registering a new Worker...................................................................60Figure 38 REST operation to unregister a Worker..............................................................................61Figure 39 REST operation for update information of a Worker ......................................................61Figure 40 REST operation for retrieving a description of the D-Cloud infrastructure .................61Figure 41 REST operation for updating the description of a D-Cloud infrastructure...................61Figure 42 REST operation for the creation of a virtual node............................................................62Figure 43 REST operation for updating a virtual node ......................................................................62Figure 44 REST operation for removal of a virtual node...................................................................62Figure 45 REST operation for requesting the discovered physical topology ..................................63Figure 46 REST operation for the creation of a virtual link ..............................................................63Figure 47 REST operation for updating a virtual link.........................................................................64Figure 48 REST operation for removal of a virtual link.....................................................................64
  • 11. xFigure 49 Example of a typical rule for ARP forwarding...................................................................65Figure 50 Example of the typical rules created for virtual links: (a) direct, (b) reverse..................65Figure 51 Example of a D-Cloud with ten workers and one Manager.............................................69Figure 52 Algorithm for allocation of virtual nodes............................................................................74Figure 53 Example illustrating the minimax path................................................................................75Figure 54 Algorithm for allocation of virtual links..............................................................................76Figure 55 The (a) old and (b) current network topologies of RNP used in simulations................77Figure 56 Results for the maximum node stress in the (a) old and (b) current RNP topology....78Figure 57 Results for the maximum link stress in the (a) old and (b) current RNP topology ......79Figure 58 Results for the mean link stress in the (a) old and (b) current RNP topology...............80Figure 59 Mean path length (a) old and (b) current RNP topology..................................................80Figure 60 Example creating a virtual network: (a) before the creation; (b) after the creation ......81Figure 61 Search procedure used by the GHS algorithm....................................................................83Figure 62 Placement procedure used by the GHS algorithm.............................................................84Figure 63 Example of the placement procedure: (a) before and (b) after placement.....................85Figure 64 Percentage of optimal samples for GHS and STA in the old RNP topology................87Figure 65 Percentage of samples reaching relative error ≤ 5% in the old RNP topology.............88Figure 66 Percentage of optimal samples for GHS and STA in the current RNP topology ........88Figure 67 Percentage of samples reaching relative error ≤ 5% in the current RNP topology......89
  • 12. xiList of TablesTable I Summary of the main aspects discussed..................................................................................27Table II MIMEtypes used in the overall communications.................................................................58Table III Models for the length of messages exchanged in the system in bytes.............................67Table IV Characteristics present in Nubilum’s resource model ........................................................71Table V Reduced set of characteristics considered by the proposed allocation algorithms ..........72Table VI Factors and levels used in the MPA’s evaluation ................................................................78Table VII Factors and levels used in the GHS’s evaluation...............................................................86Table VIII Scientific papers produced ..................................................................................................94
  • 13. xiiAbbreviations and AcronymsCDN Content Delivery NetworkCloudML Cloud Modeling LanguageD-Cloud Distribute CloudDHCP Dynamic Host Configuration ProtocolGHS Greedy Hub SelectionHTTP Hypertext Transfer ProtocolIaaS Infrastructure as a ServiceISP Internet Service ProviderLLDP Link Layer Discovery ProtocolMPA Minimax Path AlgorithmMPP Manager Positioning ProblemNaaS Network as a ServiceNV Network VirtualizationOA Optimal AlgorithmOCCI Open Cloud Computing InterfacePoP Point of PresenceREST Representational state transferRP Replica PlacementRPA Replica Placement AlgorithmSTA Steiner Tree ApproximationVM Virtual MachineVN Virtual NetworkXML Extensible Markup LanguageZAA Zhu and Ammar Algorithm
  • 14. 11 Introduction“A inea incipere.”ErasmusNowadays, it is common to access content across the Internet with little reference to the underlyingdatacenter hosting infrastructure maintained by content providers. The entire technology used toprovide such level of locality transparency offers also a new model for the provisioning ofcomputing services, known as Cloud Computing. This model is attractive as it allows resources to beprovisioned according to users’ requirements leading to overall cost reduction. Cloud users can rentresources as they become necessary, in a much more scalable and elastic way. Moreover, such userscan transfer operational risks to cloud providers. In the viewpoint of those providers, the modeloffers a way for a better utilization of their own infrastructure. Ambrust et al. [1] point out that thismodel benefits from a form of statistical multiplexing, since it allocates resources for several usersconcurrently on a demand basis. This statistical multiplexing of datacenters is subsequent to severaldecades of research in many areas such as distributed computing, Grid computing, webtechnologies, service computing, and virtualization.Current Cloud Computing providers mainly use large and consolidated datacenters in order tooffer their services. However, the ever increasing need for over-provisioning to attend peakdemands and providing redundancy against failures allied to expensive cooling needs are importantfactors increasing the energetic costs of centralized datacenters [62]. In current datacenters, thecooling technologies used for heat dissipation control accounts for as much as 50% of the totalpower consumption [38]. In addition to these aspects, it must be observed that the network betweenusers and the Cloud is often an unreliable best-effort IP service, which can harm delay-constrainedservices and interactive applications.To deal with these problems, there have been some indicatives whereby small cooperativedatacenters can be more attractive since they offer cheaper and low-power consumption alternativereducing the infrastructure costs of centralized Clouds [12]. These small datacenters can be built atdifferent geographical regions and connected by dedicated or public (provided by Internet ServiceProviders) networks, configuring a new type of Cloud, referred to as a Distributed Cloud. Such
  • 15. 2Distributed Clouds [20], or just D-Clouds, can exploit the possibility of (virtual) links creation andthe potential of sharing resources across geographic boundaries to provide latency-based allocationof resources to fully utilize this emerging distributed computing power. D-Clouds can reducecommunication costs by simply provisioning storage, servers, and network resources close to end-users.The D-Clouds can be considered as an additional step in the ongoing deployments of CloudComputing: one that supports different requirements and leverages new opportunities for serviceproviders. Users in a Distributed Cloud will be free to choose where to allocate their resources inorder to attend specific market niches, constraints on jurisdiction of software and data, or quality ofservice aspects of their clients.1.1 MotivationSimilarly to Cloud Computing, one of the most important design aspects of D-Clouds is theavailability of “infinite” computing resources which may be used on demand. Cloud users see this“infinite” resource pool because the Cloud offers the continuous monitoring and management of itsresources and the allocation of resources in an elastic way. Nevertheless, providing on-demandcomputing instances and network resources in a distributed scenario is not a trivial task. Dynamicallocation of resources and their possible reallocation are essential characteristics for accommodatingunpredictable demands and, ultimately, contributing to investment return.In the context of Clouds, the essential feature of any resource management system is toguarantee that both user and provider requirements are met satisfactorily. Particularly in D-Clouds,users may have network requirements, such as bandwidth and delay constraints, in addition to thecommon computational requirements, such as CPU, memory, and storage. Furthermore, other userrequirements are relevant including node locality, topology of nodes, jurisdiction, and applicationinteraction.The development of solutions to cope with resource management problems remains a veryimportant topic in the field of Cloud Computing. With regard to this technology, there are solutionsfocused on grid computing ([49], [70]) and on datacenters in current Cloud Computing scenarios([4]). However, such strategies do not fit well the D-Clouds as they are heavily based on assumptionsthat do not hold in Distributed Cloud scenarios. For example, such solutions are designed for over-provisioned networks and commonly do not take into consideration the cost of resources’communication, which is an important aspect for D-Clouds that must be cautiously monitoredand/or reserved in order to meet users’ requirements.
  • 16. 3The design of a resource management system involves challenges other than the specificdesign of optimization algorithms for resource management. Since D-Clouds are composed ofcomputational and network devices with different architectures, software, and hardware capabilities,the first challenge is the development of a suitable resource model covering all this heterogeneity[20],. In addition, the next challenge is to describe how resources are offered, which is importantsince the requirements supported by the D-Cloud provider are defined in this step. The otherchallenges are related with the overall operation of the resource management system. When requestsarrive, the system should be aware of the current status of resources, in order to determine if thereare sufficient available resources in the D-Cloud that could satisfy the present request. In this way,the right mechanisms for resource discovery and monitoring should also be designed, allowing thesystem to be aware of the updated status of all its resources. Then, based on the current status andthe requirements of the request, the system may select and allocate resources to serve the presentrequest.Please note that the solution to those challenges involves the fine-grained coordination ofseveral distributed components and the orchestrated execution of the several subsystems composingthe resource management system. At a first glance, these subsystems can be organized into threeparts: one responsible for the direct negotiation of requirements with users; another responsible fordeciding what resources to allocate for given applications; and one last part responsible for theeffective enforcement of these decisions on the resources.Designing such system is a very interesting and challenging task, and it raises the followingresearch questions that will be investigated in this thesis:1. How Cloud users describe their requirements? In order to enable the automaticnegotiation between users and the D-Cloud, the Cloud must recognize a language orformalism for requirements description. Thus, the investigation of this topic must determinethe proper characteristics of such a language. In addition, it must verify the existentapproaches around this topic in the many relative computing areas.2. How to represent the resources available in the Cloud? Correlated to the first question,the resource management system must also maintain an information model to represent allthe resources in the Cloud, including their relationships (topology) and their current status.3. How the users’ applications are mapped onto Cloud resources? This question is aboutthe very aspect of resource allocation, i.e., the algorithms, heuristics, and strategies that areused to decide the set of resources meeting the applications’ requirements and optimizing autility function.
  • 17. 44. How to enforce the decisions made? The effective enforcement of the decisions involvesthe extension of communication protocols or the development of new ones in order tosetup the state of the overall resources in the D-Cloud.1.2 ObjectivesThe main objective of this Thesis is to propose an integrated solution to problems related to themanagement of resources in D-Clouds. Such solution is presented as Nubilum, a resourcemanagement system that offers a self-managed system for challenges on discovery, control,monitoring, and allocation of resources in D-Clouds. Nimbulus provides fine-grained orchestrationof their components in order to allocate applications on a D-Cloud.The specific goals of this Thesis are strictly related to the research questions presented inSection 1.1, they are:• Elaborate an information model to describe D-Cloud resources and applicationrequirements as computational restrictions, topology, geographic location and othercorrelated aspects that can be employed to request resources directly to the D-Cloud;• Explore and extend communication protocols for the provisioning and allocation ofcomputational and communication resources;• Develop algorithms, heuristics, and strategies to find suitable D-Cloud resources based onseveral different application requirements;• Integrate the information model, the algorithms, and the communication protocols, into asingle solution.1.3 Organization of the ThesisThis Thesis identifies the challenges involved in the resource management on Distributed CloudComputing and presents solutions for some of these challenges. The remainder of this document isorganized as follows.The general concepts that make up the basis for all the other chapters are introduced in thesecond chapter. Its main objective is to discuss Cloud Computing while trying to explore suchdefinition and to classify the main approaches in this area.The Distributed Cloud Computing concept and several important aspects of resourcemanagement on those scenarios are introduced in the third chapter. Moreover, this chapter willmake a comparative analysis of related research areas and problems.
  • 18. 5The fourth chapter introduces the first contribution of this Thesis: the Nubilum resourcemanagement system, which aggregates the several solutions proposed on this Thesis. Moreover, thechapter highlights the rationale behind Nubilum as well as their main modules and components.The fifth chapter examines and evaluates the control plane of Nubilum. It describes theproposed Cloud Modeling Language and details the communication interfaces and protocols usedfor communicating between Nubilum components.The sixth chapter gives an overview of the resource allocation problems in DistributedClouds, and makes a thorough examination of the specific problems related to Nubilum. Someparticular problems are analyzed and a set of algorithms is presented and evaluated.The seventh chapter of this Thesis reviews the obtained evaluation results, summarizes thecontributions and sets the path to future works and open issues on D-Cloud.
  • 19. 62 Cloud Computing“Definitio est declaratio essentiae rei.”Legal ProverbIn this chapter the main concepts of Cloud Computing will be presented. It begins with a discussionon the definition of Cloud Computing (Section 2.1) and the main agents involved in CloudComputing (Section 2.2). Next, classifications of Cloud initiatives are offered in Section 2.3. Anexemplary and simple architecture of a Cloud Mediation System is presented in Section 2.4 followedby a presentation in Section 2.5 of the main technologies acting behind the scenes of CloudComputing initiatives.2.1 What is Cloud Computing?A definition of Cloud Computing is given by the National Institute of Standards and Technology(NIST) of the United States: “Cloud computing is a model for enabling convenient, on-demandnetwork access to a shared pool of configurable computing resources (e.g., networks, servers,storage, applications, and services) that can be rapidly provisioned and released with minimalmanagement effort or service provider interaction” [45]. The definition says that on-demanddynamic reconfiguration (elasticity) is a key characteristic. Additionally, the definition highlightsanother Cloud Computing characteristic: it assumes that minimal management efforts are requiredto reconfigure resources. In other words, the Cloud must offer self-service solutions that mustattend to requests on-demand, excluding from the scope of Cloud Computing those initiatives thatoperate through the rental of computing resources in a weekly or monthly basis. Hence, it restrictsCloud Computing to systems that provide automatic mechanisms for resource rental in real-timewith minimal human intervention.The NIST definition gives a satisfactory concept of Cloud Computing as a computing model.But, NIST does not cover the main object of Cloud Computing: the Cloud. Thus, in this Thesis,Cloud Computing is defined as the computing model that operates based on Clouds. In turn, theCloud is defined as a conceptual layer that operates above an infrastructure to provide elasticservices in a timely manner.
  • 20. 7This definition encompasses three main characteristics of Clouds. Firstly, it notes that a Cloudis primarily a concept, i.e., a Cloud is an abstraction over an infrastructure. Thus, it is independent ofthe employed technologies and therefore one can accept different setups, like Amazon EC2 orGoogle App Engine, to be named Clouds. Moreover, the infrastructure is defined in a broad senseonce it can be composed by software, physical devices, and/or other Clouds. Secondly, all Cloudshave the same purpose: to provide services. This means that a Cloud hides the complexity of theunderlying infrastructure while exploring the potential of overlying services and acting as amiddleware. In addition, providing a service involves, implicitly, the use of some type of agreementthat should be guaranteed by the Cloud. Such agreements can vary from pre-defined contracts tomalleable agreements defining functional and non-functional requirements. Note that these servicesare qualified as elastic ones, which has the same meaning of dynamic reconfiguration that appearedin the NIST definition. Last but not least, the Cloud must provide services as quickly as possiblesuch that the infrastructure resources are allocated and reallocated to attend the users’ needs.2.2 Agents involved in Cloud ComputingDespite previous approaches ([64], [8], [72], and [68]), this Thesis focuses only on three distinctagents in Cloud Computing as shown in Figure 1: clients, developers, and the provider. The firstnotable point is that the provider deals with two types of users that are called developers and clients.Thus, clients are the customers of a service produced by a developer. Clients use services fromdevelopers, but such use generates demand to the provider that actually hosts the service, andtherefore the client can also be considered a user of the Cloud. It is important to highlight that insome scenarios (like scientific computing or batch processing) a developer may behave as a client tothe Cloud because it is the end-user of the applications. The text will use “users” when referring toboth classes without distinctions.Figure 1 Agents in a typical Cloud Computing scenario (from [24])Developers can be service providers, independent programmers, scientific institutions, and soon, i.e., all who build applications into the Cloud. They create and run their applications whileDeveloperDeveloperClient Client Client Client
  • 21. 8keeping decisions related to maintenance and management of the infrastructure to the provider.Please note that, a priori, developers do not need to know about the technologies that makeup theCloud infrastructure, neither about the specific location of each item in the infrastructure.Lastly, the term application is used to mean all types of services that can be developed on theCloud. In addition, it is important to note that the type of applications supported by a Clouddepends exclusively on the goals of the Cloud as determined by the provider. Such a wide range ofpossible targets generates many different types of Cloud Providers that are discussed in the nextsection.2.3 Classification of Cloud ProvidersCurrently, there are several operational initiatives of Cloud Computing; however despite all beingcalled Clouds, they provide different types of services. For that reason, the academic community([64], [8], [45], [72], and [71]) classified these solutions accurately in order to understand theirrelationships. The three complementary proposals for classification are as follows.2.3.1 Classification according to the intended audienceThis first simple taxonomy is suggested by NIST [45] that organizes providers according to theaudience to which the Cloud is aimed. There are four classes in this classification: Private Clouds,Community Clouds, Public Clouds, and Hybrid Clouds.The first three classes accommodate providers in a gradual opening of the intended audiencecoverage. The Private Cloud class encompasses such types of Clouds destined to be used solely byan organization operating over their own datacenter or one leased from a third party for exclusiveuse. When the Cloud infrastructure is shared by a group of organizations with similar interests it isclassified as a Community Cloud. Furthermore, the Public Cloud class encompasses all initiativesintended to be used by the general public. Finally, Hybrid Clouds are simply the composition of twoor more Clouds pertaining to different classes (Private, Community, or Public).2.3.2 Classification according to the service typeIn [71], authors offer a classification as represented in Figure 2. Such taxonomy divides Clouds infive categories: Cloud Application, Cloud Software Environment, Cloud Software Infrastructure,Software Kernel, and Firmware/Hardware. The authors arranged the different types of Clouds in astack, showing that Clouds from higher levels are created using services in the lower levels. This ideapertains to the definitions of Cloud Computing discussed previously in Sections 2.1 and 2.2.Essentially, the Cloud provider does not need to be the owner of the infrastructure.
  • 22. 9Figure 2 Classification of Cloud types (from [71])The class in the top of the stack, also called Software-as-a-Service (SaaS), involves applicationsaccessed through the Internet, including social networks, Webmail, and Office tools. Such servicesprovide software to be used by the general public, whose main interest is to avoid tasks related tosoftware management like installation and updating. From the point of view of the Cloud provider,SaaS can decrease costs with software implementation when compared with traditional processes.Similarly, the Cloud Software Environment, also called Platform-as-a-Service (PaaS), enclosesClouds that offer programming environments for developers. Through well-defined APIs,developers can use software modules for access control, authentication, distributed processing, andso on, in order to produce their own applications in the Cloud. Moreover, developers can contractservices for automatic scalability of their software, databases, and storage services.In the middle of the stack there is the Cloud Software Infrastructure class of initiatives. Thisclass encompasses solutions that provide virtual versions of infrastructure devices found indatacenters like servers, databases, and links. Clouds in this class can be divided into three subclassesaccording to the type of resource that is offered by them. Computational resources are grouped inthe Infrastructure-as-a-service (IaaS) subclass that provides generic virtual machines that can be usedin many different ways by the contracting developer. Services for massive data storage are groupedin the Data-as-a-Service (DaaS) class, whose main mission is to store remotely users’ data on remote,which allows those users to access their data from anywhere and at anytime. Finally, the thirdsubclass, called Communications-as-a-Service (CaaS), is composed of solutions that offer virtualprivate links and routers through telecommunication infrastructures.The last two classes do not offer Cloud services specifically, but they are included in theclassification to show that providers offering Clouds in higher layers can have their own softwareand hardware infrastructure. The Software Kernel class includes all of the software necessary toprovide services to the other categories like operating systems, hypervisors, cloud management
  • 23. 10middleware, programming APIs, and libraries. Finally, the class of Firmware/Hardware covers allsale and rental services of physical servers and communication hardware.2.3.3 Classification according to programmabilityThe five-class scheme presented above can classify and organize the current spectrum of CloudComputing solutions, but such a model is limited because the number of classes and theirrelationships will need to be rearranged as new Cloud services emerge. Therefore, in this Thesis, adifferent classification model will be used based on the programmability concept, which waspreviously introduced by Endo et al. [19].Borrowed from the realm of network virtualization [11], programmability is a concept relatedto the programming features a network element offers to developers, measuring how much freedomthe developer has to manipulate resources and/or devices. This concept can be easily applied to thecomparison of Cloud Computing solutions. More programmable Clouds offer environments wheredevelopers are free to choose programming paradigms, languages, and platforms. Lessprogrammable Clouds restrict developers in some way: perhaps by forcing a set of programminglanguages or by providing support for only one application paradigm. On the other hand,programmability directly affects the way developers manage their leased resources. From this point-of-view, providers of less programmable Clouds are responsible to manage their infrastructure whilebeing transparent to developers. In turn, a more programmable Cloud leaves more of these tasks todevelopers, thus introducing management difficulties due to the more heterogeneous programmingenvironment.Thus, Cloud Programmability can be defined as the level of sovereignty under whichdevelopers have to manipulate services leased from a provider. Programmability is a relativeconcept, i.e., it was adopted to compare one Cloud with others. Also, programmability is directlyproportional to heterogeneity in the infrastructure of the provider and inversely proportional to theamount of effort that developers must spend to manage leased resources.To illustrate how this concept can be used, one can classify two current Clouds: Amazon EC2and Google App Engine. Clearly the Amazon EC2 is more programmable, since in this Clouddevelopers can choose between different virtual machine classes, operating systems, and so on. Afterthey lease one of these virtual machines, developers can configure it to work as they see fit: as a webserver, as a content server, as a unit for batch processing, and so on. On the other hand, GoogleApp Engine can be classified as a less programmable solution, because it allows developers to createWeb applications that will be hosted by Google. This restricts developers to the Web paradigm andto some programming languages.
  • 24. 112.4 Mediation SystemFigure 3 introduces an Archetypal Cloud Mediation System. This is a conceptual model that will beused as a reference to the discussion on Resource Management in this Thesis. The Archetypal CloudMediation System focuses on one principle: resource management as the main service of any CloudComputing provider. Thus, other important services like authentication, accounting, and security areout of the scope of this conceptual system and, therefore these services are separated from theMediation System in this archetypal Cloud mediation system. Clients also do not factor into thisview of the system, since resource management is mainly related to the allocation of developers’applications and meeting their requirements.Figure 3 Components of an Archetypal Cloud Mediation System (adapted from [24])The mediation system is responsible for the entire process of resource management in theCloud. Such a process covers tasks that range from the automatic negotiation of developersrequirements to the execution of their applications. It has three main layers: negotiation, resourcemanagement, and resource control.The negotiation layer deals with the interface between the Cloud and developers. In the caseof Clouds selling infrastructure services, the interface can be a set of operations based on WebServices for control of the leased virtual machines. Alternately, in the case of PaaS services, thisinterface can be an API for software development in the Cloud. Moreover, the negotiation layerhandles the process of contract establishment between the enterprises and the Cloud. Currently, thisprocess is simple and the contracts tend to be restrictive. One can expect that in the future, Cloudswill offer more sophisticated avenues for user interaction through high level abstractions and servicelevel policies.MediationSystemResourcesResource ManagementNegotiationResource ControlDevelopersAuxiliaryServicesAccountAuthenticationSecurity
  • 25. 12The resource management layer is responsible for the optimal allocation of applications forobtaining the maximum usage of resources. This function requires advanced strategies and heuristicsto allocate resources that meet the contractual requirements as established with the applicationdeveloper. These may include service quality restrictions, jurisdiction restrictions, elastic adaptation,among others.Metaphorically, one can say that while the resource management layer acts as the “brain” ofthe Cloud, the resource control layer plays the role of its “limbs”. The resource control encompassesall functions needed to enforce decisions generated by the upper layer. Beyond the tools used toconfigure the Cloud resources effectively, all communication protocols used by the Cloud areincluded in this layer.2.5 Groundwork TechnologiesSome of the main technologies that used by the current Cloud mediation systems (namely Service-oriented Computing, Virtualization, MapReduce, and Datacenters) will be discussed.2.5.1 Service-Oriented ComputingService-Oriented Computing defines a set of principles, architectural models, and technologies forthe design and development of distributed applications. The recent development of software whilefocusing on services gave rise to SOA (Service-Oriented Architecture), which can be defined as anarchitectural model “that supports the discovery, message exchange, and integration between looselycoupled services using industry standards” [37]. The common technology for the implementation ofSOA principles is the Web Service that defines a set of standards to implement services over theWorld Wide Web.In Cloud Computing, SOA is the main paradigm for the development of functions on theseveral layers of the Cloud. Cloud providers publish APIs for their services on the web, allowingdevelopers to use the Cloud and to automate several tasks related to the management of theirapplications. Such APIs can assume the form of WSDL documents or REST-based interfaces.Furthermore, providers can make available Software Development Kits (SDKs) and other toolkitsfor the manipulation of applications running on the Cloud.2.5.2 Server VirtualizationServer virtualization is a technique that allows a computer system to be partitioned onto multipleisolated execution environments offering a similar service as a single physical computer, which arecalled Virtual Machines (VM). Each VM can be configured in an independent way while having itsown operating system, applications, and network parameters. Commonly, such VMs are hosted on a
  • 26. 13physical server running a hypervisor, the software that effectively virtualizes the server and managesthe VMs [54].There are several hypervisor options that can be used for server virtualization. From the open-source community, one can cite Citrix’s Xen1and the Kernel-based Virtual Machine (KVM)2. Fromthe realm of proprietary solutions, some examples are VMWare ESX3and Microsoft’s HyperV4.The main factor that boosted up the adoption of server virtualization within CloudComputing is that such technology offers good flexibility regarding the dynamic reallocation ofworkloads across servers. Such flexibility allows, for example, providers to execute maintenance onservers without stopping developers’ applications (that are running on VMs) or to implementstrategies for better resource usage through the migration of VMs. Furthermore, server virtualizationis adapted for the fast provisioning of new VMs through the use of templates, which enablesproviders to offer elastic services for applications developers [43].2.5.3 MapReduce FrameworkMapReduce [15] is a programming framework developed by Google for distributed processing oflarge data sets across computing infrastructures. Inspired on the map and reduce primitives presentin functional languages, its authors developed an entire framework for the automatic distribution ofcomputations. In this framework, developers are responsible for writing map and reduce operationsand for using them according to their needs, which is similar to the functional paradigm. These mapand reduce operations will be executed by the MapReduce system that transparently distributescomputations across the computing infrastructure and treats all issues related to nodecommunication, load balancing, and fault tolerance. For the distribution and synchronization of thedata required by the application, the MapReduce system also requires the use of a specially tailoreddistributed file system called Google File System (GFS) [23].Despite being introduced by Google, there are some open source implementations of theMapReduce system, like Hadoop [6] and TPlatform [55]. The former is a popular open-sourcesoftware used for running applications on large clusters built of commodity hardware. This softwareis used by large companies like Amazon, AOL, and IBM, as well as in different Web applicationssuch as Facebook, Twitter, Last.fm, among others. Basically, Hadoop is composed of two modules:a MapReduce environment for distributed computing, and a distributed file system called theHadoop Distributed File System (HDFS). The latter is an academic initiative that provides a1 http://www.xen.org/products/cloudxen.html2 http://www.linux-kvm.org/page/Main_Page3 http://www.vmware.com/4 http://www.microsoft.com/hyper-v-server/en/us/default.aspx
  • 27. 14development platform for Web mining applications. Similarly to Hadoop and Google’s MapReduce,the TPlatform has a MapReduce module and a distributed file system known as the Tianwang FileSystem (TFS) [55].The use of MapReduce solutions is common groundwork technology in PaaS Clouds becauseit offers a versatile sandbox for developers. Differently from IaaS Clouds, PaaS developers using ageneral-purpose language with MapReduce support do not need to be concerned with softwareconfiguration, software updating and, network configurations. All these tasks are the responsibilityof the Cloud provider, which, in turn, benefits from the fact that such configurations will bestandardized across the overall infrastructure.2.5.4 DatacentersDevelopers who are hosting their applications on a Cloud wish to scale their leased resources,effectively increasing and decreasing their virtual infrastructure according to the demand of theirclients. This is also the case for developers making use of their own private Clouds. Thus,independently of the class of Cloud under consideration, a robust and safe infrastructure is needed.Whereas virtualization and MapReduce respond for the software solution required to attendthis demand, the physical infrastructure of Clouds is based on datacenters, which are infrastructurescomposed of TI components providing processing capacity, storage, and network services for oneor more organizations [66]. Currently, the size of a datacenter (in number of components) can varyfrom tens of components to tens of thousands of components depending on the datacenter’smission. In addition, there are several different TI components for datacenters including switchesand routers, load balancers, storage devices, dedicated storage networks, and, the main componentof any datacenter, in other words, servers [27].Cloud Computing datacenters provide the required power to attend developers’ demands interms of processing, storage, and networking capacities. A large datacenter, running a virtualizationsolution, allows for better granularity division of the hardware’s power through the statisticalmultiplexing of developers’ applications.
  • 28. 153 Distributed Cloud Computing“Quae non prosunt singula, multa iuvant.”OvidThis chapter discusses the main concepts of Distributed Cloud (D-Cloud) Computing. It beginswith a discussion of their definition (Section 3.1) in an attempt to distinguish the D-Cloud from thecurrent Clouds and highlight their main characteristics. Next, the main research challenges regardingresource management on D-Clouds will be described in Section 3.2.3.1 DefinitionsCurrent Cloud Computing setups involve a huge amount of investments as part of the datacenter,which is the common underlying infrastructure of Clouds as previously detailed in Section 2.5.4.This centralized infrastructure brings many well-known challenges such as the need for resourceover-provisioning and the high cost for heat dissipation and temperature control. In addition toconcerns with infrastructure costs, one must observe that those datacenters are not necessarily closeto their clients, i.e., the network between end-users and the Cloud is often a long best-effort IPconnection, which means longer round-trip delays.Considering such limitations, industry and academy researchers have presented indicatives thatsmall datacenters can be sometimes more attractive since they offer a cheaper and low-powerconsumption alternative while also reducing the infrastructure costs of centralized Clouds [12].Moreover, Distributed Clouds, or just D-Clouds, as pointed out by Endo et al. in [20], can exploitthe possibility of links creation and the potential of sharing resources across geographic boundariesto provide latency-based allocation of resources to ultimately fully utilize this distributed computingpower. Thus, D-Clouds can reduce communication costs by simply provisioning data, servers, andlinks close to end-users.Figure 4 illustrates how D-Clouds can reduce the cost of communication through the spreadof computational power and the usage of a latency-based allocation of applications. In Figure 4(a)the client uses an application (App) running on the Cloud through the Internet, which is subject tothe latency imposed by the best-effort network. In Figure 4(b), the client is accessing the same App,
  • 29. 16but in this case, the latency imposed by the network will be reduced due to the allocation of the Appin a server that is in a small datacenter closest to the client than the previous scenario.(a) (b)Figure 4 Comparison between (a) a current Cloud and (b) a D-CloudPlease note that the Figure 4(b) intentionally does not specify the network connecting theinfrastructure of the D-Cloud Provider. This network can be rented from different local ISPs (usingthe Internet for interconnection) or from an ISP with wide area coverage. In addition, such ISPcould be the own D-Cloud Provider itself. This may be the case as the D-Cloud paradigmintroduces an organic change in the current Internet where ISPs can start to play as D-Cloudproviders. Thus, ISPs could offer their communication and computational resources for developersinterested in deploying their applications at the specific markets covered by those ISPs.This idea is illustrated by Figure 5 that shows a D-Cloud offered by a hypothetical BrazilianISP. In this example, a developer deployed its application (App) on two servers in order to attendrequests from northern and southern clients. If the number of northeastern clients increases, thedeveloper can deploy its App (represented by the dotted box) on one server close to the northeastregion in order to improve its service quality. It is important to pay attention to the fact that thecontribution of this Thesis falls in this last scenario, i.e., a scenario where the network andcomputational resources are all controlled by the same provider.CloudProviderClientInternetAppClientAppDistributedCloudProvider
  • 30. 17Figure 5 ISP-based D-Cloud exampleD-Clouds share similar characteristics with current Cloud Computing, including essentialofferings such as scalability, on demand usage, and pay-as-you-go business plans. Furthermore, theagents already stated for current Clouds (please see Figure 1) are exactly the same in the context ofD-Clouds. Finally, the many different classifications discussed in Section 2.3 can be applied also.Despite the similarity, one may highlight two peculiarities of D-Clouds: support to geo-locality andNetwork as a Service (NaaS) provisioning ([2], [63], [17]).The geographical diversity of resources potentially improves cost and performance and givesan advantage to several different applications, particularly, those that do not require massive internalcommunication among large server pools. In this category, as pointed out by [12], one canemphasize, firstly, applications being currently deployed in a distributed manner, like VOIP (Voiceover IP) and online games; secondly, one can indicate the applications that are good candidates fordistributed implementation, like traffic filtering and e-mail distribution. In addition, there are otherdifferent types of applications that use software or data with specific legal restrictions onjurisdiction, and specific applications whose public is restricted to one or more geographical areas,like the tracking of buses or subway routes, information about entertainment events, local news, etc.Support for geo-locality can be considered to be a step further in the deployment of CloudComputing that leverages new opportunities for service providers. Thus, they will be free to choosewhere to allocate their resources in order to attend to specific niches, constraints on jurisdiction ofsoftware and data, or quality of service aspects of end-users.The NaaS (or Communication as a Service – CaaS as cited in section 2.3.2) allows serviceproviders to manage network resources, instead of just computational ones. Authors in [2] call NaaSas a service offering transport network connectivity with a level of virtualization suitable to beAppAppApp
  • 31. 18invoked by service providers. In this way, D-Clouds are able to manage their network resourcesaccording to their convenience, offering better response time for hosted applications. The NaaS isclose to the Network Virtualization (NV) research area [31], where the main problem consists inchoosing how to allocate a virtual network over a physical one, meeting requirements andminimizing usage of the physical resources. Although NV and D-Clouds are subject to similarproblems and scenarios, there is an essential difference between these two. While NV commonlymodels its resources at the infrastructure level (requests are always virtual networks mapped ongraphs), a D-Cloud can be engineered to work with applications in a different abstraction level,exactly as it occurs with actual Cloud service types like the ones described at Section 2.3.2. This way,one may see Network Virtualization simply as a particular instance of the D-Cloud. Other insightsabout NV are given in Section 3.3.2.Finally, it must be highlighted that the D-Cloud does not compete with the current CloudComputing paradigm, since the D-Cloud merely fits a certain type of applications that have hardrestrictions on geographical location, while the existent Clouds continue to be attracting forapplications demanding massive computational resources or simple applications with minor or norestrictions on geographical location. Thus, the current Cloud Computing providers are the firstpotential candidates to take advantage of the D-Cloud paradigm, since the current Clouds could hireD-Cloud resources on-demand and move the applications to certain geographical locations in orderto meet specific developers’ requirements. In addition to the current Clouds, the D-Clouds can alsoserve the developers directly.3.2 Research Challenges inherent to Resource ManagementD-Clouds face challenges similar to the ones presented in the context of current Cloud Computing.However, as stated in Chapter 1, the object of the present study is the resource management in D-Clouds. Thus, this Section gives special emphasis to the challenges for resource management in D-Clouds, while focusing on four categories as presented in [20]: a) resource modeling; b) resourceoffering and treatment; c) resource discovery and monitoring; and d) resource selection.3.2.1 Resource ModelingThe first challenge is the development of a suitable resource model that is essential to all operationsin the D-Cloud, including management and control. Optimization algorithms are also stronglydependent of the resource modeling scheme used.In a D-Cloud environment, it is very important that resource modeling takes into accountphysical resources as well as virtual ones. On one hand, the amount of details in each resourceshould be treated carefully, since if resources are described with great details, there is a risk that the
  • 32. 19resource optimization becomes hard and complex, since the computational optimization problemconsidering the several modeled aspects can create NP-hard problems. On the other hand, moredetails give more flexibility and leverage the usage of resources.There are some alternatives for resource modeling in Clouds that could be applied to D-Clouds. One can cite, for example, the OpenStack software project [53], which is focused onproducing an open standard Cloud operating system. It defines a Restful HTTP service thatsupports JSON and XML data formats and it is used to request or to exchange information aboutCloud resources and action commands. OpenStack also offers ways to describe how to scale serverdown or up (using pre-configured thresholds); it is extensible, allowing the seamless addition of newfeatures; and it returns additional error messages in faults case.Other resource modeling alternative is the Virtual Resources and Interconnection NetworksDescription Language (VXDL) [39], whose main goal is to describe resources that compose a virtualinfrastructure while focusing on virtual grid applications. The VXDL is able to describe thecomponents of an infrastructure, their topology, and an execution chronogram. These three aspectscompose the main parts of a VXDL document. The computational resource specification partdescribes resource parameters. Furthermore, some peculiarities of virtual Grids are also present,such as the allocation of virtual machines in the same hardware and location dependence. Thespecification of the virtual infrastructure can consider specific developers’ requirements such asnetwork topology and delay, bandwidth, and the direction of links. The execution chronogramspecifies the period of resource utilization, allowing efficient scheduling, which is a clear concern forGrids rather than Cloud computing. Another interesting point of VXDL is the possibility ofdescribing resources individually or in groups, according to application needs. VXDL lacks supportfor distinct services descriptions, since it is focused on grid applications only.The proposal presented in [32], called VRD hereafter, describes resources in a networkvirtualization scenario where infrastructure providers describe their virtual resources and servicesprior to offering them. It takes into consideration the integration between the properties of virtualresources and their relationships. An interesting point in the proposal is its use of functional andnon-functional attributes. Functional attributes are related to characteristics, properties, andfunctions of components. Non-functional attributes specify criteria and constraints, such asperformance, capacity, and QoS. Among the functional properties that must be highlighted is the setof component types: PhysicalNode, VirtualNode, Link, and Interface. Such properties suggest aflexibility that can be used to represent routers or servers, in the case of nodes, and wired or wirelesslinks, in the case of communication links and interfaces.
  • 33. 20Another proposal known as the Manifest language was developed by Chapman et al. [9]. Theyproposed new meta-models to represent service requirements, constraints, and elasticity rules forsoftware deployment in a Cloud. The building block of such framework is the OVF (OpenVirtualization Format) standard, which was extended by Chapman et al. to perform the vision of D-Clouds considering locality constraints. These two points are very interesting to our scenario. Withregard to elasticity, it assumes a rule-based specification formed by three fields: a monitoredcondition related to the state of the service (such as workload), an operator (relational and logicalones are accepted), and an associated action to follow when the condition is met. The locationconstraints identify sites that should be favored or avoided when selecting a location for a service.Nevertheless, the Manifest language is focused on the software architecture. Hence, the language isnot concerned with other aspects such as resources’ status or network resources.Cloud# is a language for modeling Clouds proposed by [16] to be used as a basis for Cloudproviders and clients to establish trust. The model is used by developers to understand the behaviorof Cloud services. The main goal of Cloud# is to describe how services are delivered, while takinginto consideration the interaction among physical and virtual resources. The main syntacticconstruct within Cloud# is the computation unit CUnit, which can model Cloud systems, virtualmachines, or operating systems. A CUnit is represented as a tuple of six components modelingcharacteristics and behaviors. This language gives developers a better understanding of the Cloudorganization and how their applications are dealt with.3.2.2 Resource Offering and TreatmentOnce the D-Cloud resources are modeled, the next challenge is to describe how resources areoffered to developers, which is important since the requirements supported by the provider aredefined in this step. Such challenge will also define the interfaces of the D-Cloud. This challengediffers from resource modeling since the modeling is independent of the way that resources areoffered to developers. For example, the provider could model each resource individually, likeindependent items in a fine-grained scale such as GHz of CPU or GB of memory, but could offerthem like a coupled collection of those items or a bundle, such as VM templates as cited at Section2.5.2.Recall that, in addition to computational requirements (CPU and memory) and traditionalnetwork requirements, such as bandwidth and delay, new requirements are present under D-Cloudscenarios. The topology of the nodes is a first interesting requirement to be described. Developersshould be able to set inter-nodes relationships and communication restrictions (e.g., downlink anduplink rates). This is illustrated in the scenario where servers – configured and managed by
  • 34. 21developers – are distributed at different geographical localities while it is necessary for them tocommunicate with each other in a specific way.Jurisdiction is related to where (geographically) applications and their data must be stored andhandled. Due to restrictions such as copyright laws, D-Cloud users may want to limit the locationwhere their information will be stored (such as countries or continents). Other geographicalconstraint can be imposed by a maximum (or minimum) physical distance (or delay value) betweennodes. Here, though developers do not know about the actual topology of the nodes, they maymerely establish some delay threshold value for example.Developers should also be able to describe scalability rules, which would specify how andwhen the application would grow and consume more resources from the D-Cloud. Authors in [21]and [9] define a way of doing this, allowing the Cloud user to specify actions that should be taken,like deploying new VMs, based on thresholds of metrics monitored by the D-Cloud itself.Additionally, resource offering is associated to interoperability. Current Cloud providers offerproprietary interfaces to access their services, which can hinder users within their infrastructure asthe migration of applications cannot be easily made between providers [8]. It is hoped that Cloudproviders identify this problem and work together to offer a standardized API.According to [61], Cloud interoperability faces two types of heterogeneities: verticalheterogeneity and horizontal heterogeneity. The first type is concerned with interoperability within asingle Cloud and may be addressed by a common middleware throughout the entire infrastructure.The second challenge, the horizontal heterogeneity, is related to Clouds from different providers.Therefore, the key challenge is dealing with these differences. In this case, a high level of granularityin the modeling may help to address the problem.An important effort in the search for horizontal standardization comes from the Open CloudManifesto5, which is an initiative supported by hundreds of companies that aims to discuss a way toproduce open standards for Cloud Computing. Their major doctrines are collaboration andcoordination of efforts on the standardization, adoption of open standards wherever appropriate,and the development of standards based on customer requirements. Participants of the Open CloudManifesto, through the Cloud Computing Use Case group, produced an interesting white paper [51]highlighting the requirements that need to be standardized in a cloud environment to ensureinteroperability in the most typical scenarios of interaction in Cloud Computing.5 http://www.opencloudmanifesto.org/
  • 35. 22Another group involved with Cloud standards is the Open Grid Forum6, which is intended todevelop the specification of the Open Cloud Computing Interface (OCCI)7. The goal of OCCI is toprovide an easily extendable RESTful interface Cloud management. Originally, the OCCI wasdesigned for IaaS setups, but their current specification [46] was extended to offer a generic schemefor the management of different Cloud services.3.2.3 Resource Discovery and MonitoringWhen requests reach a D-Cloud, the system should be aware of the current status of resources, inorder to determine if there are available resources in the D-Cloud that could satisfy the requests. Inthis way, the right mechanisms for resource discovery and monitoring should also be designed,allowing the system to be aware of the updated status of all its resources. Then, based on the currentstatus and request’ requirements, the system may select and allocate resources to serve these newrequest.Resource monitoring should be continuous and help taking allocation and reallocationdecisions as part of the overall resource usage optimization. A careful analysis should be done tofind a good and acceptable trade-off between the amount of control overhead and the frequency ofresource information updating.The monitoring may be passive or active. It is considered passive when there are one or moreentities collecting information. The entity may continuously send polling messages to nodes askingfor information or may do this on-demand when necessary. On the other hand, the monitoring isactive when nodes are autonomous and may decide when to send asynchronously state informationto some central entity. Naturally, D-Clouds may use both alternatives simultaneously to improve themonitoring solution. In this case, it is necessary to synchronize updates in repositories to maintainconsistency and validity of state information.The discovery and monitoring in a D-Cloud can be accompanied by the development ofspecific communication protocols. Such protocols act as a standard plane for control in the Cloud,allowing interoperability between devices. It is expected that such type of protocols can control thedifferent elements including servers, switches, routers, load balancers, and storage componentspresent in the D-Cloud. One possible method of coping with this challenge is to use smartcommunication nodes with an open programming interface to create new services within the node.One example of this type of open nodes can be seen in the emerging Openflow-enabled switches[44].6 http://www.gridforum.org/7 http://occi-wg.org/about/specification/
  • 36. 233.2.4 Resource Selection and OptimizationWith information regarding Cloud resource availability at hand, a set of appropriate candidates maythen be highlighted. Next, the resource selection process finds the configuration that fulfills allrequirements and optimizes the usage of the infrastructure. Selecting solutions from a set ofavailable ones is not a trivial task due to the dynamicity, high algorithm complexity, and all differentrequirements that must be contemplated by the provider.The problem of resource allocation is recurrent on computer science, and several computingareas have faced such type of problem since early operating systems. Particularly in the CloudComputing field, due to the heterogeneous and time-variant environment in Clouds, the resourceallocation becomes a complex task, forcing the mediation system to respond with minimalturnaround time in order to maintain the developer’s quality requirements. Also, balancingresources’ load and projecting energy-efficient Clouds are major challenges in Cloud Computing.This last aspect is especially relevant as a result of the high demand for electricity to power and tocool the servers hosted on datacenters [7].In a Cloud, energy savings may be achieved through many different strategies. Serverconsolidation, for example, is a useful strategy for minimizing energy consumption whilemaintaining high usage of servers’ resources. This strategy saves the energy migrating VMs ontosome servers and putting idle servers into a standby state. Developing automated solutions forserver consolidation can be a very complex task since these solutions can be mapped to bin-packingproblems known to be NP-hard [72].VM migration and cloning provides a technology to balance load over servers within a Cloud,provide fault tolerance to unpredictable errors, or reallocate applications before a programmedservice interruption. But, although this technology is present in major industry hypervisors (likeVMWare or Xen), there remains some open problems to be investigated. These include cloning aVM into multiple replicas on different hosts [40] and developing VM migration across wide-areanetworks [14]. Also, the VM migration introduces a network problem, since, after migration, VMsrequire adaptation of the link layer forwarding. Some of the strategies for new datacenterarchitectures explained in [67] offer solutions to this problem.Remodeling of datacenter architectures is other research field that tries to overcomelimitations on scalability, stiffness of address spaces, and node congestion in Clouds. Authors in [67]surveyed this theme, highlighted the problems on network topologies of state-of-the-art datacenters,and discussed literature solutions for these problems. One of these solutions is the D-Cloud, as
  • 37. 24pointed also by [72], which offers an energy efficient alternative for constructing a cloud and anadapted solution for time-critical services and interactive applications.Considering specifically the challenges on resource allocation in D-Clouds, one can highlightcorrelated studies based on the Placement of Replicas and Network Virtualization. The former isapplied into Content Distribution Networks (CDNs) and it tries to decide where and when contentservers should be positioned in order to improve system’s performance. Such problem is associatedwith the placement of applications in D-Clouds. The latter research field can be applied to D-Cloudsconsidering that a virtual network is an application composed by servers, databases, and the networkbetween them. Both research fields will be described in following sections.Replica PlacementReplica Placement (RP) consists of a very broad class of problems. The main objective of this typeof problems is to decide where, when, and by whom servers or their content should be positioned inorder to improve CDN performance. The correspondent existing solutions to these problems aregenerally known as Replica Placement Algorithms (RPA) [35].The general RP problem is modeled as a physical topology (represented by a graph), a set ofclients requesting services, and some servers to place on the graph (costs per server can beconsidered instead). Generally, there is a pre-established cost function to be optimized that reflectsservice-related aspects, such as the load of user’s requests, the distance from the server, etc. Aspointed out by [35], an RPA groups these aspects into two different components: the problemdefinition, which consists of a cost function to be minimized under some constraints, and aheuristic, which is used to search for near-optimal solutions in a feasible time frame, since thedefined problems are usually NP-complete.Several different variants of this general problem were already studied. But, according to [57],they fall into two classes: facility location and minimum K-median. In the facility location problem,the main goal is to minimize the total cost of the graph through the placement of a number ofservers, which have an associated cost. The minimum K-median problem, in turn, is similar butassumes the existence of a pre-defined number K of servers. More details on the modeling andcomparison between different variants of the RP problem are provided by [35].Different versions of this problem can be mapped onto resource allocation problems in D-Clouds. A very simple mapping can be defined considering an IaaS service where virtual machinescan be allocated in a geo-distributed infrastructure. In such mapping, the topology corresponds tothe physical infrastructure elements of the D-Cloud, the VMs requested by developers can betreated as servers, and the number of clients accessing each server would be their load.
  • 38. 25Qiu et al. [57] proposed three different algorithms to solve the K-median problem in a CDNscenario: Tree-based algorithm, Greedy algorithm, and Hot Spot algorithm. The Tree-based solutionassumes that the underlying graph is a tree that is divided into several small trees, placing each serverin each small tree. The Greedy algorithm places servers one at a time in order to obtain a bettersolution in each step until all servers are allocated. Finally, the Hot Spot solution attempts to placeservers in the vicinity of clients with the greatest demand. The results showed that the GreedyAlgorithm for replica placement could provide CDNs with performance that is close to optimal.These solutions can be mapped onto D-Clouds considering the simple scenario of VMallocation on a geo-distributed infrastructure with the restriction that each developer has a fixednumber of servers to attend their clients. In such case, this problem can be straightforwardlyreduced to the K-median problem and the three solutions proposed could be applied. Basically, onecould treat each developer as a different CDN and optimize each one independently still consideringa limited capacity of the physical resources caused by the allocation of other developers.Presti et al. [56], treat a RP variant considering a trade-off between the load of requests percontent and the number of replica additions and removals. Their solution considers that each serverin the physical topology decides autonomously, based on thresholds, when to clone overloadedcontents or to remove the underutilized ones. Such decisions also encompass the minimization ofthe distance between clients and the respective accessed replica. A similar problem is investigated in[50], but considering constraints on the QoS perceived by the client. The authors propose amathematical offline formulation and an online version that uses a greedy heuristic. The resultsshow that the heuristic presents good results with minor computational time.The main focus of these solutions is to provide scalability to the CDN according to the loadcaused by client requests. Thus, despite working only with the placement of content replicas, suchsolutions can be also applied to D-Clouds with some simple modifications. Considering replicas asallocated VMs, one can apply the threshold-based solution proposed in [56] to the simple scenarioof VM scalability on a geo-distributed infrastructure.Network VirtualizationThe main problem of NV is the allocation of virtual networks over a physical network [10] and [3].Analogously, D-Clouds’ main goal is to allocate application requests on physical resources accordingto some constraints while attempting to obtain a clever mapping between the virtual and physicalresources. Therefore, problems on D-Clouds can be formulated as NV problems, especially inscenarios considering IaaS-level services.
  • 39. 26Several instances of the NV based resource allocation problem can be reduced to a NP-hardproblem [48]. Even the versions where one knows beforehand all the virtual network requests thatwill arrive in the system is NP-hard. The basic solution strategy thus is to restrict the problem spacemaking it easier to deal with and also consider the use of simple heuristic-based algorithms toachieve fast results.Given a model based on graphs to represent both physical and virtual servers, switches, andlinks [10], an algorithm that allocates virtual networks should consider the constraints of theproblem (CPU, memory, location or bandwidth limits) and an objective function based on thealgorithm objectives. In [31], the authors describe some possible objective functions to beoptimized, like the ones related to maximize the revenue of the service provider, minimizing linkand nodes stress, etc. They also survey heuristic techniques used when allocating the virtualnetworks dividing them in two types: static and dynamic. The dynamic type permits reallocatingalong the time by adding more resources to already allocated virtual networks in order to obtain abetter performance. The static one means once a virtual network is allocated it will hardly everchange its setup.To exemplify the type of problems studied on NV, one can be driven to discuss the onestudied by Chowdhury et al. [10]. Its authors propose an objective function related to the cost andrevenue of the provider and constrained by capacity and geo-location restrictions. They reduce theproblem to a mixed integer programming problem and then relax the integer constraints through thederiving of two different algorithms for the solution’s approximation. Furthermore, the paper alsodescribes a Load Balancing algorithm, in which the original objective function is customizedin order to avoid using nodes and links with low residual capacity. This approach implies inallocation on less loaded components and an increase of the revenue and acceptance ratio ofthe substrate network.Such type of problem and solutions can be applied to D-Clouds. One example could be theallocation of interactive servers with jurisdiction restrictions. In this scenario, the provider mustallocate applications (which can be mapped on virtual networks) whose nodes are linked and thatmust be close to a certain geographical place according to a maximum tolerated delay. Thus, aprovider could apply the proposed algorithms with minor simple adjustments.In the paper of Razzaq and Rathore [58], the virtual network embedding algorithm is dividedin two steps: node mapping and link mapping. In the node mapping step, nodes with highestresource demand are allocated first. The link mapping step is based on an edge disjoint k-shortestpath algorithm, by selecting the shortest path which can fulfill the virtual link bandwidth
  • 40. 27requirement. In [42], a backtracking algorithm for the allocation of virtual networks onto substratenetworks based on the graph isomorphism problem is proposed. The modeling considers multiplecapacity constraints.Zhu and Ammar [74] proposed a set of four algorithms with the goal of balancing the load onthe physical links and nodes, but their algorithms do not consider capacity aspects. Their algorithmsperform the initial allocation and make adaptive optimizations to obtain better allocations. The keyidea of the algorithms is to allocate virtual nodes considering the load of the node and the load ofthe neighbor links of that node. Thus one can say that they perform the allocation in a coordinatedway. For virtual link allocation, the algorithm tries to select paths with few stressed links in thenetwork. For more details about the algorithm see [74].Considering the objectives of NV and RP problems, one may note that NV problems are ageneral form of the RP problem: RP problems try to allocate virtual servers whereas NV considersallocation of virtual servers and virtual links. Both categories of problems can be applied to D-Clouds. Particularly, RP and NV problems may be respectively mapped on two different classes ofD-Clouds: less controllable D-Clouds and more controllable ones, respectively. The RP problemsare suitable for scenarios where allocation of servers is more critical than links. In turn, the NVproblems are especially adapted to situations where the provider is an ISP that has full control overthe whole infrastructure, including the communication infrastructure.3.2.5 SummaryThe D-Clouds’ domain brings several engineering and research challenges that were discussed in thissection and whose main aspects are summarized in Table I. Such challenges are only starting toreceive attention from the research community. Particularly, the system, models, languages, andalgorithms presented in the next chapters will cope with some of these challenges.Table I Summary of the main aspects discussedCategories AspectsResource ModelingHeterogeneity of resourcesPhysical and virtual resources must be consideredComplexity vs. FlexibilityResource Offeringand TreatmentDescribe the resources offered to developersDescribe the supported requirementsNew requirements: topology, jurisdiction, scalabilityResource Discoveryand MonitoringMonitoring must be continuousControl overhead vs. Updated informationResource Selectionand OptimizationFind resources to fulfill developer’s requirementsOptimize usage of the D-Cloud infrastructureComplex problems solved by approximation algorithms
  • 41. 284 The Nubilum System“Expulsa nube, serenus fit saepe dies.”Popular ProverbSection 2.4 introduced an Archetypal Cloud Mediation system focusing specifically on the resourcemanagement process that ranges from the automatic negotiation of developers requirements to theexecution of their applications. Further, this system was divided into three layers: negotiation,resource management, and resource control. Keeping in mind this simple archetypal mediationsystem, this chapter presents Nubilum a resource management system that offers a self-managedsolution for challenges resulting from the discovery, monitoring, control, and allocation of resourcesin D-Clouds. This system appears previously in [25] under the name of D-CRAS (Distributed CloudResource Allocation System).Section 4.1 presents some decisions taken to guide the overall design and implementation ofNubilum. Section 4.2 presents a conceptual view of the Nubilum’s architecture highlighting theirmain modules. The functional components of Nubilum are detailed in Section 4.3. Section 4.4presents the main processes performed by Nubilum. Section 4.5 closes this chapter by summarizingthe contributions of the system and comparing them with correlated resource management systems.4.1 Design RationaleAs stated previously in Section 1.2, the objective of this Thesis is to develop a self-manageablesystem for resource management on D-Clouds. Before the development of the system and theircorrespondent architecture, some design decisions that will guide the development of the systemmust be delineated and justified.4.1.1 ProgrammabilityThe first aspect to be defined is the abstraction level in which Nubilum will act. Given that D-Clouds concerns can be mapped on previous approaches on Replica Placement (see Section 0) andNetwork Virtualization (see Section 0) research areas, a straightforward approach would be toconsider a D-Cloud working at the same abstraction level. Therefore, knowing that proposals inboth areas commonly seem to work at the IaaS level, i.e., providing virtualized infrastructures,Nubilum would naturally also operate at the IaaS level.
  • 42. 29Nubilum offers a Network Virtualization service. Applications can be treated as virtualnetworks and the provider’s infrastructure is the physical network. In this way, the allocationproblem is a virtual network assignment problem and previous solutions for the NV area can beapplied. Note that such approach does not exclude previous Replica Placement solutions becausesuch area can be viewed as a particular case of Network Virtualization.4.1.2 Self-optimizationAs defined in Section 2.1, the Cloud must provide services in a timely manner, i.e., resourcesrequired by users must be configured as quickly as possible. In other words, to meet such restriction,Nubilum must operate as much as possible without human intervention, which is the very definitionof self-management from Autonomic Computing [69].The operation involves maintenance and adjustment of the D-Cloud resources in the face ofchanging application demands and innocent or malicious failures. Thus, Nubilum must providesolutions to cope with the four aspects leveraged by Autonomic Computing: self-configuration, self-healing, self-optimization, and self-protection. Particularly, this Thesis focuses on investigating self-optimization – and, at some levels possibly, self-configuration – on D-Clouds. The other twoaspects are considered out of scope of this proposal.According to [69], self-optimization of a system involves letting its elements “continually seekways to improve their operation, identifying and seizing opportunities to make themselves moreefficient in performance or cost”. Such definition fits very well the aim of Nubilum, which mustensure an automatic monitoring and control of resources to guarantee the optimal functioning ofthe Cloud while meeting developers’ requirements.4.1.3 Existing standards adoptionThe Open Cloud Manifesto, an industry initiative that aims to discuss a way to produce openstandards for Cloud Computing, states that Cloud providers “must use and adopt existing standardswherever appropriate” [51]. The Manifesto argues that several efforts and investments have beenmade by the IT industry in standardization, so it seems more productive and economic to use suchstandards when appropriate. Following this same line, Nubilum will adopt some industry standardswhen possible. Such adoption is also extended to open processes and software tools.4.2 Nubilum’s conceptual viewAs shown in Figure 6, the conceptual view of Nubilum’s architecture is composed of three planes: aDecision plane, a Management plane, and an Infrastructure plane. Starting from the bottom, thelower plane nestles all modules responsible for the appropriate virtualization of each resource in the
  • 43. 30D-Cloud: servers, dedicated storage, and network, which includes links, routers, and switches. TheManagement plane is responsible for the monitoring and control of the D-Cloud, as well as theenforcement of allocation decisions taken by the upper layer. Finally, at the top of the architecture,the Decision plane is where the advanced strategies and heuristics to allocate resources areimplemented. In the following sections the modules will be detailed.Figure 6 Nubilum’s planes and modules4.2.1 Decision planeThe Decision plane is composed of four modules: Negotiator, Mapper, Application Monitors, andthe D-Cloud Monitor. The Negotiator module is the front-end of Nubilum and is similar to theNegotiation Layer presented in the Archetypal Mediation System at Section 2.4. Thus, theNegotiator offers an API that can be used by developers to contract and control their virtualresources. In this system, these operations will be implemented as Web Services using a descriptivelanguage whose definitions are illustrated in Chapter 5.The Mapper is responsible for deciding the mapping of the required virtual resources on thecorresponding physical resources. It is important to note that, in Nubilum, the mapping algorithmswork with a model of the entire infrastructure, as the effective allocation is the responsibility oflower planes. Thus, the Mapper inputs are a representation of a developer’s request and arepresentation of the current status of the D-Cloud and the output is a new representation of the D-Cloud status to be enforced over the real system. It is important to stress that the Mapper modulecontains the main intelligence of the system and it is responsible for finding a configuration thatfulfills all computational and network resources’ requirements and optimizes the usage of the D-Cloud infrastructure. The allocation algorithms used as part of Nubilum will be discussed at Chapter6.Infrastructure PlaneManagementPlaneDecisionPlaneMapperNetworkMonitorServerMonitorStorageMonitorNetworkControllerServerControllerStorageControllerResourceMaestroServerVirtualizationNetworkVirtualizationStorageVirtualizationApplicationManagersApplicationManagersApplicationMonitorNegotiatorResourceDiscovererD-CloudMonitor
  • 44. 31The Decision plane also has a set of modules managing the applications of each developer: theApplication Monitors. The Application Monitor is responsible by the monitoring part of the self-management loop of the system, since it periodically checks an application to guarantee the fulfillingof their requirements. Each application submitted to Nubilum has an Application Monitorassociated with it. This module continuously checks the current status of the application against itsrequirements and selects the appropriate actions when attempting to improve the application’sperformance, if that is necessary. Please note that the Application Monitor does not make allocationdecisions; it is merely responsible for detecting performance degradations and requesting newresources from the Allocator.The provider requirements - with respect to the usage of physical resources - are constantlymonitored and verified through the D-Cloud Monitor module. Similar to the Application Monitormodules, the D-Cloud Monitor is other module acting in the self-management loop throughmonitoring of the physical resources. It checks the current state of the physical resources against theprovider’s requirements. Upon a detection of surpassing a threshold (e.g: an increase of server loadbeyond an established threshold), the D-Cloud Monitor communicates the Mapper soliciting the re-optimization of the infrastructure.4.2.2 Management planeNubilum divides D-Cloud resources in three categories: server, network, and storage. In the servercategory there are all the devices that can host virtual machines. The network resources representlinks, network devices (routers and switches), and protocols that compose the underlying topologyof the D-Cloud. Finally, the storage resources are the nodes dedicated to storing virtual machineimages, files, or databases.Considering such division, the Management plane has different modules for controlling andmonitoring each category of resources. Thus, the Server Controller and Server Collector arerespectively responsible for the control of the servers and hosted VMs, and for the acquisition ofupdated status of the server and VMs. The Network Collector is responsible for obtaining updatedinformation about the state of the network and the load of links, whereas the Network Controller isresponsible for communicating decisions about the creation, modification, and removal of virtuallinks to network devices. Further, this controller manages the assignments of IP and MAC addressesto virtual resources. Similarly, the Storage Controller and the Storage Collector are responsible forcontrolling and collecting storage resources. All such information about resource monitoring is sentto the D-Cloud Monitor and Application Monitors.
  • 45. 32Associated with the individual controllers and collectors of each resource is the ResourceDiscoverer, which is responsible by the detection of resources in the D-Cloud and for maintaininginformation about their respective status, which will be collected by the respective monitors ofcomputing, storage, and network resources.Finally, the Resource Maestro orchestrates the creation and removal of the virtual resources inthe respective physical resources according to the decisions made by the modules at the DecisionPlane. When a new request arrives to Nubilum and the Mapper module decides where the newvirtual resources will be allocated, the Maestro enforces this decision by communicating it to thecomponents involved. One important task of this module is that it must translate high level orderstaken by the Mapper into low level requests that can be handled by the other modules. Suchtranslations must be made carefully, considering the application’s requirements, once the order ofthe enforcement of any changes may alter the application’s execution. For example, consider avirtual network of two virtual nodes A and B and one direct virtual link between them. Positioning anew virtual node C in the middle (with one virtual link from A to C and another one between A andB) involves the coordinated creation of these two virtual links in order to minimize interruptions inthe intercommunications of the virtual nodes.4.2.3 Infrastructure planeThe infrastructure plane offers tools used for the appropriate virtualization of servers, networkelements and storage devices in the D-Cloud. As a result, this plane requires three modules toaccomplish its tasks.The Server Virtualization is the module responsible for the effective management of resourcesin the physical servers. It also manages the creation and maintenance of virtual machines, i.e., itcorresponds to the hypervisor installed in the physical servers. Thus, it is important to note that,differently from the Server Controller and Server Monitor modules already described, the ServerVirtualization module is intended for the local control of a server and its resources.The Network Virtualization module corresponds to the platform used for the effectivevirtualization of the network and the associated protocols to accomplish this task. Similarly, theStorage Virtualization comprehends all the technologies used for the effective control of storagecomponents in the D-Cloud.4.3 Nubilum’s functional componentsThis section describes how the modules presented in the conceptual view in Section 4.2 arerearranged to create the functional components of Nubilum. Moreover, this section provides
  • 46. 33information about the technological solutions taken to overcome some D-Cloud-related designissues.The functional components in Nubilum are: the Allocator (located at the Decision plane inthe conceptual view), the Manager (that has modules from the Decision and Management plane), theStorage System (situated at the Infrastructure plane), the Network Devices (situated at theInfrastructure plane), and the Workers (situated in a mid-term between the Management plane andthe Infrastructure plane). Figure 7 illustrates these five components and their respective modules. Atthe top are the Allocator and the Manager responsible, respectively, for taking decisions in theallocation of incoming requests and for the overall control of the D-Cloud resources. The otherthree components are responsible for operational tasks in each type of physical resource.Figure 7 Functional components of Nubilum4.3.1 AllocatorThe Allocator (showed at Figure 8) is responsible for treating the resource requests made bydevelopers and for mapping the requested virtual resources on the physical resources of the D-Cloud.Figure 8 Schematic diagram of Allocator’s modules and relationships with other componentsManagerWorkerWorkerWorkersAllocatorNetwork DevicesStorageSystemMapper NetworkMonitorResourceMaestroNetworkVirtualizationStorageVirtualizationNetworkControllerStorageControllerStorageMonitorNetworkControllerServerMonitorServerVirtualizationServerControllerResourceDiscovererApplicationManagersApplicationManagersApplicationManagersNegotiator D-CloudManagerAllocatorMapperNegotiatorManagerCloudProviderCloudDeveloper
  • 47. 34This Allocator needs to be as simple as possible in order to leverage system performance,since the optimization problems for resource allocation tend to be computationally intensive.Therefore, it has just the Mapper module (part of the Decision plane), which communicates with theManager to obtain information about the D-Cloud state and sends commands to enforce newdecisions, through a REST-based API. The Negotiator module of the Decision plane is the externalinterface of Nubilum, which offers a REST API for communication with developers and theprovider.4.3.2 ManagerThe Manager is the central component in Nubilum that coordinates the overall control of the D-Cloud and maintains updated information about the status of all its resources. It has four modulesfrom the Management plane (Network Controller, Network Collector, Resource Discoverer, andResource Maestro) and two modules from the Decision plane (Application Monitor, and D-CloudMonitor), which are represented in Figure 9. This figure also shows the communication with othercomponents [44].Figure 9 Schematic diagram of Manager’s modules and relationships with other componentsThe Resource Discoverer module uses a simple process for discovering servers and networkdevices based on the prior registration of each individual device at the Manager. Thus, the knownaddress of the Manager must be manually configured on the Worker and the Network Devices. Theacquiring of status information of the Network Devices is made through periodical queries to theOpenflow Collector module, while the status of each Worker is sent by them when a configuredthreshold is crossed. All information is passed and maintained by the Resource Discoverer module.The Network Controller and Network Collector modules implement the functions of controland status collection of network devices individually. This module assumes the usage of Openflowas a platform for network virtualization. This platform was chosen because of its versatility. It allowsconfiguring Network Devices in order to setup a path in the physical network corresponding to thevirtual link of an application. This module can be implemented using current APIs which supportthe Openflow protocol or, alternatively, through invoking Openflow controllers such as NOX [29].ManagerWorkersWorkersNetworkMonitorResourceMaestroResourceDiscovererApplicationManagersApplicationManagersApplicationManagersD-CloudManagerInterfaceAllocatorWorkersWorkersWorkersNetworkDevicesNetworkController
  • 48. 354.3.3 WorkerThe basic component of Nubilum is the Worker. This component is installed on each physicalserver and its main function is to manage the entire life-cycle of the virtual machines hosted on thehypervisor. A schematic view of the relationship between the modules of this component ispresented in Figure 10.Figure 10 Schematic diagram of Worker modules and relationships with the server systemFrom the Infrastructure plane, the Workers execute the Server Virtualization module. TheWorkers are required to support third-party hypervisors as Xen, VMWare, or KVM, for example.To avoid the implementation of the different drivers supporting each one of these hypervisors intothe Worker, the Libvirt [5] open-source API is used. Libvirt provides a common generic and stablelayer for VM management and offers an abstraction to Cloud providers since it supports theexecution on different hypervisors like: Xen, KVM, VirtualBox, and VMWare ESX. ThroughLibvirt, the Workers can effectively control and monitor several aspects of virtual machines, whichcan be simply consumed by the other modules. The advantage obtained when using Libvirt isnotorious, since this API offers a hypervisor-agnostic alternative for virtual machine control. Inaddition, Libvirt leverages an abstraction of other aspects associated with the virtual machine asstorage and network manipulation.Workers have several modules. For the control and monitoring of the physical server andvirtual machines hosted on it, there are the Server Controller and the Server Collector modules. TheServer Controller is responsible for the overall tasks involved in the creation, maintenance, anddestruction of virtual machines, while the Server Collector maintains records of the status of theWorkerStorageControllerStorageMonitorServerMonitorServerVirtualizationServerControllerNetworkControllerLibvirtLibvirt InterfaceVM1 VM2 VMn...Manager
  • 49. 36server and the virtual machines hosted on it. To accomplish these tasks, these modules coordinatethe execution of the other modules hosted at the Worker, call the hypervisor through the LibvirtAPI and invoke the operating system through some host monitoring API available in theprogramming language (in case of Java, one could use the JavaSysMon system call library [33]).The Storage Controller and Storage Collector modules manage the devices used for storingvirtual machine images. They maintain the storage pool while creating and deleting virtual machineimages. For this, the modules use Libvirt, which offers a common interface to manage manydifferent types of storage spaces ranging from iSCSI devices to a simple directory in the samemachine. Thus, these modules can access any type of Storage, making Nubilum independent of theactual underlying storage scheme used on the D-Cloud, while the only assumption made is that allfiles are potentially accessible from every machine in the D-Cloud. More about the storage in the D-Cloud can be found in the corresponding section.The Workers are responsible for the effective assignment of IPs and MACs to the virtualmachines, this task is performed by the Network Controller module, which is composed by a built-in DHCP server and a LLDP (Link Layer Discovery Protocol) agent. The DHCP module isdynamically configured with a direct mapping between the MAC and IP numbers, which are used togive reserved IPs to each virtual machine hosted on the server. Please note that the presence ofseveral DHCP servers (one per Worker) does not produce interferences between them, since eachbuilt-in DCHP server will only respond to requests of the virtual machines hosted on the Worker,whereas requests with unknown MAC addresses will be dropped. The LLDP is employed for theadvertisement of LLDP messages used in the discovery of their links. The use of LLDP fordiscovery is detailed in Section 4.4.2.In addition to the modules used for the effective control and monitoring of the server andvirtual machines, the Worker has an Interface for communicating with Manager component. Suchinterface is a REST-based Web Service.4.3.4 Network DevicesNubilum considers using a network virtualization platform for the setup and maintenance of thevirtual links across the system. On the current version of Nubilum, it is required that all networkdevices in the system implement Openflow for network virtualization purposes. The advantage ofadopting such a solution is that not only is there a native support for the creation of virtual links, butalso no adaptations must be made to the network devices. This component has one only modulefrom the Conceptual View: the Network Virtualization module.
  • 50. 374.3.5 Storage SystemThe Storage System covers the Storage Virtualization module responsible by virtualize all the storageresources spread along the D-Cloud. As earlier stated, Nubilum does not enforce the usage of anyactual storage technology type, but entrusts this to Libvirt’s coverage of storage technologies. Theonly requirement is that all the virtual machine images are available to all servers in the D-Cloud.This occurs because Nubilum depends on the current virtual machine migration techniques thathave this particular requirement.Please note that the D-Cloud should not employ centralized storage solutions, but it canemploy distributed storage solutions, such as the Sector project [28], a file system designed fordistributed storage over WANs.4.4 ProcessesSeveral processes in Nubilum are performed as a result of the coordination between thecomponents. The present section will discuss these processes, which can be organized into threeclasses: initialization processes, discovery processes, and resource request processes.4.4.1 Initialization processesThe initialization process of the components that make up Nubilum is simple, since eachcomponent is configured before its initialization. The first component to be initiated is the Storagesystem as it is a third-party system. Similarly, the network devices must be configured with routes foraccessing the overall physical resources in the system and with the indication of the Openflowcontroller address (i.e., the known IP address of the Manager component). The connection followsthe standardized process specified in the Openflow documentation [52].The last infrastructural component, the Worker, is installed in each server of the D-Cloud andits configuration includes the Manager’s address for registration. This configuration also includesinformation about storage pools, geographical location, and base images to be used for virtualmachine creation. When started, the Worker connects with the local Libvirt API and obtainsinformation about the server and current virtual machines hosted on the server. Using suchinformation, the Worker prepares a description of the local server and virtual machines it hosts andthen initiates a simple registration process in the Manager, through the REST interface. If theManager is not initialized yet, the Worker will try again after a random sleep period.The Manager can be initialized either on a reserved server or on a virtual machine hosted by aserver. Its initialization process opens an HTTP server for listening to REST messages and waits forconnections from Workers and Allocator components. Also, an Openflow controller is started to
  • 51. 38listen for incoming connections of Network Devices. As with the Manager, the Allocator can beinitialized on any server of the D-Cloud or on a virtual machine, alternatively. The address of theManager is configured before the Allocator’s initialization.4.4.2 Discovery and monitoring processesThe resource discovery comprehends the processes for finding new resources and acquiring updatedstatus on these. The initialization processes of Workers and Network Devices – which involves theregistration of both into the Manager – is the first part of the discovery processes that find newresources on the D-Cloud. However, such processes are not sufficient to discover the connectionsbetween physical resources in the D-Cloud. Therefore, Nubilum employs a discovery strategysupported by NOX, which makes use of LLDP messages generated by the Manager and sent byeach Network Device to their neighbors [24].The link discovery strategy is illustrated at Figure 11. Firstly, the Manager sends an Openflowmessage (number 1 in the figure) to a switch in the network requesting forwarding of a LLDPpacket for each of its neighbor switches. When such neighbors receive the LLDP packet (arrowsmarked as 2) they generate a new Openflow message (marked as 3) to the Manager informing thereceiving of this specific LLDP packet. The same process is executed for all switches concurrentlyand the LLDP packet of each switch is specific for that switch in order to identify each link. Such astrategy guarantees the discovery of all links between Network Devices in the D-Cloud, since it isassumed that all Network Devices are Openflow-enabled. This process was extended to captureLLDP messages sent by Workers in order to discover links between servers and Network Devices.Figure 11 Link discovery process using LLDP and OpenflowCollecting information about the status of each resource involves retrieving information aboutthe Storage System, the Network Devices, and the Workers. The performance status of the NetworkManager1222333NetworkDeviceNetworkDeviceNetworkDeviceNetworkDevice
  • 52. 39Devices is monitored by the Manager in a passive fashion by polling the resources according to aconfigured period. Such polling is made through well-defined Openflow messages, which caninform several Nubilum counters as, for example, the number of received packets or bytes pervirtual link in a network device (flows in Openflow terminology).Differently from the Network Devices, the Workers actively send status information throughthe Storage and Server Collector modules. The information offered by the Worker is closely relatedto the support provided by Libvirt. It is possible to obtain information about CPU and Memoryusage and the disk used space in the server and virtual machines.4.4.3 Resource allocation processesThe resource allocation processes are started up when developers, the Application Monitors,or the D-Cloud Monitor, contact the Allocator requiring, respectively, resources for a newapplication, more resources for an existent application, or the reallocation of resources to meetprovider’s requirements. Figure 12 shows all messages exchanged by the resource allocation processwhen a developer requests resources for a new application. The developer sends to the Allocator aPOST message containing a CloudML document (see Chapter 5 for details about this language)describing the requirements of its application, i.e., a virtual network with virtual machines, itsgeographical position, and virtual links. After that, the Allocator sends a GET message to theManager to retrieve the current status of the entire D-Cloud, which will be used as input to theresource allocation algorithms.Figure 12 Sequence diagram of the Resource Request process for a developerPUT /dcloud (CloudML)CloudDeveloperAllocator Workers Network DevicesPOST App(CloudML)GET DCloudCloudMLResourceAllocationPOST/virnode (CloudML)POST/virnode (CloudML)POST/virnode (CloudML)ReplyPOST (CloudML)ReplyPOST (CloudML)ReplyPOST (CloudML)Create flow (Openflow)Create flow (Openflow)Create flow (Openflow)ReplyPUT (CloudML)ReplyPOST (CloudML)ManagerNubilum
  • 53. 40After the resource allocation algorithm execution, the Allocator sends a PUT message to theManager indicating what resources (and their respective configuration) must be dedicated to thisgiven developer. Then, the Manager sends POST messages to each Worker in the D-Cloud that willhost the requested virtual resources, receives respective replies, and sends Openflow messages toeach Network Device informing them of the flows for the setup of new virtual links. Then, theManager sends a reply PUT to the Allocator to confirm the allocation, and finally, the Allocatorreturns a reply POST to the developer indicating if it was possible to allocate their application andits respective IP address.The processes triggered by the D-Cloud Manager or the Application Managers are similar tothe one presented in Figure 12, except that these modules use the PUT instead of the POST methodin the initial message.4.5 Related projectsThis chapter presented the implementation guidelines of Nubilum – a resource management systemthat offers a solution for challenges related to allocation, discovery, control, and monitoring ofresources for ISP-based Distributed Clouds (please see Section 3.1 for more details). The systemmanages a D-Cloud as an IaaS Cloud offering developers an environment that manages the entirelife-cycle of their applications ranging in scope from the request of resources, which are allocatedinto a virtualized infrastructure of scattered geo-distributed servers connected by network devices, totheir removal from the D-Cloud.In order to obtain optimal or quasi-optimal solutions while maintaining scalability, Nubilumhas a centralized decision ingredient with two high-level components: the Allocator and theManager; and a decentralized control ingredient with three infrastructural components: Workers,Network Devices, and the Storage System. It also introduces a clear separation between theenforcement actions and the intelligence roles played by the Manager and the Allocator, respectively.The Manager offers an abstraction of the overall D-Cloud to the Allocator, which, in turn, isdescribed in a high-level perspective, where only the functionalities and communication processeswere defined. Thus, the proposed system intends to remain open for different model-drivenresource allocation strategies.Some architectural ideas present in Nubilum are similar to the ones presented by open sourceresource management systems for Cloud Computing like, Eucalyptus, OpenNebula, and Nimbus,which are compared in [24]. These systems propose solutions to some problems that can arise in aD-Cloud scenario, thus providing good starting solutions for the design of resource managementsystems for D-Clouds. Beyond the centralized resource management, such systems also are based in
  • 54. 41open interfaces and open tools like, respectively, Web Service-based interfaces (REST in case ofNubilum) and Libvirt as a hypervisor-agnostic solution. Particularly from OpenNebula, Nubilumleverages the idea of a conceptual stacked view separated from a functional view, as well as theseparation between the decision and the control tasks. Despite these similarities, those systems havea lack of direct support to D-Clouds, mainly with regard to the virtualization of the network, whichis a main aspect of D-Clouds discussed in this Thesis.Differently from the above-mentioned systems, the RESERVOIR project proposes a model,architecture, and functionalities for Open Federated Cloud Computing, which is a D-Cloud scenariowhere a Cloud provider may dynamically partner with others to provide a seemingly infinite resourcepool [59]. To achieve this goal, RESERVOIR leverages virtualization technologies and embedsautonomous management into the infrastructure. To cope with networking aspects, the architecturedesigns a scheme of Virtual Ethernet [30], which offers isolation while sharing network resourcesbetween the federated Clouds composing the system. In contrast, following the design decision touse existing standards, Nubilum employs Openflow as a solution for network virtualization.An initiative that works with an ISP-based D-Cloud scenario is the research project calledGEYSERS [22]. This project extends standard GMPLS-related traffic engineering protocols tovirtual infrastructure reservation, i.e., for the integrated allocation of computational and networkingresources. Nubilum adopts a different approach, working with two different standards forcommunication: REST-based interfaces combined with Libvirt for the allocation of server andstorage resources and Openflow for allocating network links. The integration of these standards ismade by the internal algorithms of the Manager. Such strategy is interesting since the system can bedeployed without support of additional protocols.Also working with an ISP-based D-Cloud, one of the SAIL project objectives is to provideresource virtualization through the CloNe (Cloud Networking) architecture [60]. CloNe addressesthe networking aspect in Cloud Computing and introduces the concept of a FNS (Flash NetworkSlice), a network resource that can be provisioned on a time scale comparable to existing computeand storage resources. The implementation of this concept already is an object of study in the SAILproject, but some solutions were presented ranging from VPNs to networks based on Openflowdevices. Thus, the network virtualization solution presented in Nubilum, which is based onOpenflow, can be viewed as a first implementation of the FNS proposed by CloNe.Another distinction between Nubilum and other existing resource managementsystems/architectures is that Nubilum is focused on the resource management properly whereas
  • 55. 42other practical aspects as security and fault tolerance were not considered, although Nubilum couldbe extended to consider those important aspects.Hitherto, one specific critic that can be made to Nubilum is that some aspects of the systemwere generically described. One example is their openness to the usage of different algorithms forresource allocation, which makes it difficult to see how the system can guarantee designs decisionrelated to self-optimization. Such aspects will be discussed in the next chapters what will specializethe architectural specifications further. Chapter 5 details the overall communication protocols andtheir respective messages used in the control plane of the system. Basically, the next chapter willdescribe the HTTP and Openflow messages used by the components as well as it will describe themodeling language CloudML. Furthermore, algorithms for resource allocation derived for specificcases will be presented at Chapter 6.
  • 56. 435 Control Plane“Si quam rem accures sobrie aut frugaliter, solet illa recte sub manus succedere.”Plautus PersaThis chapter details and evaluates the control plane of Nubilum. It consists of two main elements:the HTTP and Openflow messages for communication between the components in Nubilum, andthe Cloud Modeling Language (CloudML) that is used to describe services, resources andrequirements. Such elements are intrinsically correlated since the language represents data that isexchanged by the HTTP messages and that defines the number and format of the Openflowmessages. These also specify how developers and the provider interact with the D-Cloud and howthe effective resource allocation is made on the D-Cloud.Following the above division of the control plane, this chapter is split in three main sections: itstarts by presenting CloudML and discussing its characteristics with respect to similar descriptionlanguages in Section 5.1; next, in Section 5.2, the communication interfaces and protocols areexplained; finally, Section 5.3 evaluates the overall control plane solution.5.1 The Cloud Modeling LanguageAs explained in Chapter 3, the representation of user requirements and cloud resources is the firstchallenge to face when considering an automatic resource management system. This chapterintroduces the Cloud Modeling Language (CloudML), an XML-based language intended to copewith the aforementioned required representations. CloudML is proposed to model service profilesand developer’s requirements, while at the same time, to represent physical and virtual resourcestatus in D-Clouds. This language was previously introduced in [26].Considering previous languages ([9], [39], [32], [16], [73], [47]) for the representation ofresources and their respective limitations (more about this topic is discussed in Section 5.1.3), it wasdecided to design the Cloud Modeling Language (CloudML) to ensure three clear objectives: a) thelanguage must represent all physical and virtual resources in a D-Cloud, including their current state;b) the proposed language must be able to model the service supported by the provider; and c) thelanguage must represent a developer’s requirements while also relating them to the provider’sservices.
  • 57. 44In order to grasp how CloudML offers the integration of such objectives, please considerFigure 13. This figure depicts a scenario with three actors: the application developer, the Cloudprovider, and Nubilum. Further, the figure shows the interactions between these actors through theuse of a respective description in CloudML.Figure 13 Integration of different descriptions using CloudMLFirst, the Cloud provider should describe all services offered by the D-Cloud, generating aservice description document (step number “1” in the figure). Next, a Cloud developer may usethese descriptions (step number “2”) to verify if its requests are attended by the concerned D-Cloud.Note that the CloudML allows different D-Cloud providers to comply with their respective servicedescriptions. In this way, a developer may choose between different providers according to its owncriteria and convenience. Once a Cloud provider is selected, the developer composes a documentdescribing its requirements and may then submit its requests to the D-Cloud (step number “3”),more specifically represented by the Cloud System Management, which will ultimately allocateresources according to the requested resources and the current status of the D-Cloud. At any time,the provider may asynchronously request status or the description of resources (step number “4”) ofthe D-Cloud. This same information is the input of the resource management process performed byNubilum.As defined in Section 4.1.1, Nubilum works as an IaaS Cloud. Thus, in CloudML thedevelopers’ applications are treated as virtual networks and the provider’s infrastructure is thephysical network. Another important design aspect is the use XML technologies as the underlyingstructure to compose CloudML documents. By adopting this well-established technology, inServiceDescriptionCloud Operator describes itsservices and makes themavailable1Cloud Developer usesServices’ description tomake its requests2Cloud Operator may requeststatus/description ofresources4ResourceDescriptionRequestDescriptionCloud Developer sends itsrequests to the Cloud3NubilumCloudProviderApplicationDeveloper
  • 58. 45contrast to new ones such as JSON [13], it is possible to use solutions and APIs present in the XMLecosystem to guarantee syntax correction, query of documents, and other facilities.The rest of this section is organized as in the following: Section 5.1.1 presents and details theCloudML language and its XML Schemas; Section 5.1.2 is dedicated to illustrate the use ofCloudML in a simple scenario; finally, a qualitative comparison and discussion between CloudMLand existent languages are given in Section 5.1.3.5.1.1 CloudML SchemasThe XML Schemas of CloudML were divided into three groups: schemas for resource description,schemas for service description, and schemas for requirements description. For didactical purposes,it was opted for presenting those schemas through intuitive diagrams generated by the Web ToolsPlatform (an Eclipse plug-in [18]) instead of the tangled XML Schema code.Resource DescriptionThis first group has two subgroups: one for short reports on the status of resources, and the othergroup provides a complete description of resources.The basic XML element for reporting resources’ status is the NodeStatusType (Figure 14)which represents the status of both physical servers and virtual machines (called just nodes in ourlanguage). This type is composed by two required attributes (CPU and RAM) and a sequence ofStorage elements. These attributes are presented in percentage values while the Storage has a typedefining the absolute value of the used space (Size), the Unit relative to this space (KB, MB, GB, etc),and a logical ID since a node can have many drives for storage.Figure 14 Basic status type used in the composition of other typesThe next type is the VirNodeStatusType that is used to report the status of a specific virtualmachine (Figure 15). Such type has three attributes: the ID which is a unique value used foridentification purposes and defined when the VM is created; the Owner is the identification of thedeveloper that owns such VM; and the VMState that indicates the current state of the VM.CloudML defines three states for the VM: stopped, running, and suspended, which are self-descriptive. The associated type still has a Status element whose type is the NodeStatusType
  • 59. 46already described. The PhyNodeStatusType is similar to the one for virtual nodes, except for theomission of the VMState and Owner attributes.Figure 15 Type for reporting status of the virtual nodesThe NodesStatusType gives information about the status of the whole resources managedby the Worker. The NodesStatusType has only one root element called Nodes, as showed inFigure 16.Figure 16 XML Schema used to report the status of the physical nodeOne basic element for complete descriptions of physical nodes is the PhysicalNodeType(Figure 17). This type has the ID attribute and four inner elements: NodeParams, PhyIface,VirNodeID, and VirEnvironment.Figure 17 Type for reporting complete description of the physical nodesThe NodeParametersType (Figure 18) describes relevant characteristics including: nodeparameters (memory, processor, and storage), its geographical location (Location element), its
  • 60. 47Functionality on the network (switch, server, etc…), its current status (which is an element ofthe type NodeStatusType) and the OtherParams for general use and extension.Figure 18 Type for reporting the specific parameters of any nodeHere, there are two aspects that should be highlighted. First, the Location is the element thatenables a provider to know where resources are geo-located in the infrastructure. Second, theOtherParams element can be used by providers or equipment vendors to extend CloudMLincluding other parameters not covered by this current version. In this way, CloudML presents itselfas an extensible language.The PhysicalInterfaceType (Figure 19) is an extension of InterfaceType and is used todescribe physical links associated to the interface (PhysicalLinksID element) and virtual interfaces(VirtualInterfacesID element) also related to the physical node. Such interfaces can be, forexample, interfaces from virtual nodes. The general InterfaceType has an ID, MAC, IPv4, andIPv6 as attributes that are inherited by the PhysicalInterfaceType.
  • 61. 48Figure 19 Type for reporting information about the physical interfaceAs part of the PhysicalNodeType, the VirNodeID is a simple list of the IDs of the virtualmachines hosted on the node, and the VirEnvironment is a list containing information about thevirtualization environment. Each item in the list informs its CPU architecture (32 or 64 bits), thevirtualization method (full or paravirtualized), and the hypervisor. Thus, an item indicates a type ofvirtual machine supported.The VirtualNodeType (Figure 20) gives a complete description of a virtual machine and issimilar to the physical node. The VirtualInterfaceType also inherits from the InterfaceType,and the VirEnvironment contains only two attributes: one indicating the hypervisor and the otherindicating the virtualization mode of the VM.Figure 20 Type for reporting information about a virtual machine
  • 62. 49The VirNodesDescription and the NodesDescription are lists similar to the ones definedinto the Status XML Schemas.The InfraStructureType (Figure 21) is composed by a PhyInfra element and zero ormore VirInfra elements. The element PhyInfra is a PhysicalInfraStructureType andcorresponds to the collection of physical nodes and links. The VirtualInfraStructureTypeindicates virtual infrastructures currently hosted by the physical infrastructure.Figure 21 Type for reporting information about the whole infrastructureThe PhysicalInfraStructureType (Figure 22) has an ID attribute and is composed by twoelements: PhyNode and PhyLink; which clearly represent the nodes (computers, switches, etc…)and their connections (cable links, radio links, etc…), respectively. The PhyNode element is of thetype PhysicalNodeType which was already described whereas the PhyLink is of the typePhyisicalLinkType (Figure 23).Figure 22 Type for reporting information about the physical infrastructure
  • 63. 50Figure 23 Type for reporting information about a physical linkThe PhysicalLinkType describes physical links between physical nodes. It has an IDattribute, a LinkParams element, and zero or more VirLinkID elements. TheLinkParametersType, just as the NodeParametersType, supports all relevant characteristics ofthe link, which include: link technology (Ethernet, Wi-Fi, etc…), capacity, the current status (currentdelay, current allocated rate and current bit error rate), and also an extensible element(OtherParams) serving for future extension purposes. The VirLinkID element identifies the virtuallinks currently allocated on this physical link.Similarly to the physical infrastructure there is a type dedicated towards the collection ofvirtual nodes and virtual links called VirtualInfraStructureType (Figure 24). It has an ID, anOwner attribute (identifying the developer who owns these virtual resources) and can be composedof one or more VirNode elements (of the described VirtualNodeType) and several VirLinkelements of the VirtualLinkType, which is very similar to the type for physical links.Figure 24 Type for reporting information about the virtual infrastructureService DescriptionCloudML provides a profile-based method for describing a provider’s services, which are describedby an XML Scheme whose root element is Services from the ServiceType (Figure 25). This typehas a Version attribute and a sequence of Node and Link profiles elements. The Node element
  • 64. 51consists of the nodes profiles that are described by the NodeProfileType whereas the Linkelement uses the LinkProfileType. A Coverage element, from the CoverageType, is alsodescribed.The NodeProfileType uses the MemoryType for the RAM and Storage elements and theCPUProfileType for CPU. The first has two attributes indicating the amounts of memory and thesecond has three attributes indicating the following aspects: CPU frequency, number of cores, andCPU architecture. The LinkProfileType has only three intuitive attributes: ID for identification ofthe profile, the Rate for reserved rate, and the Delay for the maximum delay.The CoverageType is intended to inform the geographical areas that the provider covers.Thus, this type is just a sequence of Location identified by three attributes: Country, State, andCity. It is important to notice that there is a Location element in NodeParametersType (alreadyexplained) used to geo-locate nodes in the infrastructure. With these two elements (Location andCoverage) a provider is able to identify the geographical location of its resources, which allows theprovider to offer location-aware services.Figure 25 Type describing the service offered by the providerRequest DescriptionDevelopers describe their application requirements through a request document, whose rootelement is the RequestType (Figure 26). Such type is composed by three attributes: an ID, an
  • 65. 52Owner, and a Tolerance (delay value in milliseconds), which expresses how far the virtual nodescan be placed from their required location specified on the Node element.The NodeSpecType and LinkSpecType have an attribute to indicate their ID and an attributeto indicate the ID of the corresponding of the respective profile (described in the ServiceType)chosen by the developer. The NodeSpecType has also a Location element indicating where thenode must be positioned, which is defined using the LocationType. The LinkSpecType hasseveral Node elements indicating the ID of the requested nodes that the link will be connecting.Figure 26 Type describing the requirements that can be requested by a developer5.1.2 A CloudML usage exampleThis section considers an example of the CloudML usage. Through CloudML, the provider is ableto list the different profiles using the service description model, informing nodes and linksconfigurations. A developer should inform its requirements using the requirements descriptiondocument. Moreover, Nubilum uses the description and status documents to describe physical andvirtual resources with regard to the D-Cloud. These can be used for internal communicationbetween equipments and components of the system and for external communication with theprovider.Next, these XML documents are described in more details. Please notice that some irrelevantparts of the XML documents were omitted for a better visualization.Services XMLThe XML document shown in Figure 27 represents the service defined by the D-Cloud provider.There are two node profiles (nodeprofile01 and nodeprofile02), two link profiles(linkprofile01 and linkprofile02), and three Coverage items with Country, State, and Cityspecifications.The nodeprofile01 is a node running the Linux operating system with 2 GB of RAM, 80GB of storage and acts as a server. The linkprofile01 represents a link with maximum delay and
  • 66. 53capacity equal to 0.150 ms and 1.5 Mbps, respectively. The linkprofile02 is similar to that oflinkprofile01. The Coverage is defined as a set of Country, State, and City elements. In this case, theCloud provider informs that developers can request resources located at three different localities:“Brazil, São Paulo, Campinas”, “Brazil, Pernambuco, Recife”, or “Brazil, Ceará, Fortaleza”.Figure 27 Example of a typical Service description XMLThe nodeprofile02 is from a different type of the nodeprofile01. This node profile is arouter as is indicated in the Functionality tag. In this Thesis, such type of node indicates anabstract entity used to create virtual links for a specific location without use of a more robust virtualnode. Thus, a developer can request the use of this node to ask for a link with guarantees to attend aspecific geographical region.Request XMLThis example considers a developer making a simple request for two nodes and one link spanningbetween them. To compose this document, the developer should read the Service descriptionoffered by the provider, select the correspondent profile, and then make its request. The XMLdocument in Figure 28 represents such simple request. The node01 is from nodeprofile01 and islocated at the state of “São Paulo”. The node02 is at “Recife” and it is a router. The link islinkprofile01.Figure 28 Example of a Request XML
  • 67. 54Description XMLThe description document represents the infrastructure of the D-Cloud, including all physical andvirtual nodes. Depending on the size of the D-Cloud, this document can be very long. Thus, for abetter visualization, the next examples will illustrate only some parts of CloudML: the physicalinfrastructure, the virtual infrastructure, and the virtual links.The XML document at Figure 29 presents a complete description and status of all physicalnodes and physical links, with the first <PhyNode> tag informing resource characteristics (likeCPU, RAM…) for node 100. The node 101 description was omitted, since it is similar.Figure 29 Physical infrastructure description
  • 68. 55The <VirNodeID> tag informs the IDs of the virtual nodes that are running at the specificphysical node. In this case, according to our example, only the virtual node node01 is running atphysical node 100. There are also two physical links (<PhyLink> tags). The physical linkphylink01 has virtual link virlink01 associated to it. Further information about this link wasomitted here and will be described in the next Figure.Figure 30 shows the description and the status of all the virtual nodes and virtual links of aspecific owner in the D-Cloud. Particularly, this example shows how the virtual network allocatedresources after receiving the request in Figure 28. Please note that this description is very similar tothe physical infrastructure. The virtual node node01 has many characteristics, such as RAM, CPU,storage, network interface, and virtual environment. In this case, as the two virtual nodes thatresulted from the same type, the virtual node node02, omitted in the document, is a simple virtualnode with the Router functionality.Figure 30 Virtual infrastructure description
  • 69. 56Furthermore, this example is also about the description and the status of all virtual linksestablished in the D-Cloud. The virtual link virlink01 has information, such as technology andrate, described in the <LinkParams> tag. Note that virlink01 was referenced previously in thephysical infrastructure description as a link associated to a physical one.5.1.3 Comparison and discussionSome alternatives for resource and requirements description in D-Clouds were presented at Section3.2.1. In this section, CloudML characteristics will be contrasted, and a comparison will be done inorder to discuss these languages, highlighting their advantages and weaknesses.CloudML presented in this work was developed to be a vendor-neutral language forresource/request description on D-Clouds. Such neutrality is obtained both internally and externallyto the D-Cloud.First, in the internal viewpoint, CloudML brings a common language that can be implementedby different Cloud equipment vendors in order to integrate their different solutions. Certainly, suchintegration cannot be made without some common protocol implemented by the vendors, butCloudML offers a common terrain for data representation that is a crucial step towardsinteroperability. Moreover, CloudML supports vendors’ innovation offering flexibility through theuse of the OtherParams element in the description of virtual and physical nodes and links. Suchoptional field can be used by different vendors to convey private information in order to tuneequipments of the same vendor in the infrastructure. This characteristic is similar to OpenStack.In the second and external viewpoint, the supported neutrality allows developers to requestservices from different D-Cloud providers in order to compare characteristics from each one andchoose the appropriated services for their tasks. Here, it is important to notice that these providersshould use some standardized interface, such as OCCI, to handle this information model.All the languages covered in Section 3.2.1 describe, in some way, computational and networkresources in the Cloud. Service description is also a common place for description languages.However, these services are described in different manners. For example, the CloudML uses profilesto represent distinct types of nodes and links that compose services; the VXDL is itself arepresentational way to describe Grid applications; the OpenStack uses flavors idea, but it isrestricted to computational resources. Request description is not treated by the VRD.One interesting aspect of CloudML is that of geo-location. With this information, the Cloudmay offer services with location-awareness. This point is also covered by the VXDL, VRD, andManifest languages, but this aspect is described without details in the respective works.
  • 70. 57In addition to these points, the main CloudML characteristic is the description integration.With CloudML, different Cloud providers may easily describe their resources and services and makethem available to developers. Thus, developers may search for the more suitable Cloud to submittheir requests to.5.2 Communication interfaces and protocolsFigure 31 highlights the communication protocols used by Nubilum’s components, which were citedat Chapter 4. Note that the figure omits the Storage System since it is controlled through the sameinterface available at Workers. Basically, two different protocols are used for communication: theHTTP protocol employed by REST interfaces available in the Allocator, Manager, and Workercomponents, and the Openflow protocol for communication with network devices. Together thoseprotocols cope with the integrated control and monitoring of all physical and virtual resources in theD-Cloud. The HTTP protocol is also used by the D-Cloud provider to describe the supportedservices and by the developers to submit their requests to the system.Figure 31 Communication protocols employed in NubilumThe next sections will detail the communication protocols employed in Nubilum. First,Section 5.2.1, will show the REST interfaces of each component: Allocator, Manager, and Worker.After that, Section 5.2.2 will discuss how Openflow is explored by the Nubilum to setup virtual linksin the physical network.5.2.1 REST InterfacesAs showed in Section 5.1, the XML Schemas defining CLoudML were divided into three groups:schemas for resource description, schemas for service description, and schemas for requirementsdescription. Here, these schemas are mapped in seven MIMEtypes (Table II) that describe thespecific data types (xml documents) to be used by the REST interfaces of Nubilum.NUBILUMAllocator ManagerWorkerNetworkElementsD-CloudProviderApplicationDeveloperHTTPHTTPHTTPHTTP Openflow
  • 71. 58Table II MIMEtypes used in the overall communicationsMimetype Descriptioncloudml/nodesstatus+xml Status of physical and virtual nodescloudml/nodesdescription+xml Description of physical and virtual nodescloudml/virnodedescription+xml Description of a single virtual nodecloudml/infradescription+xml Descriptin of the entire D-Cloudcloudml/virinfradescription+xml Description of a particular virtual infrastructurecloudml/servicesdescription+xml Description of the service defined by the Providercloudml/appdescription+xml Description of a developer’s request for their applicationThe cloudml/nodesstatus+xml is used to exchange volatile status information betweenentities, and it refers to a short description of the status of all nodes managed by a worker, i.e., theserver (referred to as a physical node) and the virtual machines (or virtual nodes). The otherMIMEtypes offer complete descriptions of the resources. The cloudml/nodesdescription+xml(corresponding to the NodesDescription XML Schema) gives a long description of the physicalnode and virtual nodes and the cloudml/virnodedescription+xml (corresponding to theVirNodesDescription XML Schema) refers to the complete description of a specific virtual nodehosted on the server. The type cloudml/infradescription+xml refers to the entire D-cloud andit shows all the servers, physical links, virtual machines, and virtual links hosted on the D-Cloud,whereas the cloudml/virinfradescription+xml refers to the complete description of theresources in a virtual infrastructure. The cloudml/servicedescription+xml and thecloudml/appdescription+xml describe the service offered by the provider and the developer’srequests, respectively.The REST interfaces were defined according to five URL resources: the virnode resource isused to describe operations available at virtual machines; the worker resource is used for registeringand unregistering Workers at the Manager; the dcloud resource is the representation of the entireD-Cloud infrastructure; the services resource is used to configure the profiles of virtual machinesand virtual links that can be requested by developers; and the app resource is used to representoperations for requesting, updating, and releasing resources of a particular application. The nextsections offer more details about those resources.AllocatorThe interface available at the Allocator component can be accessed by developers and the provider.The developers make use of operations to submit their requests to Nubilum whereas the providercan configure the services offered by Nubilum. As stated earlier, through the services resource theprovider can configure the profiles of virtual machines and virtual links that can be requested by
  • 72. 59developers. The operations are used to retrieve and update the list of services offered by the D-Cloud provider, as showed in Figure 32 and Figure 33. The operation for updating the offeredservices uses a PUT method on the URL “/services” and it carries into the body the new parametersof the service in an XML document following the cloudml/servicesdescription+xmlMIMEtype. Developers use the GET operation to request the description of the current servicessupported by the provider.URL http://allocator_ip/services/Method GETReturns200 OK & XML (cloudml/servicesdescription+xml)401 Unauthorized404 Not FoundFigure 32 REST operation for the retrieval of service informationURL http://allocator_ip/servicesMethod PUTRequest Body XML (cloudml/servicesdescription+xml)Returns201 Created & Location400 Bad Request401 Unauthorized422 Unprocessable EntityFigure 33 REST operation for updating information of a serviceUsing the app resource, a developer can submit, delete, or update an application (Figure 34, Figure35, and Figure 36, respectively). Such operations use the cloudml/appdescription+xmlMIMEtype, which describes the virtual network, i.e., virtual nodes and virtual links requested by thedeveloper. Moreover, some error messages were defined also to deal with exception cases: 400 forsyntax errors in the XML document sent; 422 for any errors on the information contained in theXML, e.g., invalid identification numbers or server characteristics; and 401 to report authenticationerrors.URL http://allocator_ip/appMethod POSTRequest Body XML (cloudml/appdescription+xml)Returns201 Created & XML (cloudml/virinfradescription+xml)400 Bad Request401 Unauthorized409 Conflict422 Unprocessable EntityFigure 34 REST operation for requesting resources for a new application
  • 73. 60URL http://allocator_ip/app/[id]Method PUTRequest Body XML (cloudml/appdescription+xml)Returns201 Created & XML (cloudml/virinfradescription+xml)400 Bad Request401 Unauthorized409 Conflict422 Unprocessable EntityFigure 35 REST operation for changing resources of a previous requestURL http://allocator_ip/app/[id]Method DELETEReturns204 No Content401 Unauthorized404 Not FoundFigure 36 REST operation for releasing resources of an applicationThese operations define the external interface of Nubilum, but other standardized interfacesmay be used instead for the same purpose of external communication, such as OCNI (Open CloudNetworking Interface), an extension of OCCI proposed by [60] to cover network requirements.ManagerThe Manager’s interface has five operations that manipulate two resources. The worker resource isused by Workers for registering and unregistering at the Manager (Figure 37 and Figure 38,respectively). The operation for registration uses a HTTP POST method whose body sectioncontains a XML document corresponding to an entire description of the server and the virtualmachines. The second operation over the worker resource is used by Workers to unregister from theManager. In this operation, the Worker must inform the URL previously sent by the Manager in thePOST operation, which includes an ID for this Worker. The third operation (Figure 39) isemployed by Workers to update the Manager with information about the server and their virtualmachines.URL http://manager_ip/workerMethod POSTRequest Body XML (cloudml/nodesdescription+xml)Returns201 Created & Location400 Bad Request401 Unauthorized422 Unprocessable EntityFigure 37 REST operation for registering a new Worker
  • 74. 61URL http://manager_ip/worker/[id]Method DELETEReturns204 No Content401 Unauthorized404 Not FoundFigure 38 REST operation to unregister a WorkerURL http://manager_ip/worker/[id]Method PUTRequest Body XML (cloudml/nodesstatus+xml)Returns201 Created & Location400 Bad Request401 Unauthorized404 Not Found422 Unprocessable EntityFigure 39 REST operation for update information of a WorkerThe other two operations available at the Manager’s interface operate over the overall dcloudresource, which is a representation of the entire D-Cloud infrastructure (Figure 40 and Figure 41).A first operation is intended to retrieve the complete description of the D-Cloud whereas the otherone is used to submit a new description of the D-Cloud, which will then be enforced onto thephysical resources by the Manager. Both operations are used by the Allocator. Please note that theseoperations use the cloudml/infradescription+xml MIMEtype.URL http://manager_ip/dcloudMethod GETReturns200 OK & XML (cloudml/infradescription+xml)401 Unauthorized404 Not FoundFigure 40 REST operation for retrieving a description of the D-Cloud infrastructureURL http://manager_ip/dcloudMethod PUTRequest Body XML (cloudml/infradescription+xml)Returns200 OK400 Bad Request401 Unauthorized422 Unprocessable EntityFigure 41 REST operation for updating the description of a D-Cloud infrastructure
  • 75. 62WorkerThe Worker interface is focused on operations on virtual machines. Through the virnode resources,the interface of the Workers’ components offers operations for the creation, updating, and removalof virtual machines, which are respectively showed at Figure 42, Figure 43, and Figure 44.URL http://worker_ip/virnodeMethod POSTRequest Body XML (cloudml/virnodedescription+xml)Returns201 Created & XML (cloudml/virnodedescription+xml)400 Bad Request401 Unauthorized422 Unprocessable EntityFigure 42 REST operation for the creation of a virtual nodeThe operation for the creation of a virtual node carries the node parameters in an XML documentfollowing the cloudml/virnodedescription+xml MIMEtype. If executed with success, the operationreturns a 201 message and a XML document containing the parameters of the allocated virtualmachine. This document is very similar to the one that was passed in the request body but withsome additional information like the current state of the virtual machine. The other two operationsare similar to the POST operation, except that they access the new virtual node through a differentURI.URL http://worker_ip/virnode/[id]Method PUTRequest Body XML (cloudml/virnodedescription+xml)Returns201 Created & Location400 Bad Request404 Not Found401 Unauthorized422 Unprocessable EntityFigure 43 REST operation for updating a virtual nodeURL http://worker_ip/virnode/[id]Method DELETEReturns204 No Content401 Unauthorized404 Not FoundFigure 44 REST operation for removal of a virtual node
  • 76. 635.2.2 Network Virtualization with OpenflowNubilum uses the Openflow protocol for controlling and collecting information of the physical andvirtual links. As discussed in Section 4.3.2, the Manager’s modules coping with these functions areimplemented in a NOX Openflow controller through a specific application, which has also a set ofREST-based interfaces for communication with other Manager’s modules. The next sections willdescribe in details the introduced REST interfaces and will specify the process to setup the correctflows in each network device when creating virtual links.NOX REST InterfacesThe REST interface in NOX has only two resources: topo, to inform the topology of the physicalnetwork; and vlink that allows the manipulation of virtual links.The topo resource is subject to a single GET operation (see Figure 45) used by the ResourceDiscoverer module to request the topology of the physical network, which is a result of the LLDP-based discovery processes detailed in Section 4.4.2. The operation is detailed in Figure 45. Note thatwhen the operation proceeds correctly, a XML document from thecloudml/infradescription+xml MIMEtype is sent to the requester, but this descriptioncontains only reduced information about the physical resources and the links between them.URL http://nox_ip/topoMethod GETReturns200 OK & XML (cloudml/infradescription+xml)404 Not FoundFigure 45 REST operation for requesting the discovered physical topologyOver the vlink resource, three operations were defined, namely, POST (Figure 46), PUT(Figure 47), and DELETE (Figure 48), for the creation, updating, and deleting of virtual links,respectively. A new MIMEtype is also introduced that contains information describing the virtuallink: the <MAC,IP> for source and destination nodes of the virtual link, and the path of physicalnodes/links that will host this virtual link. This path is informed since the calculus of the path is nota NOX’s task but is rather a task performed by the Allocator component.URL http://nox_ip/vlinkMethod POSTRequest Body XML (cloudml/virlinkdescription+xml)Returns201 Created & XML (cloudml/virlinkdescription+xml)400 Bad Request422 Unprocessable EntityFigure 46 REST operation for the creation of a virtual link
  • 77. 64URL http://nox_ip/vlink/[id]Method PUTRequest Body XML (cloudml/virlinkdescription+xml)Returns201 Created & Location400 Bad Request404 Not Found422 Unprocessable EntityFigure 47 REST operation for updating a virtual linkURL http://nox_ip/vlink/[id]Method DELETEReturns204 No Content404 Not FoundFigure 48 REST operation for removal of a virtual linkVirtual links setupThe Openflow protocol leverages the usage of flow-based network devices that are configured byNOX. Thus, when the Manager calls NOX through a POST operation in the vlink resource, theOpenflow Controller and Collector module will send appropriated Openflow messages to eachnetwork device.In our implementation, the entire D-Cloud network is an Ethernet network and each networkdevice is a switch. Thus, the first aspect to be considered is the forwarding of ARP packets whichmust be flooded in the network without causing the retransmission of multiple copies of eachpacket. This problem can be solved by creating a spanning tree in the graph in order to allow thecommunication of all servers in the D-Cloud. The spanning tree can be created using the SpanningTree Protocol (STP) which is an optional protocol that Openflow switches can implement. The useof STP is documented in the Openflow specification document [52]. In order to support switchesthat do not perform STP, the next procedure is used in Nubilum.Considering the physical network formed by servers, switches, and their respective links, abreadth-first search was implemented in order to find the edges of the graph pertaining to thespanning tree. Based on this, each network device is configured with a number of flows equal to thenumber of ports on the switch. For each port, the module checks if the link associated with theswitch port is part of the spanning tree, if it does then all incoming ARP packets will be flooded tothe other ports except the port from where the packet arrived. If the link associated with the port isout of the spanning tree, the ARP packets arriving from that port will be dropped. Figure 49 showsan example of the typical Openflow tuple of these flows. In the example, all ARP packets incomingfrom switch port 1 will be flooded to the other ports except port 1.
  • 78. 65Figure 49 Example of a typical rule for ARP forwardingIn Nubilum, virtual links are implemented by a set of flows configured in the network devicesthat form a path between the two virtual nodes in the borders of this virtual link. This path iscalculated based on the developer’s requirements and the specific goals of the allocation algorithmsimplemented in the Mapper module at the Allocator component and no optimization is done inNOX. Thus, the creation of virtual links is based on the XML document received. As explained inthe previous section, each request indicates the 2-uple <MAC,IP> for the source and destinationnodes of the virtual link as well as the path (a list of network devices).Each network device in the path will be configured with two specific flows for bidirectionalcommunication. The flows have the characteristics presented in Figure 50. Note that these flowsinclude information about the MAC and IP addresses of the virtual machines connected by thevirtual link in order to restrict the virtual link to these machines.(a)(b)Figure 50 Example of the typical rules created for virtual links: (a) direct, (b) reverseIn Section 5.1.2, the CloudML example introduced the use of virtual routers for the creationof virtual links towards certain geographical locations. A link with a virtual router receives a differenttreatment in Nubilum since it is intended to route data from a virtual node to clients and vice versa.Thus, NOX creates two flows similar to the ones presented in Figure 50, but containing only theMAC and IP addresses of the virtual node. Thus, traffic incoming from clients in the geographicalregion can be routed to the virtual node.5.3 Control Plane EvaluationAs discussed in this Chapter, the Nubilum’s processes generate several control messages formonitoring and allocating resources, which naturally introduce load in the network. Thus, twoSwitchPortMACsrcMACdstEthtypeVLANIDIPsrcIPdstTCPsportTCPdportAction1 * * ARP * * * * * FLOODSwitchPortMACsrcMACdstEthtypeVLANIDIPsrcIPdstTCPsportTCPdportAction1 1:1:1:1:1:1 2:2:2:2:2:2 IP * 1.1.1.1 2.2.2.2 * * Outputto port2SwitchPortMACsrcMACdstEthtypeVLANIDIPsrcIPdstTCPsportTCPdportAction2 2:2:2:2:2:2 1:1:1:1:1:1 IP * 2.2.2.2 1.1.1.1 * * Outputto port1
  • 79. 66questions can be posed: “What is the impact of these control messages onto the system?”, and “howdoes this scale up with the number of Workers and network elements?”.Considering both questions, this Section shows the evaluation of the control load (totalmessage size in bytes) generated by the system through some simple and useful models derived frommeasurements made on a prototype of the system. The prototype of Nubilum was implemented in atestbed. This prototype comprehends the overall components of the system: Allocator, Manager,Worker, Network Device, and Storage System. All the communication interfaces that make theNubilum’s control plane were also implemented as described in the previous sections.After implementation, some measurements were taken in this prototype in order to measurethe number and size of HTTP and Openflow messages generated by three distinct events: theresource allocation for a new request to the system; the status update of the physical resources(Workers and Network Devices); and the release of resources of a developer’s application. Pleasenote that this section does not evaluate the algorithms for resource allocation – which will beevaluated through simulations in Chapter 6 – it only measures the control load introduced byNubilum in order to evaluate the impact that the system cause in the network due to thecommunication interfaces.The messages exchanged between the components in Nubilum were measured in an actualprototype of the system. One can divide these control messages into two major groups: HTTPmessages – used for communication with the developer, allocator, manager, and worker – andOpenflow messages – used for communication with the network devices. Furthermore, thesemessages can be divided into three sub-groups according to their associated events: messages forapplication allocation, messages for application release, and messages for status update.Table III shows the number of bytes generated by the control messages between eachcomponent of the system for each type of event. Each two lines in the table represent one interfaceof the system: one between the Developer and the Allocator, another between the Allocator and theManager, others between the Manager and each Worker, and a last one between the Manager andeach Network Device.The size of the messages depends on the specific parameters: VN = number of virtual nodes,PN = number of physical nodes, VL = number of virtual links, PL = number of physical links, IF =infrastructure description (in bytes, given by IF = 300+734*PN+857*VN+389*PL+314*VL), P =number of ports in the network device. Thus, the length of an allocation message sent fromDeveloper to Allocator (first line in the table), for example, is proportional to the number of virtualnodes (VN) and to the number of virtual links (VL) required by the Developer. Furthermore, it is
  • 80. 67important to note that for each message there is a fixed number (in bytes) that represents the HTTPheader length, a fixed XML part, and the parameters’ description length. For example, with regard tothe message between the Developer and the Allocator, one can note that 505 bytes is the length ofthe HTTP header and the fixed size XML part, 84 is the length of one virtual node description, and74 is the length of one virtual link description. If there are 10 VN and 5 VL, the total length of anallocation message submitted by a Developer to an Allocator is 1715 bytes.Table III Models for the length of messages exchanged in the system in bytesInterface Allocation event Release event Update event TypeDeveloper Allocator505+84*VN+74*VL(GET)161(DELETE)N/A HTTPAllocator Developer537+857*VN+314*VL(Reply GET)46(Reply DELETE)N/A HTTPAllocator Manager120 (GET)221+IF (PUT)221+IF(PUT)N/A HTTPManager Allocator237+IF (Reply GET)242+IF (Reply PUT)242+IF(Reply PUT)N/A HTTPManager Worker 978 (POST)169(DELETE)639+180*VN(PUT)HTTPWorker Manager 1024 (Reply POST)46(Reply DELETE)130(Reply PUT)HTTPManager Network Device 320 288 20 OpenflowNetwork Device Manager N/A 352 12+104*P OpenflowExamining Table III, one can note that the HTTP messages are bigger than the messagesgenerated by the Openflow protocol. This is expected since the HTTP messages carry CloudMLdocuments whose text format introduces flexibility but tends to be bigger than a byte-orientedprotocol. Another interesting aspect that can be highlighted is that the messages between Allocatorand Manager are bigger than other HTTP messages. This is due to the large amount of data thatneeds to be exchanged between these two components because of the way the system was designed.This needs to be done in order to allow the separation between the component that allocates newrequests and the one that gathers information about the whole system. Another reason for thisdesign choice is to allow the allocator to be stateless in respect to the status of the resources. Thuseach time a new request arrives the Allocator must acquire the current status of the D-Cloud. Noticealso that part of this infrastructural information that needs to be exchanged provides valuablefeedback about the developers’ applications, as the allocator acts as an interface between developersand the whole system.The table also reveals that the control load increases linearly with the size of the physicalinfrastructure (number of Workers and Network Devices) as well as the size of the requests. This isan important validation, as it shows that there is no unusual increase of the control traffic in thesystem according to the number of Workers, which is an aspect that affects system’s scalability.
  • 81. 686 Resource Allocation Strategies“Fugiunt amici a male gerente res suas.”Schottus, AdagiaSection 3.2 detailed four research challenges related to resource management on D-Clouds:Resource Modeling, Resource Offering and Treatment, Resource Discovery and Monitoring, andResource Selection and Optimization. The first three research challenges were discussed in thecontext of Chapter 4 and Chapter 5. In a complementary form, this chapter will discuss the fourthchallenge about resource allocation.The problem of finding the best mapping of distributed applications onto a physical networkis not a trivial task due to the dynamicity, high algorithm complexity, and all the differentrequirements (from developers and the provider) that must be contemplated by Nubilum. Moreover,this problem is not restricted to the allocation of an incoming application, but it also involves thecontinuous monitoring and maintenance of the application in order to guarantee developer’srequirements. This chapter focuses on problems for resource allocation on D-Clouds. The firstproblem investigated (in Section 6.1) is relative to the allocation of resources for the managemententities of the Cloud. Specifically, the Manager Positioning Problem will be addressed and solved.After that, Section 6.2 discusses the way Nubilum deals with the problem of allocating virtualnetworks and evaluates an algorithm for load balance. Section 6.3 presents algorithms for thecreation of a virtual network when the request is composed only by the virtual nodes. Finally,Section 6.4 discusses and summarizes the main results obtained in this chapter.6.1 Manager Positioning ProblemRecall, from Chapter 4, that Nubilum is composed by five components: Allocator, Manager,Workers, Network Devices, and the Storage System. As described earlier, the Manager is at thecenter of the entire system and maintains continuous communication with all the other components.This implicates that the relative position of the Manager in the network can influence theperformance of the entire system, since choosing a bad one will certainly increase communicationcosts with part of the network and it may also cause delayed responses to urgent system’s
  • 82. 69operational queries hence leading to performance degradation events. The problem of finding anoptimal position for the Manager entity is referred here as the Manager Positioning Problem (MPP).Let us initiate by supposing without loss of generality that the communication between theManager and the Allocator is of minor importance in comparison to the communication with themany Workers and Network Devices. This can be achieved, for example, if the two components areexecuted in the same physical node. Assume, finally, that the D-Cloud is composed by nodes able towork as Network Devices and Workers at the same time. This assumption only turns the MPP moregeneral, since each node in the D-Cloud is capable of hosting the Manager.Figure 51 shows an example of a D-Cloud containing ten Worker/Network Device nodes(represented by W nodes) and the Manager (the overlaid M node) already allocated in one of thesenodes. Furthermore, each link has an associated delay that is showed on the figure as annotatedvalues over the links. Thus, ‫ܦ‬௜,௝ is defined as the shortest path between nodes i and j considering thedelays as links’ weights. Note that this value is defined to be zero when ݅ ൌ ݆ and is positiveotherwise.Figure 51 Example of a D-Cloud with ten workers and one ManagerThe MPP can be defined to determine the position of the Manager in the D-Cloud in order tominimize the cost function ∑ ‫ܦ‬௜,ெ௜‫א‬ே , where N is the set of nodes of the graph and M is the nodewhere Manager is hosted. Please note that this problem is equivalent to solve the 1-median problem,which is seen as a very simple form of the replica problems introduced in Section 3.2.4. Thisproblem can be solved calculating the all-pairs shortest path in a weighted graph which can beobtained with the Floyd-Warshall algorithm whose complexity is ܱሺܸଷሻ [41], where ܸ is the numberof nodes. Using the distance from each node to each other, the sum of the distances can becalculated for each node in the graph that is a candidate for positioning the Manager. The node thathas the minimum sum of distances is the solution to the MPP. The next example will detail as thissimple algorithm can be used in a particular instance of the MPP.WW W WW WWWWWM0.10.50.20.010.10.2 0.30.050.2
  • 83. 70Considering the example in Figure 51 and using one of the algorithms for all-pairs shortestpath computation, one can obtain the distance matrix below (nodes are numbered from left to rightand from top to bottom):‫ۏ‬‫ێ‬‫ێ‬‫ێ‬‫ێ‬‫ێ‬‫ێ‬‫ێ‬‫ێ‬‫ۍ‬0 0.3 0.1 0.6 0.11 0.21 0.31 0.36 0.61 0.810.3 0 0.2 0.7 0.21 0.31 0.41 0.46 0.71 0.910.1 0.2 0 0.5 0.01 0.11 0.21 0.26 0.51 0.710.6 0.7 0.5 0 0.51 0.61 0.71 0.76 1.01 1.210.11 0.21 0.01 0.51 0 0.1 0.2 0.25 0.5 0.70.21 0.31 0.11 0.61 0.1 0 0.3 0.35 0.6 0.80.31 0.41 0.21 0.71 0.2 0.3 0 0.05 0.3 0.50.36 0.46 0.26 0.76 0.25 0.35 0.05 0 0.35 0.550.61 0.71 0.51 1.01 0.5 0.6 0.3 0.35 0 0.20.81 0.91 0.71 1.21 0.7 0.8 0.5 0.55 0.2 0 ‫ے‬‫ۑ‬‫ۑ‬‫ۑ‬‫ۑ‬‫ۑ‬‫ۑ‬‫ۑ‬‫ۑ‬‫ې‬.Summing up the values in the columns, each element of this cost vector will give the cost ofthe function considering that M is hosted in the respective node and the minimal value indicates thebest node to allocate the Manager. For the matrix above, the cost vector isሾ3.41 4.21 2.61 6.61 2.59 3.39 2.99 3.39 4.79 6.39ሿand the minimal cost is 2.59 corresponding to the fifth node where the Manager is already allocatedas showed at Figure 51.Please note that, the problem can be simplified considering a restricted number of nodes asWorkers. In this case, the same solution can be applied by simply restricting the path computationsto the Worker nodes and using links and intermediary nodes as the inputs for the calculus. Also, thesame idea can be applied to reduce the number of path computations considering only Workers thathave at least one virtual machine already allocated, since the status of these nodes must bemaintained more accurately than that of servers that have no running VMs.One may note that if using static delays values, the path matrix can be computed only once,saving execution time. However, a more practical approach of the problem would be to measure thedelay along the time according to the traffic in the D-Cloud. In this case, the same solution could beused repeatedly while considering a delay sample in time (or any function over this value, includinghere functions over historic values) as the input for the new path matrix computation.6.2 Virtual Network AllocationIn Nubilum, developers submit their requirements to the system as a virtual network, composed ofvirtual nodes and virtual links. A priori, Nubilum’s resource model (condensed in the CloudML)supports a wide range of virtual network algorithms while considering diverse characteristics as geo-locality, network delay, Worker capacity, network capacity, and so on. Such characteristics are used
  • 84. 71as inputs of the resource allocation algorithms and they can figure as constraints of the problem oras part of the objective function. Note that Nubilum neither restricts the objective functions thatcan be used by the allocation algorithms nor constrains the type of the algorithm (please see Section0 for discussion on the topic of Network Virtualization). However, Nubilum confines thealgorithms to work with the characteristics of their resource model that are summarized in TableIV.Table IV Characteristics present in Nubilum’s resource modelComponents CharacteristicsPhysicalNetworkPhysicalNodesMemory, Storage, CPUGeo-locationSupported virtualization environmentFunctionalityPhysical network interfacesAlready allocated virtual machinesPhysicalLinksCapacity and allocated capacityCurrent delayCurrent bit error rateTechnologyAlready allocated virtual linksRequestedVirtual NetworkVirtualNodesMemory, Storage, CPUGeo-location and tolerance delayOS typeFunctionalityVirtual LinksRequired rateMaximum delayAllocated VirtualNetworkVirtualNodesMemory, Storage, CPUGeo-locationFunctionalityVM stateOwnerVirtual LinksCapacity and current rateCurrent delayCurrent maximum bit error rateOwnerThe characteristics in the table allow that several resource allocation algorithms for virtualnetworks can be implemented in the Mapper module (see Section 4.3.1) with just minormodifications. For example, the algorithms D-ViNE and R-ViNE proposed in [10] consider aspectsas node capacity, node location, maximum tolerance distance, and link capacity for the physical andthe requested virtual networks. The algorithms consider those aspects in a abstract way, butassociating the abstract node capacity with a concrete characteristic as Memory, Storage, and/orCPU and considering the distance measured in delay, these algorithms could be implementedwithout any additional modifications. Similarly other proposals cited in [31] and [3] could be adaptedfor use in Nubilum.
  • 85. 72This section presents a resource allocation algorithm that works with subset of thecharacteristics supported by Nubilum. This algorithm has the objective of guarantying the loadbalancing of the physical resources (servers and links). The problem is to allocate an incomingvirtual network in the physical network in order to balance the load of virtual resources allocated inphysical resources, subject to some constraints, i.e., this problem can be better defined as one thatallocates the virtual network in order to minimize the sum of two components: the usage ofresources at physical nodes and the maximum number of virtual links in a given physical link.The problem considers restrictions associated to virtual nodes and virtual links. In case ofvirtual nodes memory, storage, CPU, geo-location, and functionality constraints are considered,whereas in the case of virtual links the problem considers only a required maximum delay of thephysical path. Table V shows these set of characteristics. Note that, differently from otherapproaches, the present problem does not consider link capacity since it seems to provide betterinsights and more useful results with unrestricted capacity in a pay-as-you-go business plan.Table V Reduced set of characteristics considered by the proposed allocation algorithmsComponents CharacteristicsPhysicalNetworkPhysicalNodesMemory, Storage, CPUGeo-locationFunctionalityAlready allocated virtual machinesPhysicalLinksCurrent delayAlready allocated virtual linksVirtual NetworkRequestVirtualNodesMemory, Storage, CPUGeo-locationFunctionalityVirtual Links Maximum delayFollowing, Section 6.2.1 defines the problem stated above in a formal way. In order to solvethe problem, the solution is divided into two steps: virtual node allocation, which is covered inSection 6.2.2, and the allocation of virtual links, discussed in Section 6.2.3. The proposed algorithmsare evaluated in Section 6.2.4.6.2.1 Problem definition and modelingThe physical or substrate network is represented by an undirected graph GSൌ ሺNS, ESሻ. Eachsubstrate node vS ‫א‬ NShas three distinct capacities: cሺvSሻ, mሺvSሻand sሺvSሻ, representing CPU,memory, and storage remaining capacities on each substrate node. Each substrate link has a delayassociated represented by the function dS: ES՜ R. The current stress sሺeSሻ of a substrate linkeS ‫א‬ ESis the number of virtual links that already pass through this substrate link.Each virtual network will be given as a graph GVൌ ሺNV, EVሻ. Each virtual node vV ‫א‬ NVhasthree distinct capacity requirements: cሺvVሻ, mሺvVሻand sሺvVሻ, representing CPU, memory and
  • 86. 73storage consumption. For each virtual link e୴ ‫א‬ EV, a maximum delay given by a function dV: EV՜R is associated. Each virtual node will be assigned to a substrate node. This assignment can be seenas a function MN: NV՜ NS. For each virtual node v ‫א‬ NV, the set NSሺvሻ ‫ؿ‬ NSindicates thephysical nodes where this virtual node can be allocated.Each virtual link will be assigned to a substrate simple path between the correspondingsubstrate nodes that host the virtual nodes at both ends of that virtual link. Being PSthe set ofsimple paths of GSand PSሺu, wሻ the set of simple paths of GSbetween physical nodes u and w, thisassignment can be seen as a function ME: EV՜ PSsuch that if e୴ ൌ ሺu, wሻ ‫א‬ EV, then MEሺe୴ሻ ‫א‬PSሺu, wሻ. If p ‫א‬ PSis any simple path on GS, then its delay dSሺpሻ is the sum of delays of eachphysical link of the path. For eୱ ‫א‬ ES, the stress Sሺeୱሻ generated by the allocation of GVis definedas the number of virtual links e୴ ‫א‬ EV that pass through the substrate link eୱ, i.e., the number ofvirtual links e୴ ‫א‬ EV such that eୱ is a substrate link of the path given by MEሺe୴ሻ.Given a substrate network GSൌ ሺNS, ESሻ and a virtual network GVൌ ሺNV, EVሻ, the problemis to find mappings MN: NV՜ NSand ME: EV՜ PSin order to minimize:maxୣS‫א‬ESሾsሺeSሻ ൅ SሺeSሻሿ ൅ max୴‫א‬NS቎ ෍ cሺuሻ െ cሺvሻMNሺ୳ሻୀ୴቏ ൅ max୴‫א‬NS቎ ෍ mሺuሻ െ mሺvሻMNሺ୳ሻୀ୴቏൅ max୴‫א‬NS቎ ෍ sሺuሻ െ sሺvሻMNሺ୳ሻୀ୴቏Subject to the constraints:‫׊‬vԖNV, MNሺvሻ ‫א‬ NSሺvሻ1)‫׊‬e ൌ ሺu, wሻԖEV, MEሺeሻ ‫א‬ PS൫MNሺuሻ, MNሺwሻ൯2)‫׊‬eԖE୴, dVሺeሻ ൑ dS൫MEሺeሻ൯ 3)‫׊‬vԖNS, cሺvሻ ൑ ෍ cMNሺ୳ሻୀ୴ሺuሻ4)‫׊‬vԖNS, mሺvሻ ൑ ෍ mMNሺ୳ሻୀ୴ሺuሻ5)‫׊‬vԖNS, sሺvሻ ൑ ෍ sMNሺ୳ሻୀ୴ሺuሻ6)The objective function aims to minimize the sum of maximum link stress and remaining CPU,memory and storage, thus balancing the load. The first constraint (1) implies that each virtual node
  • 87. 74will be allocated in a required physical node, according to geo-location restriction given. The secondconstraint (2) enforces the virtual link between virtual nodes u and w to be allocated in the pathbetween the physical nodes in which these virtual nodes are allocated. The third constraint (3)implies that the delay of the virtual link will be respected in the path in which it is allocated. The lastthree constrains (4), (5), and (6) enforce that the capacities restrictions for virtual nodes are fulfilled.6.2.2 Allocating virtual nodesThe problem of allocating virtual nodes on multiple physical nodes considering multiple constraintscan be reduced to that of the multidimensional multiple knapsacks, which is NP-complete [36].Therefore, the algorithm for virtual node allocation proposed in this Thesis uses a greedy approachas showed at Figure 52. This algorithm consists in selecting, for each virtual machine of a newrequest, the appropriate Workers according to the storage, memory and CPU capacities associatedwith the virtual nodes and also the geo-location constraints.The algorithm initiates sorting the virtual nodes (a vm) and Workers that will be processed.This sort is made in the following order: highest free memory, highest free CPU, and lastly highestfree storage space (line 4 for virtual machines and line 6 for Workers). For each virtual machine, thealgorithm tries to greedily allocate it to the Worker with most remaining capacities that satisfies itslocation restriction (modularized in a simple check in line 8). In other words, this check will get agiven virtual node and verify if this virtual node can be allocated in the current Worker being tested.If it is possible, the virtual node is allocated in this Worker (line 9) and the algorithm will try toallocate other virtual nodes; if it is not possible, the next Worker in the list of Workers will be tested.The algorithm allocates the virtual machines that need more resources first. Note that thealgorithm performs admission control, if a virtual machine of the list cannot be mapped to aWorker, then the entire request is rejected (line 13).Algorithm 1: Allocation of virtual nodes1 Inputs: vmList, workerList;2 Outputs: vmMapping;3 begin4 sortByDecreasingValueOfResources (vmList);5 for each vm in vmList do6 sortByRemainingResources (workerList);7 for each worker in workerList do8 if( tryToAllocate (vm, worker) ) then9 vmMapping.Add(vm, worker);10 update( worker );11 stop;12 end13 if vm not in vmMapping then stop;14 end15 endFigure 52 Algorithm for allocation of virtual nodes
  • 88. 756.2.3 Allocating virtual linksRegarding link allocation, the algorithm focuses on minimizing the maximum number of virtualpaths allocated over a physical link while obeying the restriction on the maximum delay. In order toachieve link load balancing when allocating on physical links, a minimax path with delay restrictionalgorithm was used to determine the paths that will be allocated.A minimax path between two Workers is the path that minimizes the maximum weight of anyof its links. The link weight is given by the number of virtual paths that is already allocated in aphysical link. So, the minimax path has the minimum maximum link weight. Figure 53 illustrates theminimax path concept considering the paths between two nodes S and T. The numbers annotated inthe links and separated by the semicolon are, in the left, the link weight for the minimax pathcalculus and, in the right, the delay for checking the restriction. In the example, the path costs of P1and P2 are 4 and 2, respectively, which are the maximum weights of the links in each path. Theminimax path is P2, but P1 is the chosen path when considering the delay restriction.Figure 53 Example illustrating the minimax pathTo determine the minimax path with delay restriction between two nodes one can execute abinary search on a vector ranging from the minimum link weight to the maximum link weight of thegraph. In each iteration of the binary search, a weight k is selected and all links with weight greaterthan k are removed from the graph. The Djikstra algorithm considering the delay as the cost is thenexecuted to verify if there is a path between the two nodes in this new graph that is constrained bythe maximum delay. If there is such a path in this graph the upper bound is adjusted to the currentk. Adversely, if there is not a path in this graph, the lower bound is adjusted to k. The binary searchis repeated until the vector is reduced to two elements. Thus, k is chosen as the element in the upperbound, and Djikstra is used again to verify the delay constraint. If no path is found for that link, itcannot be allocated in this network. This simple algorithm is showed only for illustrating a solutionfor the problem, but more efficient algorithms can be used, like the ones proposed for thebottleneck shortest path problem, which is a closely related problem [34].Similarly to the problem of virtual node allocation, the allocation of several virtual linksfocusing on minimizing the number of virtual links on each physical link is a NP-hard optimizationproblem (please see [74] to verify this aspect); this is the reason why the solution presented here is agreedy approach based on some heuristics.S T2;0.12;0.2 1;0.34;0.2P1P2CP1=max(2,4)=4CP2=max(2,1)=2CST=min(4,2)=2 P2 is the minmaxpathP1 is the minmax path with delay restriction
  • 89. 76In the virtual link allocation algorithm (Figure 54), each iteration computes a minimax pathwith delay restrictions for each virtual link not yet allocated. If one of those virtual links cannot beallocated, then the entire process is stopped (lines 7 and 8). The virtual link with the largest minimaxcost is allocated in the current iteration. The key idea behind this heuristic is that this is the mostrestrictive virtual link. The next iterations continue with this behavior, but the calculus of theminimax path considers this new allocated virtual link added to the graph. Thus, this greedyalgorithm performs link allocation considering the minimax path in order to achieve load balancingwithout transgressing the delay restriction.Algorithm 2: Allocation of virtual links1 Inputs: vmMapping, vLinkList, physicalLinksState;2 Outputs: linkMapping;3 begin4 while (vLinkList)5 for each vlink in vLinkList do6 status = minimaxPathWithDelay (vlink, vmMapping, physicalLinksState);7 if status = reject then8 stop;9 end10 selectedVLink = largestMinimaxCost;11 linkMapping.Add( allocate(selectedVLink));12 vLinkList.remove( selectedVLink );13 update( physicalLinksState );14 end15 endFigure 54 Algorithm for allocation of virtual links6.2.4 EvaluationAn event-driven simulator for the allocation virtual networks was implemented to compare theperformance results of the algorithms proposed by this Thesis (Minimax Path Algorithm – MPA)and the algorithm of Zhu and Ammar (ZAA), already described in Section 3.2.4. This specificevaluation is interesting since the performance of ZAA is a reference for virtual network allocationproblems with load balance. This fact can be seen in the results shown by Chowdhury et al. [10].Amongst the contribution of Chowdhury et al., they proposed a virtual network allocation algorithmwith load balance that proved to achieve worst results than ZAA. Moreover, a Hybrid solution thatapplies the ZAA heuristic for node allocation and the MPA strategy for link allocation will be tested.Thus, the Hybrid strategy allows comparing just the link allocation strategy of MPA against the linkallocation of ZAA, showing the impact of the minimax strategy.The simulator performs an event-driven simulation, whose main events are arrival anddeparture of virtual networks from a D-Cloud. The D-Cloud is represented as a topology composedof Workers connected by some links. The simulation model considers a simplified version of themodel introduced in the previous section, i.e., the several constraints as link delay, CPU, Memory,Storage, and geo-location were removed from this simulation model in order to evaluate the MPA in
  • 90. 77the same scenario for which ZAA was developed. Thus, the simulation model will consider only thestress, which is defined for Workers and physical links. In case of a Worker, the stress is the numberof virtual nodes that are currently allocated on that Worker, for a physical link, the stress is thenumber of virtual links that are currently allocated on that physical link.In this way, the node allocation strategy of MPA was modified to consider only the stress asthe metric for optimization, while disregarding specific aspects that were considered in the CPU,memory, storage, and geo-location restrictions. Thus, the algorithm is similar to the least-loadstrategy compared in Zhu and Ammar paper [74], with the difference that MPA’s link allocationalgorithm uses the minimax path while the least-load studied in their paper uses the shortest path.Observe also that the link allocation algorithm was changed to consider the delay of each link as oneunit.The adopted physical network is the backbone of the RNP (Rede Nacional de Ensino e Pesquisa),a Brazilian academic ISP, which is currently composed of 28 nodes and 33 links as showed at Figure55(b)8. In this evaluation, it was also considered an old topology of the RNP, which was composedof 27 nodes and 29 links (showed at Figure 55(a)). As one can see, the current topology has 5 nodesof degree one, and the old has 17. The impact of this aspect, as will be showed in the results, is thatthe number of disjoint paths tends to be greater in the current RNP topology affecting theperformance of MPA.(a) (b)Figure 55 The (a) old and (b) current network topologies of RNP used in simulationsThe several factors varied in the simulation are showed at Table VI. Excepting the networktopology, the other factors were obtained from the paper of Zhu and Ammar, since the idea iscomparing the algorithms in the same scenario proposed for the ZAA. The virtual network requestsarrive in a Poisson process whose average rate was set to 1, 5, 9, and 13 requests per unit time ineach scenario. The lifetime of each virtual network follows an exponential distribution of meanequal to 275 time units. The evaluation includes only networks in a star topology, since the authorsof ZAA argued that their algorithm is better for such type of networks. Observe that Zhu and8 http://www.rnp.br/backbone/index.php
  • 91. 78Ammar propose also an algorithm for graph division that subdivides a general network into severalstar networks, but, since only star networks are used in this simulation, such algorithm will not beevaluated. The size of each virtual network is uniformly distributed from 5 to 15 nodes including thecenter node. Thus, the evaluation occurs in a better case for ZAA. On the other hand, theiralgorithm for adaptive reallocation is not used in this comparison, since the objective here is toevaluate the allocation only.Table VI Factors and levels used in the MPA’s evaluationFactors LevelsPhysical network topologyOld RNP topology and Current RNP topology(one Worker per PoP)Virtual network request rateExponentially distributed with rate equal 1, 5, 9, and 13virtual networks per unit timeVirtual network lifetimeExponentially distributed with mean equal to 275 unittimesVirtual network topology Star networksSize of the virtual networks Uniformly distributed between 5 and 15 nodesAs occurs in the Zhu and Ammar’s paper, the simulation is finished after 80,000 requests havebeen serviced, and to reduce the effects of the warm-up period the metrics are collected after 40,000requests have been serviced. For each simulation time, the maximum node stress, the maximum linkstress, the mean link stress, and the mean path length are observed, which are averaged across thesimulation time. The results were calculated with 95% confidence intervals, which will be showed ifnecessary.Regarding the maximum node stress (showed at Figure 56), the results show that the MPAstrategy reaches the same performance of ZAA and Hybrid solutions. Actually, such an achievementcan be observed in the ZAA’s paper, which shows that their algorithm for allocating star networksobtains a maximum node stress equivalent to the least-load algorithm. These results are similar inboth evaluated topologies.(a) (b)Figure 56 Results for the maximum node stress in the (a) old and (b) current RNP topology010203040506070800 2 4 6 8 10 12 14MaximumNodeStressArrivalRate (requestsperunit time)ZAAMPAHybrid010203040506070800 2 4 6 8 10 12 14MaximumNodeStressArrival Rate (requestsperunit time)ZAAMPAHybrid
  • 92. 79About the maximum link stress, one can observe that Hybrid solution and ZAA outperformMPA in the old topology (Figure 57(a)). The MPA is about 43% greater than the ZAA in the worstcase at Arrival Rate equal to one, in the better case (at 13 requests per simulation time) the MPA is23% greater than the ZAA maximum node stress. This occurs because the node allocation of ZAAtakes in consideration the stress of neighbor links and the path cost has the property of adaptingbetween shortest path and a least-loaded path according to the current load. Thus, initially, ZAAtends to allocate nodes in lesser stressed regions of the graph connected through shorter links. Onthe other hand, MPA tends to allocate the virtual nodes in physical nodes that not are necessarilyclose that can cause the use of longer paths, even when the network is lightly stressed.In this same scenario, the Hybrid strategy shows as the minimax path allocation strategy canoutperform the strategy adopted in ZAA. This strategy eliminates bad the effect of the nodeallocation in MPA using the node allocation heuristic of ZAA. The gain in the maximum link stressof the Hybrid strategy is about 35% and it is practically constant over the variation of the arrivalrate.(a) (b)Figure 57 Results for the maximum link stress in the (a) old and (b) current RNP topologyMPA obtain better results than ZAA in a topology with more links available for allocation, as showsFigure 57(b). In this way, it was perceived that the addition of more links in the network mitigatesthe node positioning algorithm and turns the link positioning strategies more purposeful. When thearrival rate is one request per unit time the gain of MPA over ZAA is about 4% whereas as thearrival rate increases the MPA gain goes to about 40% over the ZAA. For the Hybrid approach thegain in the maximum link stress is about 13% and it is practically constant over the variation of thearrival rate. The MPA’s better performance is due to the fact that the minimax path algorithm triesto directly minimize the maximum number of virtual links in each physical link, which brings betterresults than searching for the path whose stress is minimal, as it is on the case with the algorithm ofZhu and Ammar.01002003004005006007000 2 4 6 8 10 12 14MaximumLinkStressArrivalRate (requestsperunit time)ZAAMPAHybrid01002003004005006007000 2 4 6 8 10 12 14MaximumLinkStressArrival Rate (requestsperunit time)ZAAMPAHybrid
  • 93. 80The mean link stress is showed at Figure 58. These results show that the MPA link allocationstrategy obtains a mean link stress higher than the ZAA, which is an expected result since ZAAoptimizes the mean link stress, while MPA optimizes directly the maximum link stress.(a) (b)Figure 58 Results for the mean link stress in the (a) old and (b) current RNP topologyFigure 59 shows the mean path length considering the mean of the length of each virtual linkin the physical network, i.e., the number of physical links on which the virtual link is allocated. Asone can see, MPA and Hybrid solutions obtain, in average, one link more than ZAA. This resultoccurs because the minimax path allocation algorithm seeks alternative paths in the network leadingto an increase of this metric.(a) (b)Figure 59 Mean path length (a) old and (b) current RNP topologyAnother point that must be highlighted is that the original problem solved by MPA is toallocate the nodes considering several restrictions including the geographical ones. In this way, if amodified version of the ZAA node allocation algorithm is used in a D-Cloud scenario it couldobtain a poor performance since the virtual nodes should be allocated in their respectivegeographical regions preventing ZAA of allocating the virtual nodes in few stressed regions of thenetwork connected by a shortest distance. Differently, the MPA’s node allocation algorithm isdesigned for the D-Cloud as it tries to maintain a lower stress level inside each geographical regionof the D-Cloud.0501001502002500 2 4 6 8 10 12 14MeanLinkStressArrival Rate (requestsperunit time)ZAAMPAHybrid0501001502002500 2 4 6 8 10 12 14MeanLinkStressArrivalRate (requestsperunit time)ZAAMPAHybrid0123456780 2 4 6 8 10 12 14MeanPathLenghtArrivalRate (requestsperunit time)ZAAMPAHybrid0123456780 2 4 6 8 10 12 14MeanPathLenghtArrivalRate (requestsperunit time)ZAAMPAHybrid
  • 94. 816.3 Virtual Network CreationIn Nubilum, the Request Description XML Scheme allows a request to have virtual nodes only (seeSection 5.1.1). In this case, the submitted request is not a virtual network and finding a specific routewith a maximum delay is not necessary. Thus, the simple routing through shortest paths would besufficient. But, the D-Cloud provider could use some solution to create a virtual network forintercommunication between the virtual machines for this new request in order to obtain a betterusage of its network resources. Working with this scenario, this section investigates the problem ofcreating a virtual network when a request is submitted without any specified links.The problem consists in creating a network connecting a subset of the physical nodes (thenodes where the virtual nodes were previously allocated by the Algorithm 1) with the objective ofbalancing the load across virtual links and, secondly, minimizing the energy consumption. Figure60(a) shows an example of a given physical topology where the virtual machines are allocated innodes A, B, and E and the number in each physical link indicates the number of current virtual linkscrossing this physical link. Figure 60(b) shows the virtual network (the grey lines) created tointerconnect those nodes. The created virtual network reaches the two objectives of balancing theload in the network while reducing the number of used links, which leads to the reduction of theenergy consumption.(a) (b)Figure 60 Example creating a virtual network: (a) before the creation; (b) after the creationThis problem can be better defined as to find a minimum length (number of links) interconnectfor a subset of nodes such that the maximum weight of its links is minimal. The minimum Steinertree problem can be reduced to this one considering minimizing the length in number of links andwith all link weights with the same arbitrary value. On the other hand, if the value of the minimalmaximum weight is given, the problem then reduces to finding a minimum length Steiner tree.As can be seen, the proposed problem is NP-hard. Therefore, approximate solutions will beproposed. The reduction of the problem to a Steiner tree gives an interesting idea for anapproximate algorithm. Such algorithm consists in executing a binary search in a vector rangingfrom the current minimum link weight to the maximum link weight of the graph. During eachiteration, a weight k is selected and all links with weight greater than k are removed from the graph.An approximate minimum length Steiner tree algorithm [65] is executed in this subgraph to verify ifDEFC0A B10 0412DEFC0A B10 1422
  • 95. 82there is a tree between the nodes. If there is a Steiner tree in this graph the upper bound is adjustedto the current k. Adversely, if there is no path in this graph, the lower bound is adjusted to k. Thebinary search is repeated until the vector is reduced to two elements. Thus, k is chosen as theelement in the upper bound, and the Steiner tree approximation gives the virtual network for theproblem.This algorithm is similar to the minimax path with delay restriction algorithm describedpreviously in Section 6.2.3, since it is defined by an outer function doing binary search and an inneralgorithm to solve a specific problem (minimum delay path, in the previous algorithm, andminimum length Steiner tree, in this one). The main idea behind this combined approach is that thebinary search provides the load balance searching by minimal maximum link weight, while theSteiner tree approximation gives the minimum length virtual network that minimizes energy. Next,the algorithms used to solve the minimum length Steiner tree problem (Section 6.3.1) and anevaluation of these algorithms are shown (Section 6.3.2).6.3.1 Minimum length Steiner tree algorithmsThis section shows three algorithms to solve the minimum length Steiner tree problem in a graph.The first algorithm is a well-known approximation based on the minimum spanning tree in thedistance graph [65]. The second and third algorithms are proposed by this Thesis: one uses a greedyheuristic that searches for the better hubs in the network in order to minimize the path lengths, andthe other is an exponential algorithm that finds the minimum Steiner tree through successive tries oflink removal.Steiner Tree Approximation (STA)Basically, the STA algorithm consists in transforming a general Steiner tree problem in a metricSteiner tree problem, which is a variant of the general problem where the graph is complete and thelinks satisfy the triangle inequality (ܿ‫ݐݏ݋‬ሺ‫,ݑ‬ ‫ݒ‬ሻ ൑ ܿ‫ݐݏ݋‬ሺ‫,ݑ‬ ‫ݓ‬ሻ ൅ ܿ‫ݐݏ݋‬ሺ‫,ݒ‬ ‫ݓ‬ሻ). The transformation ismade by calculating the all-pairs shortest path in the original graph ‫ܩ‬ and generating a newconnected graph ‫ܩ‬ᇱwhose nodes are the nodes of the original graph and the links are the cost of theshortest path between the nodes in the original graph.In ‫’ܩ‬ one can find a minimum spanning tree ܶ’ that is a 2-approximation of the minimumSteiner tree in this distance graph. Given this tree, one can replace their links by the original paths toobtain a subgraph in ‫.ܩ‬ If this subgraph contains cycles, removing links will generate a 2-approximation of the minimum Steiner tree in the general graph. More details and proofs on thisprocess can be found in [65]. The complexity of the STA algorithm is ܱሺܰଷሻ, where ܰ is thenumber of physical nodes.
  • 96. 83Greedy Hub Selection (GHS)The solution of the Steiner tree contains a subset of nodes of the graph that act like hubsinterconnecting two or more nodes of the Steiner tree (the nodes C and F in Figure 60(b)). Thus,given a graph and a subset of physical nodes containing the requested virtual nodes, the objective ofthe GHS algorithm is to find the hubs of the minimum length Steiner tree interconnecting thevirtual nodes. This problem is similar to the replica placement problems (Section 0) but with thedifference that the replicas must form a virtual tree topology, which makes the problem moredifficult.The GHS algorithm initiates with a tree formed by the physical nodes where a virtual node wasallocated. One of these nodes is chosen as the root of the tree and as the first hub. Following aniterative approach, a new hub node is placed at the best location in the network – defined as the onewhich achieves the minimal number of used links (the cost) among all the possible positions. Thislocation is then fixed. The new hub is then connected to other nodes in the tree through theshortest path, but a heuristic is used to maintain the tree. The positioning of a new hub and theimmediate link rewiring reduces the cost and these processes follow while the positioning of a newhub reduces the cost.Algorithm 3: Search Procedure1 Inputs: nodeList;2 Outputs: selectedNode, best;3 best = infinite;4 for each node in nodeList(available) do5 cost = placementProcedure(node);6 if(cost < best)7 selectedNode = node;8 best = cost;9 end10 undoInsertion(node);11 endFigure 61 Search procedure used by the GHS algorithmThe selection characterizes GHS as greedy: it selects the best configuration possible for thecurrent iteration. Thus, GHS searches for a solution with a relevant degree of optimality, calculatingthe new value of the cost for each selected candidate physical node, and selecting the one thatachieves the best cost. The pseudo-code of this search procedure is presented in Figure 61. Suchprocedure simply selects any available physical node as a candidate. The variable nodeList can beindexed with available, hub, root, or requested in order to return respectively: the availablenodes; the hub nodes other than the root; the root node; or the other requested nodes. Thus,nodeList(available) returns the nodes of the physical network that are not already a hub, theroot, or a requested node. After that, the algorithm calls the placementProcedure (Figure 62)adding a new hub in this candidate and rewiring the virtual links (line 5). The cost achieved by
  • 97. 84placing this new hub is calculated and if better than others it is selected as the best candidate node(lines 7 and 8). The line 10 returns the network to the state before the hub insertion in order toprepare the network for the next candidate hub.The placement strategy (Figure 62) is a heuristic approach that links a candidate hub to a parentand children maintaining the tree structure and guaranteeing optimality. The parent is selected firstlyas the nearest hub considering the shortest path length (line 3), but if the virtual link from thecurrent parent of this nearest hub crosses the candidate (line 4) the parent of the candidate will bethe parent of the nearest hub, and the candidate will be the new parent of this hub. After that, thechildren of the new hub are selected amongst the other hub and requested nodes (line 10). A node ischosen as a child if the new hub is a better parent (nearer) than the current parent and if the node isnot an ancestor of the new hub. After placing the new hub, the new number of used links isreturned.Algorithm 4: Placement Procedure1 Inputs: candidate2 Outputs: newCost3 nearest = nearestHub(candidate);4 if(path(nearest,parent(nearest)).contains(candidate))5 parent(candidate) = parent(nearest);6 parent(nearest) = candidate;7 else8 parent(candidate) = nearest;9 end10 for each node in nodeList(hub or requested) do11 if(distance(node, parent(node)) > distance(node, candidate) and notisAncestor(node, candidate))12 parent(node) = candidate;13 end14 end15 newCost = calculateNewCost();Figure 62 Placement procedure used by the GHS algorithmIn order to clarify the proposed heuristic, let’s consider the current virtual tree in Figure 63(a),which is formed by the grey nodes, with the R node representing the selected root node, the H nodeas a hub selected in a previous iteration, and the grey lines indicating the path of the current virtuallinks. The white nodes are available for adding new hubs. Considering the current candidate as thenode A, one must select the nearest one as node H, since it is the nearest hub from A. However, asthe path that goes from H to its parent R passes through A, the parent of A is set as R and H as achild of A. Finally, one must set all the other children of A, as any node that is not an ancestor ofthe new hub (as the parent of A was already set, its ancestor is well defined as the node R) which hasa distance to A of less than its distance to its own parent. Notice that the nodes that can be itschildren are 1, 2, and 3. For such nodes, only the node 1 is nearer to node A than to its own parent,the others are nearer to node H. So, the new children of A are 1 and H. Figure 63(b) depicts the newconfiguration. The cost function should be calculated now and compared to other candidate results.
  • 98. 85(a) (b)Figure 63 Example of the placement procedure: (a) before and (b) after placementThese two procedures are the core of GHS, but there are two additional procedures. Theprocedure for selecting the root node tries to select the virtual node that minimizes the summing ofthe distances between all the allocated virtual nodes, which is equivalent to the MPP explained inSection 6.1, where the link weights are all equal to one and the summation is made only in a subsetof nodes. This problem can be solved in the same way as the MPP. After choosing the root node,the virtual links in initial virtual tree are created trough the shortest path between the root node andeach other requested node. If the virtual link between a requested node A and the root node passesthrough a requested node B the node B will be the parent of A.The other procedure is an external loop that calls the search procedure to greedily add newhubs to the network until the cost achieved by the placement of a new hub is not reduced. Thecomplexity of GHS is ܱሺܰଷሻ.Optimal Algorithm (OA)Because a Steiner tree never contains a cycle, then there exists a subgraph of the original graphwhich is a tree and contains a minimal Steiner tree for any given subset of nodes. Observe that, inthis subgraph there is only one path between any two nodes. Thus, if the graph is already a tree, theminimal Steiner tree between the given subset of nodes can be found on linear time through adepth-first search.Considering such property, an optimal algorithm to find the minimum length Steiner tree canbe proposed: for the connected component of the graph in which the subset of nodes is – whichcan be found by a breadth-first search after removing the links with weight greater than k –, countthe minimal number of links that are needed to be removed in order to turn this graph into a tree. Ifܰ is the number of nodes and ‫ܮ‬ the number of links of the considered component, this number isgiven by ݉ ൌ ‫ܮ‬ െ ܰ ൅ 1 [41]. Then, in this component, for each subset of ݉ links, remove themand find the optimal Steiner tree in this subgraph, considering that this graph is a tree, through thedepth-first search, as observed previously. As the algorithm tries every possible way of removingABCR123H HABCR123H
  • 99. 86these links, one of them will find the tree which leads to an optimal Steiner tree. The complexity ofthis algorithm is ܱሺቀ‫ܮ‬݉ቁ ሺܰ ൅ ‫ܮ‬ሻሻ.6.3.2 EvaluationThis section evaluates the minimum length Steiner tree algorithms discussed in the previous section.The section does not evaluate the overall solution for the creation of the virtual network, which hasthe binary search as the outer procedure, but it evaluates only the inner Steiner tree procedure, i.e.,this evaluation only covers the energy reduction objective.The evaluation was made through Monte Carlo simulations considering a fixed physicaltopology with the random positioning of the virtual nodes. In each simulation sample, the stress ofeach physical link is drawn from a uniform distribution and the virtual nodes to be connected arepositioned in a uniform way. Note that, each sample is independent, and the physical network isreturned to its initial values before a new set of requested nodes is attempted. The two RNPtopologies used in Section 6.2.4 and showed at Figure 55 are also used in this experiment. At eachrun of the simulation each algorithm (STA, GHS, and OA) is submitted to the same scenario andthe number of used physical links (the cost) in the tree is measured for each algorithmindependently.The factors varied in the experiment are showed at Table VII. For the old RNP topology, thenumber of requested virtual nodes in the network is varied though the simulations from 3 to 27nodes, which is equivalent to 11% and 100% of the physical nodes, respectively. For the currentRNP topology, this number ranges from 3 to 28. For each run, from a total of 1000, the requestednodes were positioned in a different physical node in a random way, with every subset of thephysical nodes equally likely to be selected without repetitions. For each sample, the relative errorbetween the costs of the GHS and STA algorithms was calculated against the optimal cost. All theresults showed use a confidence level of 95%, which are showed in the graphs.Table VII Factors and levels used in the GHS’s evaluationFactors LevelsPhysical network topologyOld RNP topology and Current RNP topology(one Worker per PoP)Number of requested virtual nodes3 to 27 (old RNP)3 to 28 (current TNP)Figure 64, Figure 65, Figure 66, and Figure 67 show the main results for the algorithms. Thegraphs in Figure 64 and Figure 66 present the percentage of samples that reached the optimum(relative error equal zero), and the graphs in Figure 65 and Figure 67 present the percentage ofsamples that reached a relative error below than five percent. These percentages are showed
  • 100. 87according to the number of requested nodes. Results for requests with 27 nodes for the scenario ofthe old RNP topology are not showed because this scenario is simple and all the samples reach theoptimum. The same occurs for 28 nodes in the current RNP topology scenario.As shown by Figure 64, the GHS algorithm achieves the optimum cost in 100% of the samplesfor virtual networks with 3 nodes and 95% to 99% of the samples for virtual networks of 4, 5, 6,and 7 nodes. However, the performance of GHS tends to decrease when the number of requestednodes increases. In the worst case, about 75% of the samples of GHS reach the optimum in thecases from 14 to 22 requested nodes. Moreover, for small virtual networks until 7 nodes, GHSstatistically outperforms STA, with the best cases occurring with 5 and 6 nodes where the differenceis about 6%. The performance of STA is high for few requested nodes, decreases in the middle ofthe range of virtual network size, and reaches the optimum when 26 nodes are requested.Figure 64 Percentage of optimal samples for GHS and STA in the old RNP topologyAs the number of nodes in the virtual network tends to the total number of nodes (27 in the oldRNP topology), the performance of STA is improved and outperforms GHS since the problemtends to be to compute the minimum spanning tree in the physical network. In the other hand, GHSis better for small networks because the placement strategy is designed to find the hubs in thephysical network whereas the STA strategy is to find the common links in the shortest pathsbetween the requested nodes, if there are no common links in these shortest paths STA cannot findthe hubs minimizing the cost.Looking only for the samples that reached the optimum, one can conclude that the GHSalgorithm is not adequate for bigger virtual networks. But, Figure 65 shows the performance of eachalgorithm considering the samples that reached a relative cost error less than 5% in relation to theoptimum. In this case, the GHS performance has significantly improved for virtual networks greateror equal than 19 nodes. For example, in the scenario with 26 nodes – the worst scenario for GHS60,00%70,00%80,00%90,00%100,00%3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26PercentageofsamplesthatreachedtheoptimumNumber of nodes of the Virtual Networkto be createdPerformance of GHS and STA against the Optimal AlgorithmGHSSTA
  • 101. 88considering only optimum samples – all the samples of the GHS algorithm obtained a relative errorlesser than 5%. Furthermore, the STA algorithm has improved for virtual networks with 19 nodesor more. This shows that, considering all virtual networks’ sizes, even the GHS algorithm reachingoptimality only in few cases, most samples reached less than 5% of the optimum in the old RNPtopology and the worst cases was for 69.1% of the samples with 16 nodes. For STA, the worst caseis 65.6% with 14 virtual nodes.Figure 65 Percentage of samples reaching relative error ≤ 5% in the old RNP topologyThe results for the current RNP topology are presented at Figure 66 and Figure 67. In thisscenario, the behavior of the results is similar to the behavior in the previous scenario. Again, theGHS outperforms STA for smaller virtual networks from 3 to 11 nodes and the inverse occurs forvirtual networks from 16 to 27 nodes. Thus, it must be observed that increasing the quantity ofphysical links can improve the results of the GHS algorithm, since in the old RNP topology GHSoutperforms STA until 7 nodes only. Moreover, performance of GHS is substantially improvedwhen considering samples that reached 5% of the optimum as occurred in the previous scenario.Figure 66 Percentage of optimal samples for GHS and STA in the current RNP topology60,00%70,00%80,00%90,00%100,00%3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26Percentageofsamplesthatreachedrelativeerrorlessthan5%Number of nodes of the Virtual Networkto be createdPerformance of GHSand STA against the Optimal Algorithm(w/5% error)GHSSTA40,00%50,00%60,00%70,00%80,00%90,00%100,00%3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27PercentageofsamplesthatreachedtheoptimumNumber of nodes of the Virtual Networkto be createdPercentageof samples that reachedthe optimumGHS STA
  • 102. 89Figure 67 Percentage of samples reaching relative error ≤ 5% in the current RNP topology6.4 DiscussionThis chapter presented how Nubilum performs the allocation of resources in a D-Cloud. It wasshowed as the comprehensive resource model employed by Nubilum leverages the usage of previousalgorithms for virtual network allocation present in the literature. In addition, this chapter presentedthe evaluation of the specific algorithms for self-optimization of resources in a D-Cloud consideringseveral aspects such as load balancing, energy consumption, network restrictions, and serverrestrictions.Considering particularly the problems involving virtual networks – both the allocation andcreation of virtual networks – it was showed that, due to the complexity introduced when copingwith several different requirements and objectives (from developers and the provider), the problemsare in NP. Thus, the proposed solutions for such problems are greedy algorithms which employsome sort of heuristics.In the problem where the virtual network is given, the proposed solution tries to minimize thenumber of virtual links allocated in each physical link considering restrictions associated to virtualnodes and virtual links. Note that the virtual link restriction does not consider link capacity, sinceconsidering unconstrained capacities seems better suited to a D-Cloud with a pay-as-you-go businessplan, thus, the problem differs from several previous ones on the field of virtual networks allocation.Other aspect that can be highlighted is that the proposed solution is not restricted to the allocationof an incoming virtual network. It could be used for continuous maintenance of the application inorder to guarantee developer’s requirements. Particularly, the algorithm for allocation of virtual links(Figure 52) could be called by the Application Monitor to reallocate a virtual link whose currentdelay is greater than the maximum required. Also, the same algorithm could be called by the D-40,00%50,00%60,00%70,00%80,00%90,00%100,00%3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27Percentageofsamplesthatreachedrelativeerrorlessthan5%Number of nodes of the Virtual Networkto be createdPercentageof samples that reached relativeerror less than 5%GHS STA
  • 103. 90Cloud Monitor to rearrange the overall virtual links on the D-Cloud in order to obtain a better loadbalance in the physical network.Other studied problem was to build a virtual network for a given set of virtual nodes in orderto minimize the energy consumption whereas balancing the load of physical links. Such a problemconsiders that the developer requested only virtual nodes with no virtual links, thus, the problemtries to capture provider’s requirements on load balancing and energy consumption. For thisproblem, two algorithms were proposed, one an optimum algorithm and other one based onheuristics. The heuristic approach was compared with a known approximation algorithm for theSteiner tree problem, and the results showed that the proposed heuristic is better suited for smallsize virtual networks whereas the traditional approximation algorithm is better for bigger virtualnetworks. One possible implementation of this algorithm in a production scenario could consideralternating between these algorithms according to the size of the virtual network in order to obtainthe best solution in each case.The optimum algorithm was used in the experiments as a baseline that references theperformance of the others algorithms. Observe that the optimum algorithm is adapted for thephysical network used in the test scenario that is close to a tree (27 nodes and 29 links). In this casethe optimum algorithm could be used for virtual network creation since the number of possiblecombinations is low, lowering the computing time required.
  • 104. 917 Conclusion“Non datur denarius incipientibus, sed finientibus, et corona non currentibus, sed pervenientibus.”Saint AugustineSmall cooperative datacenters can be attractive since they offer cheaper and low-power consumptionalternative that reduces the costs of centralized Clouds. These small datacenters can be built atdifferent geographical regions and connected to form a Distributed Cloud, which reduces costs bysimply provisioning storage, servers, and network resources close to end-users. Users in a D-Cloudare free to choose where to allocate their resources in order to attend specific market niches,constraints on jurisdiction of software and data, or quality of service aspects of their clients.One of the most important design aspects of D-Clouds is the availability of “infinite”computing resources. But, providing on-demand computing instances and network resources in adistributed scenario is not a trivial task, and the resource management system must be carefullydesigned in order to guarantee that both user and provider requirements are met satisfactorily. Suchdesign covers several aspects such as the optimization algorithms for resource management, thedevelopment of suitable resource and offering models, and the right mechanisms for resourcediscovery and monitoring should also be designed.This Thesis introduced Nubilum, a resource management system that offers a self-managedsolution for challenges on allocation, discovery, control, and monitoring of resources in DistributedClouds. This system can be seen as an integrated solution meeting the following requirements:• An suitable information model – called as CloudML – for describing the range of D-Cloudresources and application’s computational, topologyc, and geographic restrictions;• An integrated control plane for network and computational resources, which combinesHTTP-based interfaces and the Openflow protocol to enforce decisions in the D-Cloud;• A set of algorithms for allocation of resources to developer’s applications based on severaldifferent requirements and objectives.Next, the main contributions of Nubilum are described in Section 7.1 whereas the relatedpublications obtained during the PhD are presented at Sectio 7.2. Finally, some further work arelisted in Section 7.3.
  • 105. 927.1 ContributionsNubilum introduces a clear separation between the enforcement and the intelligence rolesplayed by the Manager and the Allocator, respectively. The Manager offers an abstraction of theavailable resources on the D-Cloud to the Allocator, which, in turn, uses algorithms to allocateresources for incoming developer’s requests.Note that the system defines a resource model but it intends to remain open for differentmodel-driven resource allocation strategies. The resource model is condensed in the CloudML, adescription language that expresses resources, services, and requests in an integrated way. CloudMLpresents interesting characteristics for both providers and developers. CloudML is used in Nubilumfor representing virtual and physical infrastructure, services, and requests.In terms of the solutions employed for controlling the D-Cloud, Nubilum uses openprotocols for communication between their components and open strategies to define theirprocesses. The overall communication is made through HTTP messages (using REST) andOpenflow messages. These messages are exchanged between the components for the coordinatedexecution of each process in the D-Cloud. One of these processes is the discovery process thatcombines Openflow registration messages, REST operations, and LLDP messages in an integratedand versatile solution for discovering of servers, switches, and the network topology.In terms of the allocation process, Nubilum performs the allocation of virtual machines andvirtual links on the physical servers and links. The use of Libvirt for management of the hypervisorsallows Nubilum to work with very heterogeneous infrastructures, whereas the use of Openflowallows the versatile reorganization of the network when necessary. The IP/MAC management iscentralized into the Manager, but the actual attribution of those addresses to virtual machines ismade by each Worker through a built-in DHCP server.The overall Nubilum’s control plane solution was evaluated through measurements in aprototype implementation of Nubilum. From those measurements some models were derived toestimate the load introduced by Nubilum’s control messages when performing their mainoperations. Such models show the linear growth of the control load in respect to the growth of thenumber of resources in the D-Cloud.Finally, some optimization algorithms were developed for two different problems. The firstproblem involves the on-line allocation of virtual networks in a physical network with the objectiveof balancing the load over the physical infrastructure. Such problem also considers restrictions ongeo-location, memory, processing, and network delay. Due to the complexity of this problem, itssolution is a procedure developed in two stages: one for node allocation and another for links.
  • 106. 93Compared with a previous approach designed for load balancing, our procedure showed to beadequate in scenarios with bigger virtual networks.The second studied case addresses the problem of allocating a virtual tree for connectingnodes, when the request does not contain any links. Again load balancing is the main objective, withthe energy reduction as a secondary objective. A two-step algorithm is designed for this problem,with an outer loop responsible for load balancing and an inner loop for energy reduction. The innerloop problem can be reduced to a minimum length Steiner tree problem and two algorithms areproposed for this step. One uses a greedy strategy for hub placement and the other uses acombinatorial approach to find the optimum. The heuristic-based algorithm is compared against aSteiner tree approximation algorithm with the optimum algorithm as the baseline. The resultsshowed that the hub placement heuristic is better suited for small virtual networks, while the Steinerapproximation has better results in scenarios with greater virtual networks.7.2 PublicationsSome results presented in this Thesis were developed in cooperation with the Location andAllocation Mechanisms for Distributed Cloud Servers, a research project carried by the GPRT(Grupo de Pesquisa em Redes e Telecomunicações) and funded by Ericsson Telecomunicações S.A., Brazil.Parts of this Thesis were published in the form of scientific papers at conferences andjournals. Table VIII shows all the papers produced, including the papers already accepted and thesubmitted papers that are under revision.The results and analysis developed in papers #1 and #2 are not part of this Thesis, but theymake part of a previous evaluation of several open-source systems for management of Clouds.These earlier works were an important milestone in the development of this Thesis since theyprovided sufficient knowledge about the technologies being used to leverage current CloudComputing setups. Paper #1, particularly, contributed with the concept of programmability appliedto Cloud Computing.Papers #3 and #4 are in the group of the conceptual contributions of this Thesis whichdiscuss the research challenges related to resource management on Clouds and D-Clouds,respectively. The book chapter is partially reproduced in Chapter 2, whereas Chapter 3 is composedby some parts of paper #4.The papers #5, #6, #7, and #8 compose the group of effective contributions of this Thesis.Part of the paper #5 introducing CloudML appears in Section 5.1 of this Thesis. The short paper #6corresponds to Chapter 4 and it presents the main components and modules of Nubilum.
  • 107. 94Table VIII Scientific papers produced# Reference Type Status1 P. Endo, G. Gonçalves, J. Kelner, D. Sadok. A Survey onOpen-source Cloud Computing Solutions. VIIIWorkshop in Clouds, Grids e Applications, Brazil, June,2010.ConferenceFull PaperPublished2 T. Cordeiro, D. Damálio, N. Pereira, P. Endo, A. Palhares,G. Gonçalves, D. Sadok, J. Kelner, B. Melander, V. Souza, J.Mångs. Open Source Cloud Computing Platforms. 9thInternational Conference on Grid and Cloud Computing,China, November 2010.ConferenceFull PaperPublished3 G. Gonçalves, P. Endo, T. Cordeiro, A. Palhares, D., J.Kelner, B. Melander, J. Mångs. Resource Allocation inClouds: Concepts, Tools and Research Challenges. 29ºSimpósio Brasileiro de Redes de Computadores e SistemasDistribuídos, 2011.BookChapterPublished4 P. Endo, A. Palhares, N. Pereira, G. Gonçalves, D. Sadok, J.Kelner, B. Melander, J. Mångs. Resource Allocation forDistributed Cloud – Concepts and Research Challenges.IEEE Network Magazine, July/August 2011.JournalFull PaperPublished5 G. Gonçalves, P. Endo, M. Santos, D. Sadok, J. Kelner, B.Melander, J. Mångs. CloudML: An Integrated Languagefor Resource, Service and Request Description for D-Clouds. IEEE Conference on Cloud ComputingTechnology and Science (CloudCom), December 2011.ConferenceFull PaperPublished6 G. Gonçalves, M. Santos, G. Charamba, P. Endo, D. Sadok,J. Kelner, B. Melander, J. Mångs. D-CRAS: DistributedCloud Resource Allocation System, IEEE/IFIP NetworkOperations and Management Symposium (NOMS), 2012ConferenceShort PaperPublished7.3 Future WorkAs can be noticed throughout this Thesis, the Cloud Computing paradigm will certainly be presentin our lives during the years to come. Also, D-Clouds should gain its niche slowly. But, before seeingnew developments, new research for automating the formation of such distributed environments isstill needed. These include strategies for advertising and scavenging for resources, loading andfreeing these automatically. Similar challenges include usage of different protocols for controllingdedicated network devices as radio stations, modems, and other devices that do not support theOpenflow protocol.Future works on this Thesis include testing the current implemented version of Nubilum in anISP-scale environment. Such case study would employ all the components of the system and someof the proposed algorithms to instantiate applications on a D-cloud environment. The main idea ofthis case study is to obtain performance data of the system in a real environment, allowingidentifying the bottlenecks and, eventually, engineering challenges that could be not perceived in thetests made in laboratory.
  • 108. 95Another future possibility would be adding support for elasticity on Nubilum. This aspectinvolves reviewing several aspects of the system. One first aspect to be considered is the change onthe CloudML information model, which should be modified to support elasticity rules that would beused by Nubilum to determine when and how to scale up and down the applications. These veryspecific rules differs from the common practice used in current Cloud Computing setups sincecreating a new virtual node can require the creation of one or several links, which would beindicated by the elasticity rules informed by the developer. Moreover, specific algorithms should bedeveloped to determine the suitable physical resources for the elastic growth of the virtual networks.
  • 109. 96References[1] ARMBRUST, M., FOX, A., GRIFFITH, R., JOSEPH, A.D., KATZ, R.H., KONWINSKI,A., LEE, G., PATTERSON, D.A., RABKIN, A., STOICA, I., and ZAHARIA, M. Abovethe Clouds: A Berkeley View of Cloud Computing, Tech. Rep. UCB/EECS-2009-28,EECS Department, University of California, Berkeley, 2009.[2] BARONCELLI, F., MARTINI, B., and CASTOLDI, P. Network virtualization for cloudcomputing, Annals of Telecommunications, v. 65, pp. 713-721, Springer-Verlag, 2010.[3] BELBEKKOUCHE, A., HASAN, M., and KARMOUCH, A. Resource Discovery andAllocation in Network Virtualization, IEEE Communications Surveys & Tutorials, n. 99,pp. 1-15, 2012.[4] BELOGLAZOV, A., BUYYA, R., LEE, Y. C., and ZOMAYA, A. A Taxonomy andSurvey of Energy-Efficient Data Centers and Cloud Computing Systems, Advances inComputers, v. 82, Elsevier, pp. 47-111, 2011.[5] BOLTE, M., SIEVERS, M., BIRKENHEUER, G., NIEHÖRSTER, O., andBRINKMANN, A. Non-intrusive Virtualization Management using libvirt. Conferenceon Design, Automation and Test in Europe (DATE), Germany, March 2010.[6] BORTHAKUR, D. The Hadoop Distributed File System: Architecture and Design.Available at: http://hadoop.apache.org/core/docs/current/hdfs_design.pdf. Last access:February, 2012.[7] BUYYA, R., BELOGLAZOV, A., and ABAWAJY, J. Energy-Efficient Management ofData Center Resources for Cloud Computing: A Vision, Architectural Elements, andOpen Challenges. International Conference on Parallel and Distributed ProcessingTechniques and Applications, pp. 6-17, 2010.[8] BUYYA, R., YEO, C.S., VENUGOPAL, S., BROBERG, J., and BRANDIC, I. Cloudcomputing and emerging IT platforms: Vision, hype, and reality for deliveringcomputing as the 5th utility. Future Generation Computer Systems, Elsevier B. V, 2009[9] CHAPMAN, C., EMMERICH, W., MARQUEZ, F. G., CLAYMAN, S., and GALIS, A.Software architecture definition for on-demand cloud provisioning. Proceedings of the19th ACM International Symposium on High Performance Distributed Computing, pp. 61-72, 2010.[10] CHOWDHURY, N. M. M. K., RAHMAN, M. R., and BOUTABA, R. Virtual NetworkEmbedding with Coordinated Node and Link Mapping, IEEE INFOCOM, 2009.[11] CHOWDHURY, N.M. M. K. and BOUTABA, R. A survey of network virtualization.Computer Networks, Vol. 54, issue 5, pp. 862-876, Elsevier, 2010.[12] CHURCH, K., GREENBREG, A., and HAMILTON, J. On Delivering EmbarrassinglyDistributed Cloud Services, Workshop on Hot Topics in Networks (HotNets), 2008.[13] CROCKFORD, D. JSON: The fat-free alternative to XML. Presented at XML 2006,Boston, USA, December 2006. Available at: http://www.json.org/fatfree.html. Last access:February, 2012.[14] CULLY, B., LEFEBVRE, G., MEYER, D., FEELEY, M., HUTCHINSON, N. andWARFIELD, A. Remus: High availability via asyncronous virtual machine replication.5th USENIX Symposium on Networked Systems Design and Implementation, April 2008.
  • 110. 97[15] DEAN, J. and GHEMAWAT, S. Mapreduce: simplified data processing on largeclusters. Proceedings of the 6th conference on Symposium on Operating Systems Design& Implementation, Berkeley, CA, USA, 2004.[16] DONGXI, L. and JOHN, Z. Cloud#: A Specification Language for Modeling Cloud,IEEE International Conference on Cloud Computing, pp. 533-540, 2011.[17] DUDKOWSKI, D., TAUHID, B., NUNZI, G., and BRUNNER, M. A Prototype for In-Network Management in NaaS-enabled Networks, 12th IFIP/IEEE InternationalSymposium on Integrated Network Management, pp. 81-88, 2011.[18] ECLIPSE FOUNDATION. Web Tools Platform, 2011. Available at:http://www.eclipse.org/webtools/. Last access: February, 2012.[19] ENDO, P. T., GONÇALVES, G. E., KELNER, J., and SADOK, D. A Survey on Open-source Cloud Computing Solutions. Workshop em Clouds, Grids e Aplicações, SimpósioBrasileiro de Redes de Computadores e Sistemas Distribuídos, 2010.[20] ENDO, P. T., PALHARES, A. V. A., PEREIRA, N. N., GONÇALVES, G. E., SADOK, D.,KELNER, J., MELANDER, B., and MANGS, J. E. Resource allocation for distributedcloud: concepts and research challenges, IEEE Network Magazine, vol. 25, pp. 42-46,July 2011.[21] GALAN, F., Sampaio, A., Rodero-Merino, L., Loy, I., Gil, V., and Vaquero, L. M. Servicespecification in cloud environments based on extensions to open standards.Proceedings of the Fourth International ICST Conference on Communication Systemsoftware and middleware, 2009.[22] GEYSERS Project, Initial GEYSERS Architecture & Interfaces Specification,Deliverable D2.1 of the Generalised Architecture for Dynamic Infrastructure Services(GEYSERS) FP7 Project, January 2010.[23] GHEMAWAT, S., GOBIOFF, H., and LEUNG, S. The Google file system. 19thSymposium on Operating Systems Principles, pages 29–43, Lake George, New York,2003.[24] GONÇALVES, G., ENDO, P., CORDEIRO, T., PALHARES, A., SADOK, D., KELNER,J., MELANDER, B., and MÅNGS, J. Resource Allocation in Clouds: Concepts, Toolsand Research Challenges. Simpósio Brasileiro de Redes de Computadores e SistemasDistribuídos, June 2011.[25] GONÇALVES, G., ENDO, P., SANTOS, M., SADOK, D., KELNER, J., MELANDER, B.MÅNGS, J. CloudML: An Integrated Language for Resource, Service and RequestDescription for D-Clouds. IEEE Conference on Cloud Computing Technology andScience (CloudCom), December 2011.[26] GONÇALVES, G., ENDO, P., SANTOS, M., SADOK, D., KELNER, J., MELANDER,B., MÅNGS, J. CloudML: An Integrated Language for Resource, Service and RequestDescription for D-Clouds. IEEE Conference on Cloud Computing Technology andScience (CloudCom), December 2011.[27] GREENBERG, A., HAMILTON, J., MALTZ, D. A., and PATEL, P. The cost of a cloud:research problems in data center networks. SIGCOMM Comput. Commun. Rev. 39, n.1, pp. 68-73, 2008.[28] GU, Y., and GROSSMAN, R. Sector and Sphere: The Design and Implementation of aHigh Performance Data Cloud, Philosophical Transactions: Series A, Mathematical,physical, and engineering sciences, v. 367, pp. 2429-2445, June 2009.
  • 111. 98[29] GUDE, N., KOPONEN, T., PETTIT, J., PFAFF, B., CASADO, M., MCKEOWN, N., andSHENKER. S. NOX: towards an operating system for networks. ACM SIGCOMMComputer Communication Review, v. 38, no. 3, July 2008.[30] HADAS, D., GUENENDER, S., and ROCHWERGER, B. Virtual Network Services forFederated Cloud Computing, Technical report H-0269, IBM, 2009.[31] HAIDER, A., POTTER, R., and NAKAO, A. Challenges in Resource Allocation inNetwork Virtualization. 20th ITC Specialist Seminar, pp. 18-20, Hoi An, Vietnam, May2009.[32] HOUIDI, I., LOUATI, W., ZEGHLACHE, D., and BAUCKE, S. Virtual ResourceDescription and Clustering for Virtual Network Discovery, Proceedings of IEEE ICCWorkshop on the Network of the Future, 2009.[33] HUMBLE, J. JavaSysMon, version 0.3.0, January 2010. Available at:https://github.com/jezhumble/javasysmon. Last access: February, 2012.[34] JUNGNICKEL, D. Graphs, Networks and Algorithms. Algorithms and Computation inMathematics, v.5, 3rdEdition, Springer-Verlag, 2007.[35] KARLSSON, M., KARAMANOLIS, C., and MAHALINGAM, M. A Framework forEvaluating Replica Placement Algorithms. Technical Report HPL-2002, HPLaboratories, July 2002.[36] KELLERER, H., PFERSCHY, U., and PISINGER, D. Knapsack Problems, 1stEdition,Springer-Verlag, 2004.[37] KHOSHAFIAN, S. Service Oriented Enterprises. Auerbach Publications, 2007.[38] KOOMEY, J. Growth in Data center electricity use 2005 to 2010. Analytics Press,Oakland, August 2011. Available at: http://www.analyticspress.com/datacenters.html. Lastaccess: February, 2012.[39] KOSLOVSKI, G.P., PRIMET, P.V.B., and CHARAO, A.S. VXDL: Virtual resources andinterconnection networks description language, Networks for Grid Applications, pp.138-154, Springer, 2009.[40] LAGAR-CAVILLA, H. A., WHITNEY, J. A., SCANNELL, A. M., PATCHIN, P.,RUMBLE, S. M., LARA, E., BRUDNO, M., and SATYANARAYANAN, M. SnowFlock:rapid virtual machine cloning for cloud computing. Fourth ACM European Conferenceon Computer Systems, 2009.[41] LEISERSON, C. E., STEIN, C., RIVEST, R. L., and CORMEN, T. H. Algoritmos: Teoriae Prática. Campus, ed. 1, 2002.[42] LISCHKA, J. and KARL, H. A virtual network mapping algorithm based on subgraphisomorphism detection, ACM SIGCOMM Workshop on Virtualized InfastructureSystems and Architectures, 2009.[43] MARSHALL, P., KEAHEY, K., and FREEMAN, T. Elastic Site: Using Clouds toElastically Extend Site Resources, 10th IEEE/ACM International Conference on Cluster,Cloud and Grid Computing, pp. 43-52, Australia, June, 2010.[44] MCKEOWN, N., ANDERSON, T., BALAKRISHNAN, H., PARULKAR, G.,PETERSON, L., REXFORD, J., SHENKER, S., and TURNER, J. OpenFlow: enablinginnovation in campus networks, ACM SIGCOMM Computer Communication Review,2008.[45] MELL, P. and GRANCE, T. The NIST Definition of Cloud Computing. NationalInstitute of Standards and Technology, Information Technology Laboratory, 2009.
  • 112. 99[46] METSCH, T., EDMONDS, A., and NYRÉN, R. Open Cloud Computing Interface –Core, Open Grid Forum, OCCI-WG, Specification Document. Available at:http://forge.gridforum.org/sf/go/doc16161?nav=1, 2010. Last access: February, 2012.[47] MORÁN, D., VAQUERO, L. M., and GALÁN, F. Elastically Ruling the Cloud:Specifying Application’s Behavior in Federated Clouds, IEEE International Conferenceon Cloud Computing, pp. 89-96, 2011.[48] MOSHARAF, N., CHOWDHURY, K., and BOUTABA, R. A survey of networkvirtualization. Computer Networks, v. 54, i. 5, pp. 862–876, April 2010.[49] MURPHY, M, and GOASGUEN, S. Virtual Organization Clusters: Self-provisionedclouds on the grid. In Future Generation Computer Systems, 2010.[50] NEVES, T. A., DRUMMOND, L. M. A., OCHI, L. S., ALBUQUERQUE, C., andUCHOA, E. Solving Replica Placement and Request Distribution in ContentDistribution Networks, Electronic Notes in Discrete Mathematics, Volume 36, pp. 89-96,ISSN 1571-0653, 2010.[51] OPENCLOUD. The Open Could Manifesto - Dedicated to the belief that the cloudshould be open, 2009. Available at: http://www.opencloudmanifesto.org/. Last access:February, 2012.[52] OPENFLOW PROJECT. OpenFlow Switch Specification, Version 1.1.0 Implemented,February 28, 2011.[53] OPENSTACK – Developer Guide – API v1.1. September, 2011. Available at:http://docs.openstack.org/api/openstack-compute/1.1/os-compute-devguide-1.1.pdf. Lastaccess: February, 2012.[54] PADALA, P. Automated management of virtualized data centers. Ph.D. Thesis, Univ.Michigan, USA, 2010.[55] PENG, B., CUI, B. and LI, X. Implementation Issues of a Cloud Computing Platform.IEEE Data Engineering Bulletin, volume 32, issue 1, 2009.[56] PRESTI, F. L., PETRIOLI, C., and VICARI, C. Distributed dynamic replica placementand request redirection in content delivery networks, MASCOTS, pp. 366–373, 2007.[57] QIU, L., PADMANABHAN, V., and VOELKER, G. On the Placement of Web ServerReplicas. Proceedings of IEEE INFOCOM, pages 1587–1596, April 2001.[58] RAZZAQ, A. and RATHORE, M. S. An approach towards resource efficient virtualnetwork embedding, IEEE Conference on High Performance Switching and Routing,2010.[59] ROCHWERGER, B., BREITGAND, D., EPSTEIN, A., HADAS, D., LOY, I., NAGIN, K.,TORDSSON, J., RAGUSA, C., VILLARI, M., CLAYMAN, S., LEVY, E.,MARASCHINI, A., MASSONET, P., MUÑOZ, H., and TOFETTI, G. Reservoir - WhenOne Cloud Is Not Enough. IEEE Computer, v. 44, i. 3, pp. 44-51, 2011.[60] SAIL Project, Cloud Network Architecture Description, Deliverable D-D.1 of theScalable and Adaptable Internet Solutions (SAIL) FP7 Project, July 2011.[61] SHETH, A and RANABAHU, A. Semantic Modeling for Cloud Computing, Part I.IEEE Computer Society - Semantics & Services, 2010.[62] VALANCIUS, V., LAOUTARIS, N., MASSOULIE, L., DIOT, C., and Rodriguez, P.Greening the Internet with Nano Data Centers. Proceedings of the 5th internationalconference on Emerging networking experiments and technologies, pp. 37-48. 2009.
  • 113. 100[63] VAQUERO, L. M., MERINO, L. R., and BUYYA, R. Dynamically Scaling Applicationsin the Cloud, ACM SIGCOMM Computer Communication Review, v. 41, n. 1, pp. 45- 52,January 2011.[64] VAQUERO, L., MERINO, L., CACERES, J., and LINDNER, M. A Break in the Clouds:Towards a Cloud Definition, vol. 39, pp. 50–55, January 2008.[65] VAZIRANI, V. V. Approximation Algorithms, 2ndEdition, Springer-Verlag, 2003.[66] VERAS, M. Datacenter: Componente Central da Infraestrutura de TI. Brasport Livrose Multimídia, Rio de Janeiro, 2009.[67] VERDI, F. L., ROTHENBERG, C. E., PASQUINI, R., and MAGALHÃES, M. Novasarquiteturas de data center para cloud computing. SBRC 2010 – Minicursos, 2010.[68] VOUK, M.A. Cloud Computing – Issues, Research and Implementations. Journal ofComputing and Information Technology, pages 235–246. University of Zagreb, Croatia,2008.[69] WHITE, S.R., HANSON, J.E., WHALLEY, I., CHESS, D.M., KEPHART, J.O. Anarchitectural approach to autonomic computing, Proceedings. International Conferenceon Autonomic Computing, vol., no., pp. 2- 9, 17-18, May 2004.[70] XHAFA, F. and ABRAHAM, A. Computational models and heuristics methods forGrid scheduling problems. In Future Generation Computer Systems, 2010.[71] YOUSEFF, L., BUTRICO, M., and SILVA, D. Toward a unified ontology of cloudcomputing. Grid Computing Environments Workshop, 2008.[72] ZHANG, Q., CHENG, L., and BOUTABA, R. Cloud computing: state-of-the-art andresearch challenges. Journal of Internet Service Applications, Springer, pp. 7-18, 2010.[73] ZHOU, D., ZHONG, L., WO, T., and KAN, J. CloudView: Describe and MaintainResource View in Cloud, IEEE Cloud Computing Technology and Science (CloudCom),pp. 151-158, 2010.[74] ZHU, Y. and AMMAR, M. Algorithms for assigning substrate network resources tovirtual network components, IEEE INFOCOM, pp. 1-12, 2006.