2013 05-fite-club-working-models-cloud-growing-up

Working Models ...
or how cloud is growing up and thinking about applications
Alex Heneveld / @ahtweetin
Fite Club, 22 May 2013

Cloud
It’s not just about VM’s...

It’s about storage, networking, maas, ...
Cloud

Ultimately it’s about applications.
Cloud

Declarative — AWS CloudFormation and OpenStack Heat
Procedural — Apache Whirr and Cloudsoft Brooklyn
OpenStack Summit — Red Hat, RAX, IBM, OASIS TOSCA and CAMP
DSLs in Progress — OpenStack Heat and CAMP PDP
Ultimately it’s about applications.

AWS CloudFormation
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "WordPress is web software …",
"Parameters" : {
"KeyName" : {
"Description" : "Name of an existing EC2 KeyPair for SSH access",
"Type" : "String" },
... },
"Mappings" : {
"AWSInstanceType2Arch" : {
"m1.small" : { "Arch" : "64" },
"m1.large" : { "Arch" : "64" },
... },
"AWSRegionArch2AMI" : {
"us-east-1" : { "32" : "ami-31814f58", "64" : "ami-1b814f72" }, ... } },
"Resources" : {
"WebServer": {
"Type": "AWS::EC2::Instance",
"Metadata" : {
"AWS::CloudFormation::Init" : {
"config" : {
"packages" : {
"yum" : {
"gcc-c++" : [],
"make" : [],
...

{
"Parameters" : {
"KeyName" : {
... },
"Mappings" : {
"m1.small" : { "Arch" : "64" },
"m1.large" : { "Arch" : "64" },
... },
"Resources" : {
"WebServer": {
"Metadata" : {
"config" : {
"packages" : {
"yum" : {
"gcc-c++" : [],
"make" : [],
...
"Resources" : {
"WebServer": {
"Type": "AWS::EC2::Instance"
AWS CloudFormation

"Resources" : {
"WebServer": {
Parameters
Mappings
(conﬁg metadata)
WaitConditions &
WaitHandles
(orchestration)
Outputs
Functions
AWS CloudFormation

"Resources" : {
"WebServer": {
AWS CloudFormation

"Type": "AWS::AutoScaling::AutoScalingGroup"
"Type": "AWS::EC2::EIP"
"Resources" : {
"WebServer": {
"Type": "AWS::SQS::Queue"
"Type": "AWS::CloudFormation::Stack"
"Type": "AWS::S3::Bucket"
AWS CloudFormation

"Type": "AWS::AutoScaling::AutoScalingGroup"
"Type": "AWS::EC2::EIP"
"Resources" : {
"WebServer": {
"Type": "AWS::SQS::Queue"
"Type": "AWS::CloudFormation::Stack"
"Type": "AWS::S3::Bucket"
AWS CloudFormation
Composition!

— Limited to AWS cloud
— Limited to AWS concepts
— Cumbersome to write
— Limited dependency injection
Nice, but:
AWS CloudFormation

— CloudFormation clone
— Targetting OpenStack resources and AWS compatibility
— Adds YAML support (comments and easier)
— Multi-cloud support using Apache Deltacloud
OpenStack Heat

(AWS CloudFormation)
{
"Parameters" : {
"KeyName" : {
... },
"Mappings" : {
"m1.small" : { "Arch" : "64" },
"m1.large" : { "Arch" : "64" },
... },
"Resources" : {
"WebServer": {
"Metadata" : {
"config" : {
"packages" : {
"yum" : {
"gcc-c++" : [],
"make" : [],
...
OpenStack Heat

OpenStack Heat
HeatTemplateFormatVersion: 2012-12-12
Description: WordPress is web software ...
Parameters:
  KeyName:
Description: Name of an existing EC2 KeyPair for SSH access
Type: String
...
Mappings:
  AWSInstanceType2Arch:
    m1.small: {Arch: '32'}
    m1.large: {Arch: '64'}
...
  DistroArch2AMI:
    F18: {'32': F18-i386-cfntools, '64': F18-x86_64-cfntools}, ...
Resources:
  WebServer:
    Type: AWS::EC2::Instance
    Metadata:
       AWS::CloudFormation::Init:
         config:
          packages:
             yum:
              gcc-c++: []
               make: []
...

— CloudFormation clone
— Targetting OpenStack resources and AWS compatibility
— Adds YAML support (comments and easier)
— Multi-cloud support using Apache Deltacloud
OpenStack Heat
Available NOW in Grizzly!

OpenStack Heat
— Still limited to selected clouds
— Still limited in concepts
— Only slightly less cumbersome to write
— Very limited dependency injection
But...

Procedural Approach
10 INPUT "What app do you like? ", A$
20 INPUT "What cloud do you like? ", C$
30 GOSUB 100
40 END
100 REM deploy C$ to A$
110 RETURN

Apache Whirr
provision
install
configure
manage
— Apache top level project
— Deploy multi-node applications
— Hadoop, Hama, Mahout, web,
Cassandra & many more
— Define topologies declaratively
— Add new roles, configuration,
constraints and wiring procedurally

$ whirr launch-cluster
--cluster-name=hbase
--instance-templates='1 hbase-master,6 hbase-regionserver'
— Deﬁne topologies declaratively
Apache Whirr

— Add new roles, conﬁguration,
constraints and wiring procedurally
spec = new ClusterSpec();
spec.setProvider("cloudservers-uk");
spec.setIdentity(apikey);
spec.setCredential(secret);
spec.setClusterName("hbase");
spec.setInstanceTemplates(ImmutableList.of(
new InstanceTemplate(1, "hbase-master"),
new InstanceTemplate(6, "hbase-regionserver")));
cluster = new ClusterController().launchCluster(spec);
Apache Whirr

— Limited parameterisation
— Deﬁning new roles requires coding
— Strict phases limit applicability
But...
— Composable to one dimension
— Portable locations support
— Pluggable with Chef, Puppet, and more
Apache Whirr

Brooklyn & Cloudsoft AMP
• Brooklyn = Apache-licensed OSS
• AMP = Enterprise + Support
• Deployment and Management
• Hierarchical Autonomic Policies

Brooklyn
Contention Management
Custom PaaS
Big Data

Region #1
Presentation
Geographic
DNS Load
Balancer
Tomcat
Processing (Monterey)
Data (Gemﬁre)
Tomcat
Tomcat
Region #2
Presentation
Load
Balancer
Tomcat
Data (Gemﬁre)
Tomcat
Tomcat
Provisioning
Monitoring
Management
Brooklyn: Example, Physical Architecture

Data (Gemﬁre)Processing (Monterey)
Region #1
Presentation
Geographic
DNS
Load
Balancer
Tomcat
Processing (Monterey) Data (Gemﬁre)
Tomcat
Tomcat
Region #2
Presentation
Load
Balancer
Tomcat
Tomcat
Tomcat
Provisioning Monitoring Management
WebappFabric
TomcatNodes
NginxNode
LoadBalancedWebAppCluster
TomcatNodes
NginxNode
WebappGeoDnsFabric MontereyFabric
MontereyCluster
MontereyNodes
MontereyNodes
MontereySegments
GemfireFabric
MontereyCluster
GemfireCluster
GemfireCluster
Geoscaling
Brooklyn
and friends
GemfireNodes
GemfireNodes
Brooklyn: Example, Logical Architecture

Autonomic Hierarchical Management
Geographic
DNS
Presentation
Load
Balancer
Tomcat Tomcat
WebappFabric
Cluster Cluster Cluster Cluster
MyApplication
Policy
follow-the-
{sun,moon,X}
Policy
resizer
Policy
restart-2x
Policy
Brooklyn
and friends
WebappGeoDnsFabric MontereyFabric GemfireFabric
Brooklyn

Typed Blueprints
replace groups with PaaS tools and
have the same sensors and policies
Geographic
DNS
Presentation
WebappFabric
PaaS
Cluster Cluster Service Cluster
MyApplication
Brooklyn
and friends
WebappGeoDnsFabric MontereyFabric GemfireFabric
Brooklyn
Policy
follow-the-
{sun,moon,X}
Policy
resizer
Policy
restart-2x
Policy

Data (Gemfire)
Data (Gemfire)
Presentation
Geographic
DNS
Load
Balancer
Tomcat
Tomcat
Tomcat
Presentation
Load
Balancer
Tomcat
Tomcat
Tomcat
WebappFabric
TomcatNodes
NginxNode
TomcatNodes
NginxNode
MontereyCluster
MontereyNodes
MontereyNodes
MontereySegments
GemfireFabric
MontereyCluster
GemfireCluster
GemfireCluster
Geoscaling
GemfireNodes
GemfireNodes fixed IP
&c
Brooklyn: Run in Many Locations

Data (Gemfire)
Data (Gemfire)
Presentation
Geographic
DNS
Load
Balancer
Tomcat
Tomcat
Tomcat
Presentation
Load
Balancer
Tomcat
Tomcat
Tomcat
WebappFabric
TomcatNodes
NginxNode
TomcatNodes
NginxNode
MontereyCluster
MontereyNodes
MontereyNodes
MontereySegments
GemfireFabric
MontereyCluster
GemfireCluster
GemfireCluster
Geoscaling
GemfireNodes
GemfireNodes
Policies
Policies
WAR
file
code schema
credentials
GeoDNS
config actor
descriptor
Regions
Brooklyn: finally { Simplify }

public class MyWebCluster extends AbstractApplication
implements MyWebClusterConstants {
public void init() {
MySqlNode mysql = addChild(EntitySpecs.spec(MySqlNode.class)
.configure("creationScriptUrl", DB_SETUP_SQL_URL));
ControlledDynamicWebAppCluster web = addChild(EntitySpecs.spec(
ControlledDynamicWebAppCluster.class)
.configure(WebAppService.HTTP_PORT, PortRanges.fromString("8080+"))
.configure(JavaWebAppService.ROOT_WAR, WAR_PATH)
.configure(javaSysProp("brooklyn.example.db.url"),
formatString("jdbc:%s%s?user=%s&password=%s",
attributeWhenReady(mysql, MySqlNode.MYSQL_URL),
DB_TABLE, DB_USERNAME, DB_PASSWORD)) );
}
}
Brooklyn: Nested Elastic Blueprints

public class MyWebCluster extends AbstractApplication
implements MyWebClusterConstants {
public void init() {
MySqlNode mysql = ...;
ControlledDynamicWebAppCluster web = ...;
web.addEnricher(HttpLatencyDetector.builder().url(ROOT_URL).
rollup(10, SECONDS).build());
web.getCluster().addPolicy(AutoScalerPolicy.builder().
metric(REQUESTS_PER_SECOND_IN_WINDOW_PER_NODE).
metricRange(10, 100).sizeRange(2, 5).build());
addEnricher(SensorPropagatingEnricher.newInstanceListeningTo(web,
ROOT_URL, REQUESTS_PER_SECOND_IN_WINDOW, REQUEST_LATENCY_IN_SECONDS_IN_WINDOW));
}
}
Brooklyn: Policies, Enrichers and KPI’s

Brooklyn
— Deﬁning blueprints requires coding
But...
— Easily parameterisable blueprints
— Composable and substitutable
— Portable and powerful locations support
— Pluggable with VM images, Heat, Chef, more

Applications
— Components (IaaS-resources)
— DeclarativeInfrastructure
— Procedural
— Extensible (new resources)
— Composable (stacks)
— Management
— Portable (hybrid, metal)

Heat in Grizzly
Management
Composable
Declarative
Components
Extensible
Portable
thanks largely to Red Hat

Portland Design Summit — Heat
Management
Composable
Declarative
Portable
Components
Extensible

Management
Composable
Declarative
Portable
Components
Extensible
lots of new blood (interest and ideas)

Composable
Declarative
Components
Management
Portable
Ease-of-Use
Extensible
ingredients from Brooklyn, Whirr, OoO and others

Composable
Declarative
Components
Management
Portable
Extensible
Auto-wiring
a special ingredient from Rackspace
Ease-of-Use

Composable
Declarative
Components
Management
Extensible
Auto-wiring
Relationships
Ease-of-Use
Portable
a special ingredient from IBM

Management
Composable
Declarative
Components
Auto-wiring
Relationships
Extensible
Ease-of-Use
Portable

Heat in Havana
Management
Composable
Declarative
Components
Auto-wiring
Relationships
Extensible
Ease-of-Use
Portable
Concurrent
Distributed
Resilient
Functions / DI

Heat in Havana
Management
Composable
Declarative
Components
Auto-wiring
Relationships
Extensible
Ease-of-Use
Portable
Concurrent
Distributed
Resilient
Functions / DI
Standards ?

Oasis
TOSCA
http://www.oasis-open.org/committees/tosca/
Topology and Orchestration Speciﬁcation for Cloud Applications

Modeling Topologies with TOSCA
Service Topologies are described using the TOSCA “Meta-model”:
Artifacts
 Describe Installables and Executables required to
instantiate and manage a service. Currently, they
include:
 Implementation Artifacts:
– Executables or Plans that implement a Node’s
or Relationship’s Operations (e.g. a Bash script)
 Deployment Artifacts:
– Installables of the components (e.g. a TAR file)
A service’s Topology Model is included in a TOSCA Service Template which is packaged and shared, along with
all dependent artifacts, as a TOSCA Cloud Service Archive (CSAR)
Service Templates
 Group the nodes and relationships that make up
a service’s topology
– Allowing modeling of sub-topologies
 Service Templates “look like nodes” enabling:
 Composition of applications from one or more
service templates
 Substitution of abstract Node types with
available service templates of the same type
Nodes
 Represent Components of an application or service
and their Properties. Example nodes include:
– Infrastructure: Compute, Network, Storage, etc.
– Platform: OS, VM, DB, Web Server, etc.
– Granular: functional Libraries, Modules, etc.
 Include Operations which are the management
functions for the node
– e.g. deploy(), start(), stop(), connect(), etc.
 Export their dependencies on other nodes
as Requirement and Capabilities
Relationships
 Represent the logical Relationships between
nodes
– e.g. “hostedOn”, “connectsTo”, etc.
 Describes the valid Source and Target nodes they
are designed to couple
– e.g. source “web application” node is designed
to “connectTo” a target “database” node
 Have their own Properties and Constraints
Slide origin: IBM
TOSCA: Modelling Topologies

<Definitions id="PayrollDefinitions" targetNamespace="http://www.example.com/ste"
xmlns:pay="http://www.example.com/ste/Types">
<Import namespace="http://www.example.com/ste/Types" location="http://.../PayrollTypes.tosca"
importType=" http://docs.oasis-open.org/tosca/ns/2011/12"/>
<ServiceTemplate id="Payroll" name="Payroll Service Template">
<TopologyTemplate ID="PayrollTemplate">
<NodeTemplate id="Payroll Application" type="pay:ApplicationNodeType">
<DeploymentArtifacts>
<DeploymentArtifact name="PayrollEAR" type="http://..../CSARref">
EARs/Payroll.ear
</DeploymentArtifact>
</DeploymentArtifacts>
</NodeTemplate>
<NodeTemplate id="Application Server" type="pay:ApplicationServerNodeType">
<DeploymentArtifacts>
<DeploymentArtifact name="ApplicationServerImage" type="http://.../AMIref">
ami-edf2cf99
</DeploymentArtifact>
</DeploymentArtifacts>
</NodeTemplate>
<RelationshipTemplate id="deployed_on" type="pay:deployed_on">
<SourceElement ref="Payroll Application"/>
<TargetElement ref="Application Server"/>
</RelationshipTemplate>
</TopologyTemplate>
</ServiceTemplate>
</Definitions>
TOSCA: Sample Deﬁnition

— Fails the“ease-of-use”test
— Does not standardise an API
But...
— Powerful and broad modelling coverage
— Workﬂow, locations, substitution
— Relatively mature
TOSCA

Oasis
CAMP
www.cloudspecs.org
Cloud Application Management Platform
http://www.oasis-open.org/committees/camp/

CAMP: Objectives and Non-Objectives
— Software
— an Orchestration Spec
— a Modelling System
But CAMP is NOT
— a REST API
— for Deployment and Management
— of Applications
It provides a ﬂexible way to use many platforms.
Interoperability.

CAMP: Key Concepts
Clients
REST
Platform
Platform
Component
Template
Requirement
Type
Platform
Component
TemplatePlatform
Component
Template
Requirement
Type
App Component
Type
App Component
Type

Clients Platform
Platform
Component
Template
Requirement
Type
Platform
Component
TemplatePlatform
Component
Template
Requirement
Type
App Component
Type
App Component
Type
PDP = Platform
Deployment Plan
(blueprint)
— Deployment PlanCAMP: Key Concepts
DSL & REST

App Component
Type
PDP = Platform
Deployment Plan
(blueprint)
CAMP: Key Concepts
Clients
REST
Platform
Platform
Component
Template
Requirement
Type
Platform
Component
TemplatePlatform
Component
Template
Requirement
Type
App Component
Type
— Creating Templates
POST

App Component
Template
Clients
REST
Platform
Platform
Component
Template
Requirement
Type
Platform
Component
TemplatePlatform
Component
Template
Requirement
Type
App Component
Type
App Component
Type
— ResolutionCAMP: Key Concepts
Assembly
Template
(blueprint)
PDP = Platform
Deployment Plan
(blueprint)POST

CAMP: Key Concepts
Clients
REST
Platform
Platform
Component
Template
Requirement
Type
Platform
Component
TemplatePlatform
Component
Template
Requirement
Type
App Component
Type
App Component
Type
App Component
Template
Requirement
Type
Platform
Component
Template
Requirement
Type
App Component
Type
— Resolution
Assembly
Template
(blueprint)
POST

Assembly
Template
(blueprint)
CAMP: Key Concepts
Clients
REST
Platform
Platform
Component
Template
Requirement
Type
Platform
Component
TemplatePlatform
Component
Template
Requirement
Type
App Component
Type
App Component
Type
App Component
Template
— Resolution
200
Assembly
Template
POST

Assembly
Template
(blueprint)
CAMP: Key Concepts
Clients
REST
Platform
Platform
Component
Template
Requirement
Type
Platform
Component
TemplatePlatform
Component
Template
Requirement
Type
App Component
Type
App Component
Type
App Component
Template
— Deployment

Assembly
(blueprint) Platform
Component
Platform
Component
App Component
Assembly
Template
(blueprint)
CAMP: Key Concepts
Clients
REST
Platform
Platform
Component
Template
Requirement
Type
Requirement
Type
App Component
Type
App Component
Type
App Component
Template
— Deployment
POST
Platform
Component
Template
Platform
Component
Template

Assembly
Component
Platform
Component
App Component
CAMP: Key Concepts
Clients
REST
Platform
— Deployment
Assembly
Component
Platform
Component
App Component
Assembly
Template
(blueprint)
Platform
Component
Template
App Component
Template
Platform
Component
Template
POST
200
Assembly

CAMP
— depends on orchestrations supporting this
— wants standardised requirement and component types
— still needs a nice DSL
But...
— all very general
— designed to map on to a wide range of orchestrations
PaaS, TOSCA, Brooklyn, Heat
— OSS code in progress
github.com/brooklyncentral/camp-server

CAMP: PDP/Blueprint DSL (work in progress)
Parameters
Components
Requirements
Functions
parameters & sensors
Outputs
*
*

CAMP: PDP/Blueprint DSL (work in progress)
Parameters
Functions
parameters & sensors
Outputs
*
*
components:
my_app:
type: WAR
content: { href: hello.war }
requirements:
hosted_on:
javaVersion: [1.7,)

Mappings
(conﬁg metadata)
WaitConditions &
WaitHandles
(orchestration)
Parameters
components:
my_instance:
type: server
ram: 2048m
requires:
connects_to:
target: my_database
Heat: New HOT DSL (also work in progress)
Functions
Outputs
Heat
Orchestration
Template
Coming in
Havana

Wrap-Up
Declarative — AWS CloudFormation and OpenStack Heat
Procedural — Apache Whirr and Cloudsoft Brooklyn
OpenStack Summit — Red Hat, RAX, IBM, OASIS TOSCA and CAMP
DSLs in Progress — OpenStack Heat and CAMP PDP
It’s about the apps.
Come get involved!

Thank you.
alex.heneveld@cloudsoftcorp.com
@ahtweetin

2013 05-fite-club-working-models-cloud-growing-up

More Related Content

Similar to 2013 05-fite-club-working-models-cloud-growing-up

More from Alex Heneveld

Recently uploaded

2013 05-fite-club-working-models-cloud-growing-up