☔
VERTEBRA
Bringing the Cloud Down To Earth
Jayson Vantuyl
Chief Systems Architect
Engine Yard, Inc.

Who Is This Guy?
Jayson Vantuyl
Chief Systems Architect /
Founder
Architecting Large
Networks Since 1994
Engine Yard, Inc.
Large Hosting Provider
focused on hosting Ruby
Applications
Managing Over 2,000
Virtual Servers

The Classic Blockers
My existing resources are not in the cloud.
A critical component is not in the cloud.
Critical components don’t fit together.
I’m on a budget: small first, scale later.
My boss wants it all (or some of it) in-house.

Fog Computing
It’s a Cloud, but it’s “down
to Earth”. Fog, get it?
Focus on What, Not Where
Clouds Without Borders
Don’t Rain On My Parade
A Sky Full Of Clouds

The Nuts and Bolts
Agents
Agents
XMPP
XMPP
Discovery
Security Security Workflow
& Resource Engine
Database
Discovery

XMPP
XML Messaging and Ordered Message Delivery,
Presence Protocol Presence
IETF Standard (RFC 3920, Many Implementations,
3921) Designed To Be Extensible
Each Connection Is Server-to-Server Security via
Essentially Two Streaming Dial-Back
XML Documents
Application-Level Reliability
Authentication via SRV DNS Records
Transport Security

Resources
Every point of control in the system is described
by a set of a “resources” (much like a URL).
These resources need not be unique in the cloud.
They are organized into hierarchies.
They are the point of abstraction.

Some Possible Resources
bob@mycompany.com: sally@mycompany.com:
/us/ny/new_york /ca/qc/montreal
Each box is an agent that /sales/copy_writer /sales/rep
offers operations on the jack@mycompany.com: jill@mycompany.com:
listed resources /us/dc/washington /us/fl/tampa
/research/weapons /ops/helpdesk
/clearance/s
/clearance/c
Geography /clearance/ers
/clearance/rs
web@mycompany.com: mail@mycompany.com:
Organizational Structure /net/tcpip /net/tcpip
/server/http /server/smtp
/blog /service/mailer
Security Clearance us_sec@mycompany.com: ca_sec@mycompany.com:
/security /security
/us /ca
Type Of Service us_hr@mycompany.com: ca_hr@mycompany.com:
/hr /hr
/us /ca

Operations
Fundamental Unit Of Work In Vertebra
Shaped Like Remote Procedure Call
Receives Parameters In A Namespace
Discover Agents That Provide The Resources
Request — Ack — Result* — Done
Scope (i.e. shotgun or sniper?)

Example Operations
instant_message(who=/sales,msg=”Goodbye.”)
fire(who=/manage/sales)
kick_out(who=/security/sales)
Completely Unrealistic add_entry(site=/blog,message=”Fired Sales!”)
Example
bob@mycompany.com: sally@mycompany.com:
/us/ny/new_york /ca/qc/montreal
/sales/copy_writer /sales/rep
Let’s Fire Some People, The
us_hr@mycompany.com: ca_hr@mycompany.com:
Entire Sales Dept. /manage /manage
/us /ca
Then Blog About It us_sec@mycompany.com:
/security
ca_sec@mycompany.com:
/security
/us /ca
web@mycompany.com:
/net/tcpip
/server/http
/blog

An Ominous Message
instant_message(who=/sales,msg=”Goodbye.”)
“who” is a resource, fire(who=/manage/sales)
discovery is done on kick_out(who=/security/sales)
/sales
add_entry(site=/blog,message=”Fired Sales!”)
We discover bob@mycompany.com:
/us/ny/new_york
sally@mycompany.com:
/ca/qc/montreal
sally@mycompany.com and /sales/copy_writer /sales/rep
bob@mycompany.com us_hr@mycompany.com: ca_hr@mycompany.com:
/manage /manage
/us /ca
Both get “instant_message” us_sec@mycompany.com: ca_sec@mycompany.com:
operation, and the agent on /security
/us
/security
/ca
their computer delivers the
message, “Goodbye.” web@mycompany.com:
/net/tcpip
/server/http
/blog

You’re Fired!
instant_message(who=/sales,msg=”Goodbye.”)
fire(who=/manage/sales)
“who” is a resource,
discovery is done on kick_out(who=/security/sales)
/manage/sales add_entry(site=/blog,message=”Fired Sales!”)
bob@mycompany.com: sally@mycompany.com:
We discover us_hr and /us/ny/new_york
/sales/copy_writer
/ca/qc/montreal
/sales/rep
ca_hr, which both offer
us_hr@mycompany.com: ca_hr@mycompany.com:
/manage /manage /manage
/us /ca
the “fire” operation goes to us_sec@mycompany.com:
/security
ca_sec@mycompany.com:
/security
the the HR departments for /us /ca
both countries web@mycompany.com:
/net/tcpip
/server/http
/blog

Kick ‘Em Out!
instant_message(who=/sales,msg=”Goodbye.”)
“who” is a resource, fire(who=/manage/sales)
discovery is done on kick_out(who=/security/sales)
/security/sales
add_entry(site=/blog,message=”Fired Sales!”)
We discover us_sec and bob@mycompany.com:
/us/ny/new_york
sally@mycompany.com:
/ca/qc/montreal
ca_sec, which provide /sales/copy_writer /sales/rep
/security us_hr@mycompany.com: ca_hr@mycompany.com:
/manage /manage
/us /ca
Security (in both countries) us_sec@mycompany.com: ca_sec@mycompany.com:
gets the operation to kick out /security
/us
/security
/ca
everybody that matches /
sales web@mycompany.com:
/net/tcpip
/server/http
/blog

Gloat!
instant_message(who=/sales,msg=”Goodbye.”)
fire(who=/manage/sales)
“site” is a resource, discovery
is performed on kick_out(who=/security/sales)
/blog add_entry(site=/blog,message=”Fired Sales!”)
bob@mycompany.com: sally@mycompany.com:
We discover /us/ny/new_york
/sales/copy_writer
/ca/qc/montreal
/sales/rep
web@mycompany.com,
us_hr@mycompany.com: ca_hr@mycompany.com:
which provides /blog /manage /manage
/us /ca
The operation to post the us_sec@mycompany.com:
/security
ca_sec@mycompany.com:
/security
blog entry goes to the web /us /ca
server agent. web@mycompany.com:
/net/tcpip
/server/http
/blog

Get Your Head In The Cloud
Operations in many agents with one request
Replace the “sales” entry with “us” to fire the
whole US workforce
Replace “sales” with “us/ny” to fire the New York
office
Can Subdivide the HR office any time and just
offer the correct resources

Get Your Head In The Cloud
The agents are abstracted.
HR in the US might be done by human with
some CRM.
HR in Canada might be automated, or might be
paper-only, with people doing the work.
Same for security, and the web server.

Reduced To Set Notation
In the abstract, a resource is a set that There are useful operations on sets of
maps to a set of agents resources, such as “coverage”
Resources may be singly inherited For any operation, the set of
discovered agents is the union of the
The set that maps to a resource is a sets mapped to by the intersection of
subset of the set that maps to its the set of requested resources and set
parent of actual resources

What Do We Get Out Of This?
Computational Patterns Component Abstraction
map-reduce Same Resources,
Different Code
traditional RPC
Same Resources, Same
message passing Code
job queue Different Resources,
Same Code
observer

What I Didn’t Tell You About
Responses from Operations Resource Advertising
and How They Aggregate
Federating Clouds
The RBAC Security
Implementation Fault-Tolerant, Distributed
Core Implementation
The Workflow Engine
Monitoring Protocol
The Resource-Keyed
Database Job Management Protocol
Intrinsic I18n Distributed Audit System
Extensible Data Marshaling Distributed Log System

This innovation brought to you by
Reliability, Ease of Use, Scalability