Vertebra: Bringing The Cloud Down To Earth

☔ VERTEBRA Bringing the Cloud Down To Earth Jayson Vantuyl Chief Systems Architect Engine Yard, Inc.

Who Is This Guy? Jayson Vantuyl Chief Systems Architect / Founder Architecting Large Networks Since 1994 Engine Yard, Inc. Large Hosting Provider focused on hosting Ruby Applications Managing Over 2,000 Virtual Servers

The Classic Blockers My existing resources are not in the cloud. A critical component is not in the cloud. Critical components don’t ﬁt together. I’m on a budget: small ﬁrst, scale later. My boss wants it all (or some of it) in-house.

Fog Computing It’s a Cloud, but it’s “down to Earth”. Fog, get it? Focus on What, Not Where Clouds Without Borders Don’t Rain On My Parade A Sky Full Of Clouds

The Nuts and Bolts Agents Agents XMPP XMPP Discovery Security Security Workﬂow & Resource Engine Database Discovery

XMPP XML Messaging and Ordered Message Delivery, Presence Protocol Presence IETF Standard (RFC 3920, Many Implementations, 3921) Designed To Be Extensible Each Connection Is Server-to-Server Security via Essentially Two Streaming Dial-Back XML Documents Application-Level Reliability Authentication via SRV DNS Records Transport Security

Resources Every point of control in the system is described by a set of a “resources” (much like a URL). These resources need not be unique in the cloud. They are organized into hierarchies. They are the point of abstraction.

Some Possible Resources bob@mycompany.com: sally@mycompany.com: /us/ny/new_york /ca/qc/montreal Each box is an agent that /sales/copy_writer /sales/rep offers operations on the jack@mycompany.com: jill@mycompany.com: listed resources /us/dc/washington /us/fl/tampa /research/weapons /ops/helpdesk /clearance/s /clearance/c Geography /clearance/ers /clearance/rs web@mycompany.com: mail@mycompany.com: Organizational Structure /net/tcpip /net/tcpip /server/http /server/smtp /blog /service/mailer Security Clearance us_sec@mycompany.com: ca_sec@mycompany.com: /security /security /us /ca Type Of Service us_hr@mycompany.com: ca_hr@mycompany.com: /hr /hr /us /ca

Operations Fundamental Unit Of Work In Vertebra Shaped Like Remote Procedure Call Receives Parameters In A Namespace Discover Agents That Provide The Resources Request — Ack — Result* — Done Scope (i.e. shotgun or sniper?)

Example Operations instant_message(who=/sales,msg=”Goodbye.”) fire(who=/manage/sales) kick_out(who=/security/sales) Completely Unrealistic add_entry(site=/blog,message=”Fired Sales!”) Example bob@mycompany.com: sally@mycompany.com: /us/ny/new_york /ca/qc/montreal /sales/copy_writer /sales/rep Let’s Fire Some People, The us_hr@mycompany.com: ca_hr@mycompany.com: Entire Sales Dept. /manage /manage /us /ca Then Blog About It us_sec@mycompany.com: /security ca_sec@mycompany.com: /security /us /ca web@mycompany.com: /net/tcpip /server/http /blog

An Ominous Message instant_message(who=/sales,msg=”Goodbye.”) “who” is a resource, fire(who=/manage/sales) discovery is done on kick_out(who=/security/sales) /sales add_entry(site=/blog,message=”Fired Sales!”) We discover bob@mycompany.com: /us/ny/new_york sally@mycompany.com: /ca/qc/montreal sally@mycompany.com and /sales/copy_writer /sales/rep bob@mycompany.com us_hr@mycompany.com: ca_hr@mycompany.com: /manage /manage /us /ca Both get “instant_message” us_sec@mycompany.com: ca_sec@mycompany.com: operation, and the agent on /security /us /security /ca their computer delivers the message, “Goodbye.” web@mycompany.com: /net/tcpip /server/http /blog

You’re Fired! instant_message(who=/sales,msg=”Goodbye.”) fire(who=/manage/sales) “who” is a resource, discovery is done on kick_out(who=/security/sales) /manage/sales add_entry(site=/blog,message=”Fired Sales!”) bob@mycompany.com: sally@mycompany.com: We discover us_hr and /us/ny/new_york /sales/copy_writer /ca/qc/montreal /sales/rep ca_hr, which both offer us_hr@mycompany.com: ca_hr@mycompany.com: /manage /manage /manage /us /ca the “ﬁre” operation goes to us_sec@mycompany.com: /security ca_sec@mycompany.com: /security the the HR departments for /us /ca both countries web@mycompany.com: /net/tcpip /server/http /blog

Kick ‘Em Out! instant_message(who=/sales,msg=”Goodbye.”) “who” is a resource, fire(who=/manage/sales) discovery is done on kick_out(who=/security/sales) /security/sales add_entry(site=/blog,message=”Fired Sales!”) We discover us_sec and bob@mycompany.com: /us/ny/new_york sally@mycompany.com: /ca/qc/montreal ca_sec, which provide /sales/copy_writer /sales/rep /security us_hr@mycompany.com: ca_hr@mycompany.com: /manage /manage /us /ca Security (in both countries) us_sec@mycompany.com: ca_sec@mycompany.com: gets the operation to kick out /security /us /security /ca everybody that matches / sales web@mycompany.com: /net/tcpip /server/http /blog

Gloat! instant_message(who=/sales,msg=”Goodbye.”) fire(who=/manage/sales) “site” is a resource, discovery is performed on kick_out(who=/security/sales) /blog add_entry(site=/blog,message=”Fired Sales!”) bob@mycompany.com: sally@mycompany.com: We discover /us/ny/new_york /sales/copy_writer /ca/qc/montreal /sales/rep web@mycompany.com, us_hr@mycompany.com: ca_hr@mycompany.com: which provides /blog /manage /manage /us /ca The operation to post the us_sec@mycompany.com: /security ca_sec@mycompany.com: /security blog entry goes to the web /us /ca server agent. web@mycompany.com: /net/tcpip /server/http /blog

Get Your Head In The Cloud Operations in many agents with one request Replace the “sales” entry with “us” to fire the whole US workforce Replace “sales” with “us/ny” to fire the New York office Can Subdivide the HR office any time and just offer the correct resources

Get Your Head In The Cloud The agents are abstracted. HR in the US might be done by human with some CRM. HR in Canada might be automated, or might be paper-only, with people doing the work. Same for security, and the web server.

Reduced To Set Notation In the abstract, a resource is a set that There are useful operations on sets of maps to a set of agents resources, such as “coverage” Resources may be singly inherited For any operation, the set of discovered agents is the union of the The set that maps to a resource is a sets mapped to by the intersection of subset of the set that maps to its the set of requested resources and set parent of actual resources

What Do We Get Out Of This? Computational Patterns Component Abstraction map-reduce Same Resources, Different Code traditional RPC Same Resources, Same message passing Code job queue Different Resources, Same Code observer

What I Didn’t Tell You About Responses from Operations Resource Advertising and How They Aggregate Federating Clouds The RBAC Security Implementation Fault-Tolerant, Distributed Core Implementation The Workﬂow Engine Monitoring Protocol The Resource-Keyed Database Job Management Protocol Intrinsic I18n Distributed Audit System Extensible Data Marshaling Distributed Log System

This innovation brought to you by Reliability, Ease of Use, Scalability

Vertebra: Bringing The Cloud Down To Earth

by jvantuyl on Jun 24, 2008

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Vertebra: Bringing The Cloud Down To Earth — Presentation Transcript