Diego: Re-envisioning the Elastic Runtime (Cloud Foundry Summit 2014)
Upcoming SlideShare
Loading in...5
×
 

Diego: Re-envisioning the Elastic Runtime (Cloud Foundry Summit 2014)

on

  • 4,601 views

Keynote delivered by Onsi Fakhouri, Engineering Manager at Pivotal. ...

Keynote delivered by Onsi Fakhouri, Engineering Manager at Pivotal.

Diego is a ground-up rewrite of the DEA - a major component of the Cloud Foundry Elastic Runtime. This talk will motivate the need for Diego, the philosophy behind Diego, and present a few choice technical details to illustrate some of the more interesting ideas we've been playing with.

Statistics

Views

Total Views
4,601
Views on SlideShare
1,387
Embed Views
3,214

Actions

Likes
3
Downloads
82
Comments
0

11 Embeds 3,214

http://www.gopivotal.com 2663
https://2014.event.cfsummit.com 320
http://direct.gopivotal.com 88
https://www.gopivotal.com 46
http://www.dev.gopivotal.com 41
https://direct.gopivotal.com 32
https://twitter.com 14
http://www.pivotal.io 7
http://beratav037.vertexinc.com 1
http://mixmodelos.com 1
http://plus.url.google.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Diego: Re-envisioning the Elastic Runtime (Cloud Foundry Summit 2014) Diego: Re-envisioning the Elastic Runtime (Cloud Foundry Summit 2014) Presentation Transcript

  • Onsi Fakhouri DIEGOElasticRuntime 2.0 TECHNICAL
  • What? Why? Show me… The future DIEGOElasticRuntime 2.0
  • DIEGOElasticRuntime 2.0 What? Why? Show me… The future
  • Cloud Controller What is being rewritten? Stage App Run n App Instances (and keep them running) http://… Push App > cf Route to App
  • DEA Pool (Droplet Execution Agent) What is being rewritten? http://… Push App > cf Cloud Controller Router (API)
  • What is being rewritten? http://… Push App > cf Cloud Controller Router DEA Pool (Droplet Execution Agent) (API)
  • What is being rewritten? http://… Push App > cf Cloud Controller Router DEA Pool (Droplet Execution Agent) DEA Staging Apps Running Apps(API)
  • What is being rewritten? http://… Push App > cf Cloud Controller Router DEA Pool (Droplet Execution Agent) DEA Staging Apps Running Apps Warden Containerization (API)
  • What is being rewritten? http://… Push App > cf Cloud Controller Router DEA Pool (Droplet Execution Agent) DEA Staging Apps Running Apps Warden Containerization Health Manager (API)
  • What is being rewritten? Push App http://… > cf Cloud Controller Router Health Manager DEA Pool (Droplet Execution Agent) DEA Staging Apps Running Apps Warden Containerization NATS (message bus) (API)
  • What is being rewritten? Push App http://… > cf Cloud Controller Router Health Manager DEA Pool (Droplet Execution Agent) DEA Staging Apps Running Apps Warden Containerization NATS (message bus) (API)
  • What? Why? Show me… The future DIEGOElasticRuntime 2.0
  • Why rewrite? Push App http://… > cf Cloud Controller Router Health Manager NATS (message bus) DEA Pool (Droplet Execution Agent) DEA Staging Apps Running Apps Warden Containerization
  • Why rewrite? Hard to add new features Hard to maintain existing features Why?
  • Why rewrite? Cloud Controller Router Health Manager NATS (message bus) DEA Pool (Droplet Execution Agent) DEA Staging Apps Running Apps Warden Containerization Tight Coupling Poor separation of concerns Or ch es tr at ion
  • Why rewrite? Tight Coupling Poor separation of concerns Or ch es tr at ion > cf scale
  • Why rewrite? Tight Coupling Poor separation of concerns Or ch es tr at ion Cloud Controller > cf scale
  • Why rewrite? Cloud Controller Tight Coupling Poor separation of concerns Or ch es tr at ion > cf scale “Make it so”
  • Why rewrite? Cloud Controller Tight Coupling Poor separation of concerns Or ch es tr at ion > cf scale start/stop
  • Why rewrite? Cloud Controller Tight Coupling Poor separation of concerns Or ch es tr at ion > cf scale DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden
  • Why rewrite? Cloud Controller Tight Coupling Poor separation of concerns Or ch es tr at ion > cf scale DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden
  • Why rewrite? Cloud Controller Tight Coupling Poor separation of concerns Or ch es tr at ion > cf scale DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden
  • Why rewrite? Cloud Controller Tight Coupling Poor separation of concerns Or ch es tr at ion > cf scale DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden start start
  • Why rewrite? Cloud Controller Tight Coupling Poor separation of concerns Or ch es tr at ion > cf scale DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden start start
  • Why rewrite? Cloud Controller Tight Coupling Poor separation of concerns Or ch es tr at ion > cf scale DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden start start
  • Why rewrite? Cloud Controller Tight Coupling Poor separation of concerns Or ch es tr at ion > cf scale DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden start fails
  • Why rewrite? Cloud Controller Tight Coupling Poor separation of concerns Or ch es tr at ion > cf scale DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden start fails
  • Why rewrite? Cloud Controller Tight Coupling Poor separation of concerns Or ch es tr at ion > cf scale DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden start fails Too much responsiblity
  • Why rewrite? Tight Coupling Poor separation of concerns Cloud Controller DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden DEA Warden Triangular Dependencies
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden DEA Warden Cloud Controller DEA Warden
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden DEA Warden Cloud Controller DEA Warden When it’s time to upgrade the DEAs When it’s time to upgrade the DEAs
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden DEA Warden Cloud Controller DEA Warden When it’s time to upgrade the DEAs we perform a rolling deploy
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden DEA Warden DEA Warden Cloud Controller
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden DEA Warden DEA Warden Cloud Controller “bye!”
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden Cloud Controller “bye!” DEA Warden DEA Warden
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden Cloud Controller start! “bye!” DEA Warden DEA Warden
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden Cloud Controller start! “bye!” DEA Warden DEA Warden start!
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden Cloud Controller start! “bye!” DEA Warden DEA Warden start!
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden Cloud Controller start! “bye!” DEA Warden DEA Warden start! all clear!
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden Cloud Controller start! “bye!” DEA Warden DEA Warden start! all clear!
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden Cloud Controller start! “bye!” DEA Warden DEA Warden start! all clear! Problematic
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden Cloud Controller start! “bye!” DEA Warden DEA Warden start! all clear! Problematic
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden Cloud Controller start! “bye!” DEA Warden DEA Warden start! all clear! Problematic ?? ??
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden Cloud Controller start! “bye!” DEA Warden DEA Warden start! all clear! Problematic
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden Cloud Controller start! “bye!” DEA Warden DEA Warden start! all clear! Problematic
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden Cloud Controller start! “bye!” DEA Warden DEA Warden start! all clear! Problematic
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden Cloud Controller start! “bye!” DEA Warden DEA Warden all clear! Problematic start!
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden Cloud Controller start! “bye!” DEA Warden DEA Warden all clear! Problematic start!
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Health Manager DEA Warden DEA Warden Cloud Controller start! “bye!” DEA Warden DEA Warden all clear! Problematic start!
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Or ch es tr at ion
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Or ch es tr at ion complex interactions
  • Why rewrite? Tight Coupling Poor separation of concerns Triangular Dependencies Or ch es tr at ion hard to test complex interactions
  • Why rewrite? Tight Coupling Poor separation of concerns hard to test hard to reason through complex interactions Triangular Dependencies Or ch es tr at ion
  • Why rewrite? Domain Specific (app, app, app, app)
  • Why rewrite? Domain Specific (app, app, app, app) Push App http://… > cf Cloud Controller Router Health Manager NATS (message bus) DEA Pool (Droplet Execution Agent) DEA Staging Apps Running Apps Warden Containerization App
  • Push App http://… > cf Cloud Controller Router Health Manager NATS (message bus) DEA Pool (Droplet Execution Agent) DEA Staging Apps Running Apps Warden Containerization App Why rewrite? Domain Specific (app, app, app, app) App App Apps Apps App App App App App App App App App App App App App App App App App App
  • Why rewrite? Domain Specific (app, app, app, app) Hard to extend to new domains (e.g. cron-like jobs) Push App http://… > cf Cloud Controller Router Health Manager NATS (message bus) DEA Pool (Droplet Execution Agent) DEA Staging Apps Running Apps Warden Containerization App App App Apps Apps App App App App App App App App App App App App App App App App App App
  • Why rewrite? Platform Specific
  • DEA Staging Apps Running Apps Warden Containerization Why rewrite? Platform Specific
  • DEA Staging Apps Running Apps Warden Containerization Why rewrite? Platform Specific
  • DEA Staging Apps Running Apps Warden Containerization Why rewrite? Platform Specific DEA Staging Apps Running Apps Warden Containerization
  • DEA Staging Apps Running Apps Warden Containerization DEA Staging Apps Running Apps Warden Containerization Why rewrite? Platform Specific
  • DEA Staging Apps Running Apps Warden Containerization DEA Staging Apps Running Apps Warden Containerization Why rewrite? Platform Specific hard to maintain
  • DEA Staging Apps Running Apps Warden Containerization Why rewrite? Long-lived processes Tons of concurrency Low-level os interactions
  • Why rewrite? Platform Specific Domain Specific (app, app, app, app) Tight Coupling Poor separation of concerns Or ch es tr at ion Triangular Dependencies Hard to add new features to maintain existing features
  • What? Why? Show me… The future DIEGOElasticRuntime 2.0
  • Show me Diego Strong concurrency support Written in Golang Strongly typed Explicit error handling Promotes developer discipline Strong low-level OS support
  • Show me Diego Domain Specific (app, app, app, app) One-off Tasks (guaranteed to only run once) Long Running Processes (n monitored instances) The Right(?) Abstraction
  • Cloud Controller Show me Diego The Right(?) Abstraction
  • Cloud Controller Show me Diego The Right(?) Abstraction Executor Pool Run Tasks Launch Long Running Processes
  • Cloud Controller Executor Pool Show me Diego The Right(?) Abstraction Run Tasks Launch Long Running Processes Stager Stage App Run Task
  • Cloud Controller Executor Pool Show me Diego The Right(?) Abstraction Run Tasks Launch Long Running Processes App-Manager Run App Launch LRP Stager Stage App Run Task
  • Cloud Controller Executor Pool Show me Diego The Right(?) Abstraction App-Manager Run App Launch LRP Run Tasks Launch Long Running Processes Stager Stage App Run Task Express specific domain
  • Cloud Controller Executor Pool Show me Diego The Right(?) Abstraction App-Manager Launch LRP Run Tasks Launch Long Running Processes Stager Run Task Express specific domain In terms of generic recipes Run App Stage App
  • Cloud Controller Executor Pool Show me Diego The Right(?) Abstraction App-Manager Stager Express specific domain In terms of generic recipes Run Tasks Launch LRPs Rep Launch LRP Run Task Run App Stage App
  • Cloud Controller Executor Pool Show me Diego The Right(?) Abstraction App-Manager Stager Express specific domain In terms of generic recipes Exec Recipes Exec Run Tasks Launch LRPs Rep Launch LRP Run Task Run App Stage App
  • Cloud Controller Executor Pool Show me Diego The Right(?) Abstraction App-Manager Stager Express specific domain In terms of generic recipes Exec Recipes Exec Garden Manage Containers Run Tasks Launch LRPs Rep Launch LRP Run Task Run App Stage App
  • Cloud Controller Executor Pool Show me Diego The Right(?) Abstraction App-Manager Stager Express specific domain In terms of generic recipes Run Tasks Launch LRPs Rep Exec Recipes Exec Garden Manage Containers Linux Backend Run Containers Launch LRP Run Task Run App Stage App
  • Cloud Controller Executor Pool Show me Diego App-Manager Stager Express specific domain In terms of generic recipes Run Tasks Launch LRPs Rep Exec Recipes Exec Garden Manage Containers Linux Backend Run Containers GenericSpecific Launch LRP Run Task Run App Stage App
  • Cloud Controller Executor Pool Show me Diego App-Manager Stager Express specific domain In terms of generic recipes Run Tasks Launch LRPs Rep Exec Recipes Exec Garden Manage Containers Linux Backend Run Containers GenericSpecific Launch LRP Run Task Run App Stage App New features go here! (e.g. cron-like tasks)
  • Cloud Controller Executor Pool Show me Diego App-Manager Stager Express specific domain In terms of generic recipes Run Tasks Launch LRPs Rep Exec Recipes Exec Garden Manage Containers Linux Backend Run Containers GenericSpecific Flexibility Launch LRP Run Task Run App Stage App New features go here! (e.g. cron-like tasks)
  • Show me Diego Platform Specific
  • Show me Diego Platform Independent ✓ Cloud Controller Executor Pool App-Manager Run App Launch LRP Stager Stage App Run Task Express specific domain In terms of generic recipes Run Tasks Launch LRPs Rep Exec Recipes Exec Garden Manage Containers Linux Backend Run Containers
  • Cloud Controller Executor Pool App-Manager Run App Launch LRP Stager Stage App Run Task Express specific domain In terms of generic recipes Run Tasks Launch LRPs Rep Exec Recipes Exec Garden Manage Containers Linux Backend Run Containers Show me Diego Platform Independent ✓ ✓ ✓ ✓ ✓ ✓ ✓
  • Cloud Controller Executor Pool App-Manager Run App Launch LRP Stager Stage App Run Task Express specific domain In terms of generic recipes Run Tasks Launch LRPs Rep Exec Recipes Exec Garden Manage Containers Linux Backend Run Containers Show me Diego ✓ ✓ ✓ ✓ ✓ ✓ Platform Independent ✓
  • Show me Diego Linux Backend Run Containers Win Backend Run Containers Just 2 Things: Platform Independent ✓
  • Show me Diego Linux Backend Run Containers Win Backend Run Containers Just 2 Things: Platform Independent ✓
  • Tight Coupling Poor separation of concerns Or ch es tr at ion Triangular Dependencies Show me Diego
  • Health Manager Cloud Controller Show me Diego Rep Exec Rep Exec Rep Exec Rep Exec Or ch es tr at ion
  • Health Manager Cloud Controller Show me Diego Rep Exec Rep Exec Rep Exec Rep Exec Start! Start! Stop! Or ch es tr at ion
  • Health Manager Cloud Controller Show me Diego Rep Exec Rep Exec Rep Exec Rep Exec Want 3 Or ch es tr at ion
  • Health Manager Cloud Controller Show me Diego Rep Exec Rep Exec Rep Exec Rep Exec Want 3 Hold auctions… Or ch es tr at ion
  • Health Manager Cloud Controller Show me Diego Rep Exec Rep Exec Rep Exec Rep Exec Want 3 Hold auctions… … to distribute LRPs Or ch es tr at ion
  • Health Manager Cloud Controller Show me Diego Rep Exec Rep Exec Rep Exec Rep Exec Want 3 Hold auctions… … to distribute LRPs
  • Health Manager Cloud Controller Show me Diego Rep Exec Rep Exec Rep Exec Rep Exec Want 3 Hold auctions… … to distribute LRPs Triangular Dependencies
  • Health Manager Cloud Controller Show me Diego Rep Exec Rep Exec Rep Exec Rep Exec Want 3 Triangular Dependencies self managing monitoring healing
  • Health Manager Cloud Controller Show me Diego Rep Exec Rep Exec Rep Exec Rep Exec Want 3 self managing monitoring healing Triangular Dependencies
  • Health Manager Cloud Controller Show me Diego Rep Exec Rep Exec Rep Exec Rep Exec Want 3 self managing monitoring healing eventually consistent Triangular Dependencies
  • Show me Diego Cloud Controller Rep Exec Rep Exec Rep Exec Rep Exec Want 3 self managing monitoring healing eventually consistent
  • Show me Diego Rep Exec Rep Exec Rep Exec Rep Exec self managing monitoring healing eventually consistent robust Cloud Controller Want 3
  • Show me Diego Rep Exec Rep Exec Rep Exec Rep Exec but…
  • Show me Diego Rep Exec Rep Exec Rep Exec Rep Exec distributed auction is complex emergent behavior
  • Show me Diego Rep Exec Rep Exec Rep Exec Rep Exec distributed auction is complex emergent behavior Simulation-Driven Development
  • Show me Diego complex interactions hard to test hard to reason through
  • Show me Diego simulation driven complex interactions hard to test hard to reason through
  • complex interactions hard to test hard to reason through Show me Diego simulation driven Cloud Controller Executor Pool App-Manager Run App Launch LRP Stager Stage App Run Task Express specific domain In terms of generic recipes Run Tasks Launch LRPs Rep Exec Recipes Exec Garden Manage Containers Linux Backend Run Containers
  • Show me Diego executor rep stager 14 small single-responsibility components! app-manager auctioneer converger etcd-metrics-server etcd file-server garden linux-circus metricz route-emitter tps simulation driven complex interactions hard to test hard to reason through
  • Show me Diego executor rep stager app-manager auctioneer converger etcd-metrics-server etcd file-server garden linux-circus metricz route-emitter tps ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓✓ ✓ ✓ ✓ unit-tested✓ simulation driven complex interactions hard to test hard to reason through
  • Show me Diego executor rep stager app-manager auctioneer converger etcd-metrics-server etcd file-server garden linux-circus metricz route-emitter tps ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓✓ ✓ ✓ ✓ ?unit-tested✓ simulation driven complex interactions hard to test hard to reason through
  • Show me Diego rep✓ garden ✓ linux-circus✓ auctioneer✓ metricz✓ route-emitter✓ stager✓ app-manager✓ executor✓ file-server ✓ tps✓ etcd✓ converger ✓ etcd-metrics-server✓ unit-tested✓ simulation driven Actors complex interactions hard to test hard to reason through
  • Show me Diego unit-tested✓ simulation driven Diego is a play Actors rep✓ garden ✓ linux-circus✓ auctioneer✓ metricz✓ route-emitter✓ stager✓ app-manager✓ executor✓ file-server ✓ tps✓ etcd✓ converger ✓ etcd-metrics-server✓ complex interactions hard to test hard to reason through
  • Show me Diego rep✓ garden ✓ linux-circus✓ auctioneer✓ metricz✓ route-emitter✓ stager✓ app-manager✓ executor✓ file-server ✓ tps✓ etcd✓ converger ✓ etcd-metrics-server✓ communication and role encoded via shared library script shared narrative unit-tested✓ simulation driven Diego is a play Actors complex interactions hard to test hard to reason through
  • Show me Diego executor rep stager app-manager auctioneer converger etcd-metrics-server etcd file-server garden linux-circus metricz route-emitter tps ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ communication and role encoded via shared library script ✓integration tests ✓ Diego is a play Actors shared narrative unit-tested✓ simulation driven complex interactions hard to test hard to reason through
  • Show me Diego complexity in a distributed system of this scope is real and necessary Diego embraces this and tries to make its complexity: explicit transparent ∴ easier to reason about integration tests ✓ shared narrative unit-tested✓ simulation driven complex interactions hard to test hard to reason through
  • Show me Diego flexible abstraction extensible robust agile Tasks/LRPs Platform-Independent SELFManaging Handle on Complexity
  • What? Why? Show me… The future DIEGOElasticRuntime 2.0
  • The future staging running + buildpacks placement pools .NET process types auto-rebalancing 0-downtime deploys dockerfiles custom health-checks shell access persistent disk
  • DIEGOElasticRuntime 2.0 Rep Exec Rep Exec Rep Exec Rep Exec