SlideShare a Scribd company logo
1 of 71
Download to read offline
Habitat 301: Building
Habitats
Chef Conf 2016
Jamie Winsor
@resetexistence
github.com/reset
Chef Conf 2016
NEW SLIDE HERE
Chef Conf 2016
Chef Conf 2016
Great for infrastructure,
Good for applications
Chef Conf 2016
Application Automation Principles
4 Automation Needs To Live With Your Application
4 Build Failures over Runtime Failures
4 Choreography over Orchestration
Chef Conf 2016
Atomic, Isolated,
Repeatable & Auditable
Chef Conf 2016
Chef Conf 2016
Builder
SaaS To Build Your Packages
Chef Conf 2016
It's already live!
4 Web - https://app.habitat.sh
4 API - https://app.habitat.sh/v1
Chef Conf 2016
Why build Builder?
4 Public hosting for packages
4 Public builders for packages
4 History of where a package came from
Chef Conf 2016
Automatic Re-Build/
Publish On Dependency
Update
Chef Conf 2016
pkg_name=hab-builder-api
pkg_origin=core
pkg_version=0.7.0
pkg_maintainer="Jamie Winsor <reset@chef.io>"
pkg_license=('apachev2')
pkg_source=nosuchfile.tar.gz
pkg_bin_dirs=(bin)
pkg_deps=(core/glibc core/openssl core/coreutils core/gcc-libs
core/zeromq core/libsodium core/libarchive)
pkg_build_deps=(core/protobuf core/protobuf-rust core/coreutils
core/cacerts core/rust core/gcc core/pkg-config
core/node core/phantomjs)
pkg_expose=(9636)
srv_bin="bldr-api"
pkg_svc_run="bin/$srv_bin start -c ${pkg_svc_path}/config.toml"
Chef Conf 2016
Building Builder
4 Educate about Builder
4 For users
4 For collaborators
4 Show how Habitat helped to build Builder
Chef Conf 2016
It's about to get
technical
Chef Conf 2016
Service Oriented Design
4 Gateway nodes
4 Router nodes
4 Service nodes
Chef Conf 2016
Service Layout
Chef Conf 2016
Why Rust?
4 Systems programming language
4 Fast
4 Memory safety without a
virtual machine
4 Thread safety guarantees
4 Great tooling
4 Basically a modern C
Chef Conf 2016
Walkthrough: Starting a
new build
Chef Conf 2016
Hop 1/10: Client
4 Web Client
4 https://app.habitat.sh
4 Angular 2
4 HTTP Client
4 REST
4 application/json
Chef Conf 2016
Hop 2/10: Gateway
4 Public facing edge node
4 External authentication/
identification by OAuth/
Authorization header
4 Transform client-request into
net-request & forward to
router
4 Await reply if transactional
and send reply to web-client
Chef Conf 2016
Authorization
4 Read value of Authorization
header into protocol message
4 Forward protocol message to
any RouteSrv
4 RouteSrv forward request to
appropriate SessionSrv
Chef Conf 2016
Builder-Protocol/Net
Protobuf & ZeroMQ Sockets
Chef Conf 2016
Protobuf
4 Language agnostic DSL
4 Backwards compatibility between service versions
4 Composable, modular messages
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
}
Chef Conf 2016
ZeroMQ
4 Socket library for building distributed services
4 Not a message queue/message bus
4 Makes no assumptions for your protocol
4 Guarantees for multi-part message delivery
Chef Conf 2016
components/builder-protocol/
protocols/net.proto
message Msg {
// hint for decoders
required string message_id = 1;
// serialized msg body
required bytes body = 2;
// used by RouteSrv to know where to send msg
optional RouteInfo route_info = 3;
}
Chef Conf 2016
Authorization (2)
// components/builder-protocol/protocols/sessionsrv.proto
message SessionGet {
required string token = 1;
}
4 Create new SessionGet msg
4 Write value of Authorization header into token field
4 Forward SessionGet message to any RouteSrv
Chef Conf 2016
Hop 3/10: RouteSrv
// components/builder-protocol/protocols/net.proto
message RouteInfo {
// Determines destination service
required Protocol protocol = 1;
// Determines destination shard
optional uint64 hash = 2;
}
enum Protocol {
Net = 0;
RouteSrv = 1;
SessionSrv = 2;
VaultSrv = 3;
JobSrv = 4;
}
4 Read RouteInfo and forward to
appropriate service
Chef Conf 2016
Hop 4/10:
SessionSrv
1. Receive message from
RouteSrv
2. Dispatch SessionGet to handler
3. Handler reads token field of
message and looks in datastore
for session
Chef Conf 2016
Replying to Request
4 On Success: Session
4 On Failure: NetError
message Session {
required uint64 id = 1;
required string email = 2;
required string name = 3;
required string token = 4;
}
Chef Conf 2016
NetError: Transaction Failures
message NetError {
required ErrCode code = 1;
required string msg = 2;
}
enum ErrCode {
// other codes...
SESSION_EXPIRED = 8;
// more codes...
}
Chef Conf 2016
Dissecting The Message
Code: 8, Msg: rg:auth:1
4 Code: for the user
4 Msg: for the developer
4 service: rg
4 operation/function: auth
4 case: 1
Chef Conf 2016
Hop 5/10: RouteSrv
4 Receive transaction reply from
SessionSrv
4 Send response to appropriate
HTTP-Gateway
Chef Conf 2016
Hop 6a/10: Gateway
1. Receive reply from RouteSrv
2. Forward reply to client thread
3. Client thread creates
ProjectGet message from HTTP
params
Chef Conf 2016
Hop 6b/10: Gateway
// protocols/vaultsrv.proto
message ProjectGet {
required uint64 id = 1;
}
4 Send ProjectGet through
RouteSrv to VaultSrv
Chef Conf 2016
Actual Service Cluster
Chef Conf 2016
Chef Conf 2016
Builder Is Service Oriented & Sharded
4 Service goes down: partial outage for all users
4 Shard goes down: partial outage of service for
subset of users
4 Easily add new machines by re-balancing shards
Chef Conf 2016
Message Routing (cont.)
// components/builder-protocol/protocols/net.proto
message RouteInfo {
// Determines destination service
required Protocol protocol = 1;
// Determines destination shard
optional uint64 hash = 2;
}
4 Populate protocol with message's protocol
4 Populate hash with a message's RouteKey
Chef Conf 2016
Defining The RouteKey
Chef Conf 2016
Routable
/// Defines a contract for protocol messages to be routed through `RouteSrv`.
pub trait Routable: protobuf::Message + protobuf::MessageStatic {
/// Type of the route key
type H: RouteKey + fmt::Display;
/// Return a `RouteKey` for `RouteSrv` to know which key's
/// value to route on.
///
/// If `Some(T)`, the message will be routed by hashing the
/// value of the route key and modding it against the shard
/// count. This is known as "randomly deterministic routing".
///
/// If `None`, the message will be randomly routed to an available node.
fn route_key(&self) -> Option<Self::H>;
}
Chef Conf 2016
Protobuf Definition
message ProjectGet {
required uint64 id = 1;
}
Trait Implementation
impl Routable for ProjectGet {
type H = InstaId;
fn route_key(&self) -> Option<Self::H> {
Some(InstaId(self.get_id()))
}
}
Chef Conf 2016
InstaId
4 64-bit integer containing
4 create-time
4 sequence-id (generated by database, 0-1024)
4 shard (0-127)
4 Inspired by Instagram engineering1
1
http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram
Chef Conf 2016
pub const EPOCH_MS: u64 = 1460499133628;
pub const SHARD_COUNT: u32 = 128;
pub struct InstaId(pub u64);
impl InstaId {
pub fn generate(auto_id: u64) -> Self {
let time = Self::since_epoch();
let id = auto_id % 1024;
let shard_id = id % SHARD_COUNT as u64;
let mut iid: u64 = time << 23;
iid |= id << 13;
iid |= shard_id;
InstaId(iid)
}
pub fn since_epoch() -> u64 {
let timespec = time::get_time();
let sec: u64 = timespec.sec as u64 * 1000;
let nsec: u64 = timespec.nsec as u64 / 1000 / 1000;
(sec + nsec) - EPOCH_MS
}
}
Chef Conf 2016
Hop 7/10: VaultSrv
1. Receive message from
RouteSrv
2. Dispatch ProjectGet to handler
3. Read Project from database
4. Reply to HTTP-Gateway
through RouteSrv
Chef Conf 2016
What is VaultSrv?
4 Persistent storage for
4 Origin memberships
4 Origin invitations
4 Projects
Chef Conf 2016
Persisting Protobufs Into
The Database
Chef Conf 2016
Persistable
/// Defines a contract for protocol messages to be persisted to a datastore.
pub trait Persistable: protobuf::Message + protobuf::MessageStatic {
/// Type of the primary key
type Key: fmt::Display;
/// Returns the value of the primary key.
fn primary_key(&self) -> Self::Key;
/// Sets the primary key to the given value.
fn set_primary_key(&mut self, value: Self::Key) -> ();
}
Chef Conf 2016
Protobuf Definition
message Project {
required uint64 id = 1;
required uint64 owner_id = 2;
required uint64 origin_id = 3;
required string plan_path = 4;
oneof vcs {
VCSGit git = 5;
}
}
Trait Implementation
impl Persistable for Project {
type Key = u64;
fn primary_key(&self) -> Self::Key {
self.get_id()
}
fn set_primary_key(&mut self, value: Self::Key) {
self.set_id(value);
}
}
Chef Conf 2016
Hop 8/10: REST-
Gateway
1. Receive reply from RouteSrv
2. Create new JobSpec from
Session & Project
3. Send JobSpec through RouteSrv
to JobSrv
message JobSpec {
required uint64 owner_id = 1;
required vault.Project project = 2;
}
Chef Conf 2016
Hop 9/10: JobSrv
1. Receive message from
RouteSrv
2. Dispatch JobSpec to job_create
handler
3. Create Job from JobSpec
4. Persist Job to database/queue
Chef Conf 2016
What is JobSrv?
4 Store job history
4 Job is an entity created by a
project
4 Active jobs are placed in a
queue
4 Different qeueues for
different OSes
4 Job create success/failure
Chef Conf 2016
Hop 9b/10: JobSrv
1. Send Job through RouteSrv to
HTTP-Gateway
message Job {
required uint64 id = 1;
required uint64 owner_id = 2;
required JobState state = 3;
required vault.Project project = 4;
}
enum JobState {
Pending = 0;
Processing = 1;
Complete = 2;
Rejected = 3;
Failed = 4;
}
Chef Conf 2016
Hop 10/10: REST-
Gateway
1. Receive reply from RouteSrv
2. Forward reply to client thread
3. Client thread serialize Job into
JSON
4. Send 200 to client with encoded
Job
Chef Conf 2016
JSON Trait Implementation
impl ToJson for Job {
fn to_json(&self) -> Json {
let mut m = BTreeMap::new();
m.insert("id".to_string(), self.get_id().to_json());
m.insert("state".to_string(), self.get_state().value().to_json());
Json::Object(m)
}
}
JSON Response
{
"id": 62214184531664897,
"state": 0,
}
Chef Conf 2016
Worker Pool 11/10
Chef Conf 2016
Chef Conf 2016
Workers
4 (N) workers connected to (X) JobSrv
4 OS specific
4 Pops a message off a JobSrv queue
4 Builds are done within a studio on the worker
4 Finished builds published to public depot as project's
package ident
Chef Conf 2016
Habitat Builder Is Self Hosted
4 Habitat supervises all processes
4 Supervisor watches public depot (self) for package
updates
4 Builder builds it's own updates
4 Automatically updates when a build is published to
public depot
Chef Conf 2016
Chef Conf 2016
Chef Conf 2016
Where Is config/default.toml?
Chef Conf 2016
Builder Uses Native Habitat Config
[cfg]
http_addr = "0.0.0.0:9636"
[cfg.github]
client_id = "{clientid}"
client_secret = "{clientsecret}"
[[bind.router.members]]
alive = true
ip = "10.0.0.190"
port = "5562"
service = "hab-builder-router"
Chef Conf 2016
Director
Chef Conf 2016
Director systemd
[Unit]
Description=Habitat Director
[Service]
ExecStart=/bin/hab-director start -c /hab/etc/director/config.toml
[Install]
WantedBy=default.target
Chef Conf 2016
Director Config
[cfg.services.core.redis.${env}]
start = "--permanent-peer --peer ${peer_ip}:9000"
[cfg.services.core.hab-builder-api.${env}]
start = "--permanent-peer --bind database:redis.${env},router:hab-builder-router.${env}
--peer ${peer_ip}:9000"
Chef Conf 2016
Bind Is Service Discovery
Chef Conf 2016
Shard Allocation
Chef Conf 2016
Electing A Master Router
4 Automatically elected and configured by Habitat
through the --topology leader flag
4 Configured to expect (N) service nodes
4 Gives out shard assignments on service connect
Chef Conf 2016
Bring Your Own Worker
Chef Conf 2016
Habitat Hack Day
Thursday, 10AM-3PM
Chef Conf 2016

More Related Content

What's hot

Essential Tools for Modern PHP
Essential Tools for Modern PHPEssential Tools for Modern PHP
Essential Tools for Modern PHPAlex Weissman
 
Fast and Reliable Swift APIs with gRPC
Fast and Reliable Swift APIs with gRPCFast and Reliable Swift APIs with gRPC
Fast and Reliable Swift APIs with gRPCTim Burks
 
Go 1.8 'new' networking features
Go 1.8 'new' networking featuresGo 1.8 'new' networking features
Go 1.8 'new' networking featuresstrikr .
 
Create a PHP Library the right way
Create a PHP Library the right wayCreate a PHP Library the right way
Create a PHP Library the right wayChristian Varela
 
Openshift: The power of kubernetes for engineers - Riga Dev Days 18
Openshift: The power of kubernetes for engineers - Riga Dev Days 18Openshift: The power of kubernetes for engineers - Riga Dev Days 18
Openshift: The power of kubernetes for engineers - Riga Dev Days 18Jorge Morales
 
Red Hat OpenShift Operators - Operators ABC
Red Hat OpenShift Operators - Operators ABCRed Hat OpenShift Operators - Operators ABC
Red Hat OpenShift Operators - Operators ABCRobert Bohne
 
DevOps with OpenShift - Fabien Dupont - ManageIQ Design Summit 2016
DevOps with OpenShift - Fabien Dupont - ManageIQ Design Summit 2016DevOps with OpenShift - Fabien Dupont - ManageIQ Design Summit 2016
DevOps with OpenShift - Fabien Dupont - ManageIQ Design Summit 2016ManageIQ
 
kubernetes operators
kubernetes operatorskubernetes operators
kubernetes operatorsJuraj Hantak
 
Kubernetes Operators: Rob Szumski
Kubernetes Operators: Rob SzumskiKubernetes Operators: Rob Szumski
Kubernetes Operators: Rob SzumskiRedis Labs
 
Meetup Openshift Geneva 03/10
Meetup Openshift Geneva 03/10Meetup Openshift Geneva 03/10
Meetup Openshift Geneva 03/10MagaliDavidCruz
 
Ruby and Rails Packaging to Production
Ruby and Rails Packaging to ProductionRuby and Rails Packaging to Production
Ruby and Rails Packaging to ProductionFabio Kung
 
Enforcing API Design Rules for High Quality Code Generation
Enforcing API Design Rules for High Quality Code GenerationEnforcing API Design Rules for High Quality Code Generation
Enforcing API Design Rules for High Quality Code GenerationTim Burks
 
Deployment Patterns in the Ruby on Rails World
Deployment Patterns in the Ruby on Rails WorldDeployment Patterns in the Ruby on Rails World
Deployment Patterns in the Ruby on Rails WorldNikhil Mungel
 
One commit, one release. Continuously delivering a Symfony project.
One commit, one release. Continuously delivering a Symfony project.One commit, one release. Continuously delivering a Symfony project.
One commit, one release. Continuously delivering a Symfony project.Javier López
 
Continous Delivering a PHP application
Continous Delivering a PHP applicationContinous Delivering a PHP application
Continous Delivering a PHP applicationJavier López
 
Architecting the Future: Abstractions and Metadata - KCDC
Architecting the Future: Abstractions and Metadata - KCDCArchitecting the Future: Abstractions and Metadata - KCDC
Architecting the Future: Abstractions and Metadata - KCDCDaniel Barker
 
DevOps Fest 2019. Gianluca Arbezzano. DevOps never sleeps. What we learned fr...
DevOps Fest 2019. Gianluca Arbezzano. DevOps never sleeps. What we learned fr...DevOps Fest 2019. Gianluca Arbezzano. DevOps never sleeps. What we learned fr...
DevOps Fest 2019. Gianluca Arbezzano. DevOps never sleeps. What we learned fr...DevOps_Fest
 
Refactoring a go project
Refactoring a go projectRefactoring a go project
Refactoring a go projectDan Tran
 

What's hot (19)

Essential Tools for Modern PHP
Essential Tools for Modern PHPEssential Tools for Modern PHP
Essential Tools for Modern PHP
 
Fast and Reliable Swift APIs with gRPC
Fast and Reliable Swift APIs with gRPCFast and Reliable Swift APIs with gRPC
Fast and Reliable Swift APIs with gRPC
 
Go 1.8 'new' networking features
Go 1.8 'new' networking featuresGo 1.8 'new' networking features
Go 1.8 'new' networking features
 
Create a PHP Library the right way
Create a PHP Library the right wayCreate a PHP Library the right way
Create a PHP Library the right way
 
Openshift: The power of kubernetes for engineers - Riga Dev Days 18
Openshift: The power of kubernetes for engineers - Riga Dev Days 18Openshift: The power of kubernetes for engineers - Riga Dev Days 18
Openshift: The power of kubernetes for engineers - Riga Dev Days 18
 
Calico docker+ipam
Calico docker+ipamCalico docker+ipam
Calico docker+ipam
 
Red Hat OpenShift Operators - Operators ABC
Red Hat OpenShift Operators - Operators ABCRed Hat OpenShift Operators - Operators ABC
Red Hat OpenShift Operators - Operators ABC
 
DevOps with OpenShift - Fabien Dupont - ManageIQ Design Summit 2016
DevOps with OpenShift - Fabien Dupont - ManageIQ Design Summit 2016DevOps with OpenShift - Fabien Dupont - ManageIQ Design Summit 2016
DevOps with OpenShift - Fabien Dupont - ManageIQ Design Summit 2016
 
kubernetes operators
kubernetes operatorskubernetes operators
kubernetes operators
 
Kubernetes Operators: Rob Szumski
Kubernetes Operators: Rob SzumskiKubernetes Operators: Rob Szumski
Kubernetes Operators: Rob Szumski
 
Meetup Openshift Geneva 03/10
Meetup Openshift Geneva 03/10Meetup Openshift Geneva 03/10
Meetup Openshift Geneva 03/10
 
Ruby and Rails Packaging to Production
Ruby and Rails Packaging to ProductionRuby and Rails Packaging to Production
Ruby and Rails Packaging to Production
 
Enforcing API Design Rules for High Quality Code Generation
Enforcing API Design Rules for High Quality Code GenerationEnforcing API Design Rules for High Quality Code Generation
Enforcing API Design Rules for High Quality Code Generation
 
Deployment Patterns in the Ruby on Rails World
Deployment Patterns in the Ruby on Rails WorldDeployment Patterns in the Ruby on Rails World
Deployment Patterns in the Ruby on Rails World
 
One commit, one release. Continuously delivering a Symfony project.
One commit, one release. Continuously delivering a Symfony project.One commit, one release. Continuously delivering a Symfony project.
One commit, one release. Continuously delivering a Symfony project.
 
Continous Delivering a PHP application
Continous Delivering a PHP applicationContinous Delivering a PHP application
Continous Delivering a PHP application
 
Architecting the Future: Abstractions and Metadata - KCDC
Architecting the Future: Abstractions and Metadata - KCDCArchitecting the Future: Abstractions and Metadata - KCDC
Architecting the Future: Abstractions and Metadata - KCDC
 
DevOps Fest 2019. Gianluca Arbezzano. DevOps never sleeps. What we learned fr...
DevOps Fest 2019. Gianluca Arbezzano. DevOps never sleeps. What we learned fr...DevOps Fest 2019. Gianluca Arbezzano. DevOps never sleeps. What we learned fr...
DevOps Fest 2019. Gianluca Arbezzano. DevOps never sleeps. What we learned fr...
 
Refactoring a go project
Refactoring a go projectRefactoring a go project
Refactoring a go project
 

Similar to Habitat 301: Building Habitats

Xamarin Form using ASP.NET Core SignalR client
Xamarin Form using ASP.NET Core SignalR clientXamarin Form using ASP.NET Core SignalR client
Xamarin Form using ASP.NET Core SignalR clientChen Yu Pao
 
#CNX14 - Dive Deep into the ExactTarget Fuel APIs
#CNX14 - Dive Deep into the ExactTarget Fuel APIs#CNX14 - Dive Deep into the ExactTarget Fuel APIs
#CNX14 - Dive Deep into the ExactTarget Fuel APIsSalesforce Marketing Cloud
 
Fullstack workshop
Fullstack workshopFullstack workshop
Fullstack workshopAssaf Gannon
 
Azure handsonlab
Azure handsonlabAzure handsonlab
Azure handsonlabChef
 
Compliance Automation with Inspec Part 4
Compliance Automation with Inspec Part 4Compliance Automation with Inspec Part 4
Compliance Automation with Inspec Part 4Chef
 
Command Line Tool in swift
Command Line Tool in swiftCommand Line Tool in swift
Command Line Tool in swiftYusuke Kita
 
Integrate any Angular Project into WebSphere Portal
Integrate any Angular Project into WebSphere PortalIntegrate any Angular Project into WebSphere Portal
Integrate any Angular Project into WebSphere PortalHimanshu Mendiratta
 
Spring and Pivotal Application Service - SpringOne Tour - Boston
Spring and Pivotal Application Service - SpringOne Tour - BostonSpring and Pivotal Application Service - SpringOne Tour - Boston
Spring and Pivotal Application Service - SpringOne Tour - BostonVMware Tanzu
 
From ZERO to REST in an hour
From ZERO to REST in an hour From ZERO to REST in an hour
From ZERO to REST in an hour Cisco DevNet
 
Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example
Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example
Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example Anna Klepacka
 
Running gRPC Services for Serving Legacy API on Kubernetes
Running gRPC Services for Serving Legacy API on KubernetesRunning gRPC Services for Serving Legacy API on Kubernetes
Running gRPC Services for Serving Legacy API on KubernetesSungwon Lee
 
Real time Communication with Signalr (Android Client)
Real time Communication with Signalr (Android Client)Real time Communication with Signalr (Android Client)
Real time Communication with Signalr (Android Client)Deepak Gupta
 
RoR Workshop - Web applications hacking - Ruby on Rails example
RoR Workshop - Web applications hacking - Ruby on Rails exampleRoR Workshop - Web applications hacking - Ruby on Rails example
RoR Workshop - Web applications hacking - Ruby on Rails exampleRailwaymen
 
Introduction to google endpoints
Introduction to google endpointsIntroduction to google endpoints
Introduction to google endpointsShinto Anto
 
CIP Developing Curator Tool Wizards
CIP Developing Curator Tool WizardsCIP Developing Curator Tool Wizards
CIP Developing Curator Tool WizardsEdwin Rojas
 
Php Asp Net Interoperability Rc Jao
Php Asp Net Interoperability Rc JaoPhp Asp Net Interoperability Rc Jao
Php Asp Net Interoperability Rc Jaojedt
 
.NET Core Today and Tomorrow
.NET Core Today and Tomorrow.NET Core Today and Tomorrow
.NET Core Today and TomorrowJon Galloway
 
harePoint Framework Webinar Series: Consume Graph APIs in SharePoint Framework
harePoint Framework Webinar Series: Consume Graph APIs in SharePoint FrameworkharePoint Framework Webinar Series: Consume Graph APIs in SharePoint Framework
harePoint Framework Webinar Series: Consume Graph APIs in SharePoint FrameworkJenkins NS
 

Similar to Habitat 301: Building Habitats (20)

Xamarin Form using ASP.NET Core SignalR client
Xamarin Form using ASP.NET Core SignalR clientXamarin Form using ASP.NET Core SignalR client
Xamarin Form using ASP.NET Core SignalR client
 
flask.pptx
flask.pptxflask.pptx
flask.pptx
 
#CNX14 - Dive Deep into the ExactTarget Fuel APIs
#CNX14 - Dive Deep into the ExactTarget Fuel APIs#CNX14 - Dive Deep into the ExactTarget Fuel APIs
#CNX14 - Dive Deep into the ExactTarget Fuel APIs
 
Fullstack workshop
Fullstack workshopFullstack workshop
Fullstack workshop
 
Azure handsonlab
Azure handsonlabAzure handsonlab
Azure handsonlab
 
Compliance Automation with Inspec Part 4
Compliance Automation with Inspec Part 4Compliance Automation with Inspec Part 4
Compliance Automation with Inspec Part 4
 
Command Line Tool in swift
Command Line Tool in swiftCommand Line Tool in swift
Command Line Tool in swift
 
Integrate any Angular Project into WebSphere Portal
Integrate any Angular Project into WebSphere PortalIntegrate any Angular Project into WebSphere Portal
Integrate any Angular Project into WebSphere Portal
 
Spring and Pivotal Application Service - SpringOne Tour - Boston
Spring and Pivotal Application Service - SpringOne Tour - BostonSpring and Pivotal Application Service - SpringOne Tour - Boston
Spring and Pivotal Application Service - SpringOne Tour - Boston
 
From ZERO to REST in an hour
From ZERO to REST in an hour From ZERO to REST in an hour
From ZERO to REST in an hour
 
Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example
Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example
Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example
 
Intro to CakePHP
Intro to CakePHPIntro to CakePHP
Intro to CakePHP
 
Running gRPC Services for Serving Legacy API on Kubernetes
Running gRPC Services for Serving Legacy API on KubernetesRunning gRPC Services for Serving Legacy API on Kubernetes
Running gRPC Services for Serving Legacy API on Kubernetes
 
Real time Communication with Signalr (Android Client)
Real time Communication with Signalr (Android Client)Real time Communication with Signalr (Android Client)
Real time Communication with Signalr (Android Client)
 
RoR Workshop - Web applications hacking - Ruby on Rails example
RoR Workshop - Web applications hacking - Ruby on Rails exampleRoR Workshop - Web applications hacking - Ruby on Rails example
RoR Workshop - Web applications hacking - Ruby on Rails example
 
Introduction to google endpoints
Introduction to google endpointsIntroduction to google endpoints
Introduction to google endpoints
 
CIP Developing Curator Tool Wizards
CIP Developing Curator Tool WizardsCIP Developing Curator Tool Wizards
CIP Developing Curator Tool Wizards
 
Php Asp Net Interoperability Rc Jao
Php Asp Net Interoperability Rc JaoPhp Asp Net Interoperability Rc Jao
Php Asp Net Interoperability Rc Jao
 
.NET Core Today and Tomorrow
.NET Core Today and Tomorrow.NET Core Today and Tomorrow
.NET Core Today and Tomorrow
 
harePoint Framework Webinar Series: Consume Graph APIs in SharePoint Framework
harePoint Framework Webinar Series: Consume Graph APIs in SharePoint FrameworkharePoint Framework Webinar Series: Consume Graph APIs in SharePoint Framework
harePoint Framework Webinar Series: Consume Graph APIs in SharePoint Framework
 

More from Jamie Winsor

Elixir Into Production
Elixir Into ProductionElixir Into Production
Elixir Into ProductionJamie Winsor
 
Stay Awhile And Fiddle With Your Smartphone
Stay Awhile And Fiddle With Your SmartphoneStay Awhile And Fiddle With Your Smartphone
Stay Awhile And Fiddle With Your SmartphoneJamie Winsor
 
Building And Releasing A Massively Multiplayer Online Game
Building And Releasing A Massively Multiplayer Online GameBuilding And Releasing A Massively Multiplayer Online Game
Building And Releasing A Massively Multiplayer Online GameJamie Winsor
 

More from Jamie Winsor (6)

Elixir Into Production
Elixir Into ProductionElixir Into Production
Elixir Into Production
 
Stay Awhile And Fiddle With Your Smartphone
Stay Awhile And Fiddle With Your SmartphoneStay Awhile And Fiddle With Your Smartphone
Stay Awhile And Fiddle With Your Smartphone
 
Building And Releasing A Massively Multiplayer Online Game
Building And Releasing A Massively Multiplayer Online GameBuilding And Releasing A Massively Multiplayer Online Game
Building And Releasing A Massively Multiplayer Online Game
 
Atmosphere 2014
Atmosphere 2014Atmosphere 2014
Atmosphere 2014
 
Chef conf-2014
Chef conf-2014Chef conf-2014
Chef conf-2014
 
The Berkshelf Way
The Berkshelf WayThe Berkshelf Way
The Berkshelf Way
 

Recently uploaded

Defining the Clouds for entriprises.pptx
Defining the Clouds for entriprises.pptxDefining the Clouds for entriprises.pptx
Defining the Clouds for entriprises.pptxAshwiniTodkar4
 
Overview of IS 16700:2023 (by priyansh verma)
Overview of IS 16700:2023 (by priyansh verma)Overview of IS 16700:2023 (by priyansh verma)
Overview of IS 16700:2023 (by priyansh verma)Priyansh
 
Madani.store - Planning - Interview Questions
Madani.store - Planning - Interview QuestionsMadani.store - Planning - Interview Questions
Madani.store - Planning - Interview QuestionsKarim Gaber
 
Novel 3D-Printed Soft Linear and Bending Actuators
Novel 3D-Printed Soft Linear and Bending ActuatorsNovel 3D-Printed Soft Linear and Bending Actuators
Novel 3D-Printed Soft Linear and Bending ActuatorsResearcher Researcher
 
Network Enhancements on BitVisor for BitVisor Summit 12
Network Enhancements on BitVisor for BitVisor Summit 12Network Enhancements on BitVisor for BitVisor Summit 12
Network Enhancements on BitVisor for BitVisor Summit 12cjchen22
 
Submerged Combustion, Explosion Flame Combustion, Pulsating Combustion, and E...
Submerged Combustion, Explosion Flame Combustion, Pulsating Combustion, and E...Submerged Combustion, Explosion Flame Combustion, Pulsating Combustion, and E...
Submerged Combustion, Explosion Flame Combustion, Pulsating Combustion, and E...Ayisha586983
 
A brief look at visionOS - How to develop app on Apple's Vision Pro
A brief look at visionOS - How to develop app on Apple's Vision ProA brief look at visionOS - How to develop app on Apple's Vision Pro
A brief look at visionOS - How to develop app on Apple's Vision ProRay Yuan Liu
 
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.elesangwon
 
Road Development plans of India-Transportation Engineering
Road Development plans of India-Transportation EngineeringRoad Development plans of India-Transportation Engineering
Road Development plans of India-Transportation EngineeringSayli Madhale
 
Chapter 9 Mechanical Injection Systems.pdf
Chapter 9 Mechanical Injection Systems.pdfChapter 9 Mechanical Injection Systems.pdf
Chapter 9 Mechanical Injection Systems.pdfFaizanAhmed396943
 
AI Powered Ecover creator that creates everything that you want
AI Powered Ecover creator that creates everything that you wantAI Powered Ecover creator that creates everything that you want
AI Powered Ecover creator that creates everything that you wantsuja868966
 
Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...
Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...
Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...IJAEMSJORNAL
 
WAVELET SCATTERING TRANSFORM FOR ECG CARDIOVASCULAR DISEASE CLASSIFICATION
WAVELET SCATTERING TRANSFORM FOR ECG CARDIOVASCULAR DISEASE CLASSIFICATIONWAVELET SCATTERING TRANSFORM FOR ECG CARDIOVASCULAR DISEASE CLASSIFICATION
WAVELET SCATTERING TRANSFORM FOR ECG CARDIOVASCULAR DISEASE CLASSIFICATIONgerogepatton
 
Ergodomus - LOD 400 Production Drawings Exampes - Copy.pdf
Ergodomus - LOD 400 Production Drawings Exampes - Copy.pdfErgodomus - LOD 400 Production Drawings Exampes - Copy.pdf
Ergodomus - LOD 400 Production Drawings Exampes - Copy.pdfgestioneergodomus
 
Introduction of Object Oriented Programming Language using Java. .pptx
Introduction of Object Oriented Programming Language using Java. .pptxIntroduction of Object Oriented Programming Language using Java. .pptx
Introduction of Object Oriented Programming Language using Java. .pptxPoonam60376
 
LM7_ Embedded Sql and Dynamic SQL in dbms
LM7_ Embedded Sql and Dynamic SQL in dbmsLM7_ Embedded Sql and Dynamic SQL in dbms
LM7_ Embedded Sql and Dynamic SQL in dbmsBalaKrish12
 
Madani.store - Planning - Interview Questions
Madani.store - Planning - Interview QuestionsMadani.store - Planning - Interview Questions
Madani.store - Planning - Interview QuestionsKarim Gaber
 
22CYT12 & Chemistry for Computer Systems_Unit-II-Corrosion & its Control Meth...
22CYT12 & Chemistry for Computer Systems_Unit-II-Corrosion & its Control Meth...22CYT12 & Chemistry for Computer Systems_Unit-II-Corrosion & its Control Meth...
22CYT12 & Chemistry for Computer Systems_Unit-II-Corrosion & its Control Meth...KrishnaveniKrishnara1
 

Recently uploaded (20)

Defining the Clouds for entriprises.pptx
Defining the Clouds for entriprises.pptxDefining the Clouds for entriprises.pptx
Defining the Clouds for entriprises.pptx
 
Overview of IS 16700:2023 (by priyansh verma)
Overview of IS 16700:2023 (by priyansh verma)Overview of IS 16700:2023 (by priyansh verma)
Overview of IS 16700:2023 (by priyansh verma)
 
Madani.store - Planning - Interview Questions
Madani.store - Planning - Interview QuestionsMadani.store - Planning - Interview Questions
Madani.store - Planning - Interview Questions
 
Novel 3D-Printed Soft Linear and Bending Actuators
Novel 3D-Printed Soft Linear and Bending ActuatorsNovel 3D-Printed Soft Linear and Bending Actuators
Novel 3D-Printed Soft Linear and Bending Actuators
 
Network Enhancements on BitVisor for BitVisor Summit 12
Network Enhancements on BitVisor for BitVisor Summit 12Network Enhancements on BitVisor for BitVisor Summit 12
Network Enhancements on BitVisor for BitVisor Summit 12
 
Submerged Combustion, Explosion Flame Combustion, Pulsating Combustion, and E...
Submerged Combustion, Explosion Flame Combustion, Pulsating Combustion, and E...Submerged Combustion, Explosion Flame Combustion, Pulsating Combustion, and E...
Submerged Combustion, Explosion Flame Combustion, Pulsating Combustion, and E...
 
A brief look at visionOS - How to develop app on Apple's Vision Pro
A brief look at visionOS - How to develop app on Apple's Vision ProA brief look at visionOS - How to develop app on Apple's Vision Pro
A brief look at visionOS - How to develop app on Apple's Vision Pro
 
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
 
Road Development plans of India-Transportation Engineering
Road Development plans of India-Transportation EngineeringRoad Development plans of India-Transportation Engineering
Road Development plans of India-Transportation Engineering
 
Chapter 9 Mechanical Injection Systems.pdf
Chapter 9 Mechanical Injection Systems.pdfChapter 9 Mechanical Injection Systems.pdf
Chapter 9 Mechanical Injection Systems.pdf
 
AI Powered Ecover creator that creates everything that you want
AI Powered Ecover creator that creates everything that you wantAI Powered Ecover creator that creates everything that you want
AI Powered Ecover creator that creates everything that you want
 
Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...
Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...
Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...
 
Versatile Engineering Construction Firms
Versatile Engineering Construction FirmsVersatile Engineering Construction Firms
Versatile Engineering Construction Firms
 
WAVELET SCATTERING TRANSFORM FOR ECG CARDIOVASCULAR DISEASE CLASSIFICATION
WAVELET SCATTERING TRANSFORM FOR ECG CARDIOVASCULAR DISEASE CLASSIFICATIONWAVELET SCATTERING TRANSFORM FOR ECG CARDIOVASCULAR DISEASE CLASSIFICATION
WAVELET SCATTERING TRANSFORM FOR ECG CARDIOVASCULAR DISEASE CLASSIFICATION
 
Ergodomus - LOD 400 Production Drawings Exampes - Copy.pdf
Ergodomus - LOD 400 Production Drawings Exampes - Copy.pdfErgodomus - LOD 400 Production Drawings Exampes - Copy.pdf
Ergodomus - LOD 400 Production Drawings Exampes - Copy.pdf
 
Introduction of Object Oriented Programming Language using Java. .pptx
Introduction of Object Oriented Programming Language using Java. .pptxIntroduction of Object Oriented Programming Language using Java. .pptx
Introduction of Object Oriented Programming Language using Java. .pptx
 
Neometrix Optical Balloon Theodolite.pptx
Neometrix Optical Balloon Theodolite.pptxNeometrix Optical Balloon Theodolite.pptx
Neometrix Optical Balloon Theodolite.pptx
 
LM7_ Embedded Sql and Dynamic SQL in dbms
LM7_ Embedded Sql and Dynamic SQL in dbmsLM7_ Embedded Sql and Dynamic SQL in dbms
LM7_ Embedded Sql and Dynamic SQL in dbms
 
Madani.store - Planning - Interview Questions
Madani.store - Planning - Interview QuestionsMadani.store - Planning - Interview Questions
Madani.store - Planning - Interview Questions
 
22CYT12 & Chemistry for Computer Systems_Unit-II-Corrosion & its Control Meth...
22CYT12 & Chemistry for Computer Systems_Unit-II-Corrosion & its Control Meth...22CYT12 & Chemistry for Computer Systems_Unit-II-Corrosion & its Control Meth...
22CYT12 & Chemistry for Computer Systems_Unit-II-Corrosion & its Control Meth...
 

Habitat 301: Building Habitats

  • 5. Great for infrastructure, Good for applications Chef Conf 2016
  • 6. Application Automation Principles 4 Automation Needs To Live With Your Application 4 Build Failures over Runtime Failures 4 Choreography over Orchestration Chef Conf 2016
  • 7. Atomic, Isolated, Repeatable & Auditable Chef Conf 2016
  • 9. Builder SaaS To Build Your Packages Chef Conf 2016
  • 10. It's already live! 4 Web - https://app.habitat.sh 4 API - https://app.habitat.sh/v1 Chef Conf 2016
  • 11. Why build Builder? 4 Public hosting for packages 4 Public builders for packages 4 History of where a package came from Chef Conf 2016
  • 12. Automatic Re-Build/ Publish On Dependency Update Chef Conf 2016
  • 13. pkg_name=hab-builder-api pkg_origin=core pkg_version=0.7.0 pkg_maintainer="Jamie Winsor <reset@chef.io>" pkg_license=('apachev2') pkg_source=nosuchfile.tar.gz pkg_bin_dirs=(bin) pkg_deps=(core/glibc core/openssl core/coreutils core/gcc-libs core/zeromq core/libsodium core/libarchive) pkg_build_deps=(core/protobuf core/protobuf-rust core/coreutils core/cacerts core/rust core/gcc core/pkg-config core/node core/phantomjs) pkg_expose=(9636) srv_bin="bldr-api" pkg_svc_run="bin/$srv_bin start -c ${pkg_svc_path}/config.toml" Chef Conf 2016
  • 14. Building Builder 4 Educate about Builder 4 For users 4 For collaborators 4 Show how Habitat helped to build Builder Chef Conf 2016
  • 15. It's about to get technical Chef Conf 2016
  • 16. Service Oriented Design 4 Gateway nodes 4 Router nodes 4 Service nodes Chef Conf 2016
  • 18. Why Rust? 4 Systems programming language 4 Fast 4 Memory safety without a virtual machine 4 Thread safety guarantees 4 Great tooling 4 Basically a modern C Chef Conf 2016
  • 19. Walkthrough: Starting a new build Chef Conf 2016
  • 20. Hop 1/10: Client 4 Web Client 4 https://app.habitat.sh 4 Angular 2 4 HTTP Client 4 REST 4 application/json Chef Conf 2016
  • 21. Hop 2/10: Gateway 4 Public facing edge node 4 External authentication/ identification by OAuth/ Authorization header 4 Transform client-request into net-request & forward to router 4 Await reply if transactional and send reply to web-client Chef Conf 2016
  • 22. Authorization 4 Read value of Authorization header into protocol message 4 Forward protocol message to any RouteSrv 4 RouteSrv forward request to appropriate SessionSrv Chef Conf 2016
  • 23. Builder-Protocol/Net Protobuf & ZeroMQ Sockets Chef Conf 2016
  • 24. Protobuf 4 Language agnostic DSL 4 Backwards compatibility between service versions 4 Composable, modular messages message Person { required string name = 1; required int32 id = 2; optional string email = 3; } Chef Conf 2016
  • 25. ZeroMQ 4 Socket library for building distributed services 4 Not a message queue/message bus 4 Makes no assumptions for your protocol 4 Guarantees for multi-part message delivery Chef Conf 2016
  • 26. components/builder-protocol/ protocols/net.proto message Msg { // hint for decoders required string message_id = 1; // serialized msg body required bytes body = 2; // used by RouteSrv to know where to send msg optional RouteInfo route_info = 3; } Chef Conf 2016
  • 27. Authorization (2) // components/builder-protocol/protocols/sessionsrv.proto message SessionGet { required string token = 1; } 4 Create new SessionGet msg 4 Write value of Authorization header into token field 4 Forward SessionGet message to any RouteSrv Chef Conf 2016
  • 28. Hop 3/10: RouteSrv // components/builder-protocol/protocols/net.proto message RouteInfo { // Determines destination service required Protocol protocol = 1; // Determines destination shard optional uint64 hash = 2; } enum Protocol { Net = 0; RouteSrv = 1; SessionSrv = 2; VaultSrv = 3; JobSrv = 4; } 4 Read RouteInfo and forward to appropriate service Chef Conf 2016
  • 29. Hop 4/10: SessionSrv 1. Receive message from RouteSrv 2. Dispatch SessionGet to handler 3. Handler reads token field of message and looks in datastore for session Chef Conf 2016
  • 30. Replying to Request 4 On Success: Session 4 On Failure: NetError message Session { required uint64 id = 1; required string email = 2; required string name = 3; required string token = 4; } Chef Conf 2016
  • 31. NetError: Transaction Failures message NetError { required ErrCode code = 1; required string msg = 2; } enum ErrCode { // other codes... SESSION_EXPIRED = 8; // more codes... } Chef Conf 2016
  • 32. Dissecting The Message Code: 8, Msg: rg:auth:1 4 Code: for the user 4 Msg: for the developer 4 service: rg 4 operation/function: auth 4 case: 1 Chef Conf 2016
  • 33. Hop 5/10: RouteSrv 4 Receive transaction reply from SessionSrv 4 Send response to appropriate HTTP-Gateway Chef Conf 2016
  • 34. Hop 6a/10: Gateway 1. Receive reply from RouteSrv 2. Forward reply to client thread 3. Client thread creates ProjectGet message from HTTP params Chef Conf 2016
  • 35. Hop 6b/10: Gateway // protocols/vaultsrv.proto message ProjectGet { required uint64 id = 1; } 4 Send ProjectGet through RouteSrv to VaultSrv Chef Conf 2016
  • 38. Builder Is Service Oriented & Sharded 4 Service goes down: partial outage for all users 4 Shard goes down: partial outage of service for subset of users 4 Easily add new machines by re-balancing shards Chef Conf 2016
  • 39. Message Routing (cont.) // components/builder-protocol/protocols/net.proto message RouteInfo { // Determines destination service required Protocol protocol = 1; // Determines destination shard optional uint64 hash = 2; } 4 Populate protocol with message's protocol 4 Populate hash with a message's RouteKey Chef Conf 2016
  • 41. Routable /// Defines a contract for protocol messages to be routed through `RouteSrv`. pub trait Routable: protobuf::Message + protobuf::MessageStatic { /// Type of the route key type H: RouteKey + fmt::Display; /// Return a `RouteKey` for `RouteSrv` to know which key's /// value to route on. /// /// If `Some(T)`, the message will be routed by hashing the /// value of the route key and modding it against the shard /// count. This is known as "randomly deterministic routing". /// /// If `None`, the message will be randomly routed to an available node. fn route_key(&self) -> Option<Self::H>; } Chef Conf 2016
  • 42. Protobuf Definition message ProjectGet { required uint64 id = 1; } Trait Implementation impl Routable for ProjectGet { type H = InstaId; fn route_key(&self) -> Option<Self::H> { Some(InstaId(self.get_id())) } } Chef Conf 2016
  • 43. InstaId 4 64-bit integer containing 4 create-time 4 sequence-id (generated by database, 0-1024) 4 shard (0-127) 4 Inspired by Instagram engineering1 1 http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram Chef Conf 2016
  • 44. pub const EPOCH_MS: u64 = 1460499133628; pub const SHARD_COUNT: u32 = 128; pub struct InstaId(pub u64); impl InstaId { pub fn generate(auto_id: u64) -> Self { let time = Self::since_epoch(); let id = auto_id % 1024; let shard_id = id % SHARD_COUNT as u64; let mut iid: u64 = time << 23; iid |= id << 13; iid |= shard_id; InstaId(iid) } pub fn since_epoch() -> u64 { let timespec = time::get_time(); let sec: u64 = timespec.sec as u64 * 1000; let nsec: u64 = timespec.nsec as u64 / 1000 / 1000; (sec + nsec) - EPOCH_MS } } Chef Conf 2016
  • 45. Hop 7/10: VaultSrv 1. Receive message from RouteSrv 2. Dispatch ProjectGet to handler 3. Read Project from database 4. Reply to HTTP-Gateway through RouteSrv Chef Conf 2016
  • 46. What is VaultSrv? 4 Persistent storage for 4 Origin memberships 4 Origin invitations 4 Projects Chef Conf 2016
  • 47. Persisting Protobufs Into The Database Chef Conf 2016
  • 48. Persistable /// Defines a contract for protocol messages to be persisted to a datastore. pub trait Persistable: protobuf::Message + protobuf::MessageStatic { /// Type of the primary key type Key: fmt::Display; /// Returns the value of the primary key. fn primary_key(&self) -> Self::Key; /// Sets the primary key to the given value. fn set_primary_key(&mut self, value: Self::Key) -> (); } Chef Conf 2016
  • 49. Protobuf Definition message Project { required uint64 id = 1; required uint64 owner_id = 2; required uint64 origin_id = 3; required string plan_path = 4; oneof vcs { VCSGit git = 5; } } Trait Implementation impl Persistable for Project { type Key = u64; fn primary_key(&self) -> Self::Key { self.get_id() } fn set_primary_key(&mut self, value: Self::Key) { self.set_id(value); } } Chef Conf 2016
  • 50. Hop 8/10: REST- Gateway 1. Receive reply from RouteSrv 2. Create new JobSpec from Session & Project 3. Send JobSpec through RouteSrv to JobSrv message JobSpec { required uint64 owner_id = 1; required vault.Project project = 2; } Chef Conf 2016
  • 51. Hop 9/10: JobSrv 1. Receive message from RouteSrv 2. Dispatch JobSpec to job_create handler 3. Create Job from JobSpec 4. Persist Job to database/queue Chef Conf 2016
  • 52. What is JobSrv? 4 Store job history 4 Job is an entity created by a project 4 Active jobs are placed in a queue 4 Different qeueues for different OSes 4 Job create success/failure Chef Conf 2016
  • 53. Hop 9b/10: JobSrv 1. Send Job through RouteSrv to HTTP-Gateway message Job { required uint64 id = 1; required uint64 owner_id = 2; required JobState state = 3; required vault.Project project = 4; } enum JobState { Pending = 0; Processing = 1; Complete = 2; Rejected = 3; Failed = 4; } Chef Conf 2016
  • 54. Hop 10/10: REST- Gateway 1. Receive reply from RouteSrv 2. Forward reply to client thread 3. Client thread serialize Job into JSON 4. Send 200 to client with encoded Job Chef Conf 2016
  • 55. JSON Trait Implementation impl ToJson for Job { fn to_json(&self) -> Json { let mut m = BTreeMap::new(); m.insert("id".to_string(), self.get_id().to_json()); m.insert("state".to_string(), self.get_state().value().to_json()); Json::Object(m) } } JSON Response { "id": 62214184531664897, "state": 0, } Chef Conf 2016
  • 58. Workers 4 (N) workers connected to (X) JobSrv 4 OS specific 4 Pops a message off a JobSrv queue 4 Builds are done within a studio on the worker 4 Finished builds published to public depot as project's package ident Chef Conf 2016
  • 59. Habitat Builder Is Self Hosted 4 Habitat supervises all processes 4 Supervisor watches public depot (self) for package updates 4 Builder builds it's own updates 4 Automatically updates when a build is published to public depot Chef Conf 2016
  • 63. Builder Uses Native Habitat Config [cfg] http_addr = "0.0.0.0:9636" [cfg.github] client_id = "{clientid}" client_secret = "{clientsecret}" [[bind.router.members]] alive = true ip = "10.0.0.190" port = "5562" service = "hab-builder-router" Chef Conf 2016
  • 65. Director systemd [Unit] Description=Habitat Director [Service] ExecStart=/bin/hab-director start -c /hab/etc/director/config.toml [Install] WantedBy=default.target Chef Conf 2016
  • 66. Director Config [cfg.services.core.redis.${env}] start = "--permanent-peer --peer ${peer_ip}:9000" [cfg.services.core.hab-builder-api.${env}] start = "--permanent-peer --bind database:redis.${env},router:hab-builder-router.${env} --peer ${peer_ip}:9000" Chef Conf 2016
  • 67. Bind Is Service Discovery Chef Conf 2016
  • 69. Electing A Master Router 4 Automatically elected and configured by Habitat through the --topology leader flag 4 Configured to expect (N) service nodes 4 Gives out shard assignments on service connect Chef Conf 2016
  • 70. Bring Your Own Worker Chef Conf 2016
  • 71. Habitat Hack Day Thursday, 10AM-3PM Chef Conf 2016