Jamie Winsor's Chef Conf 2016 talk on Habitat's upcoming distributed bulid system and how it, itself, was built with Habitat

1. Habitat 301: Building
Habitats
Chef Conf 2016

2. Jamie Winsor
@resetexistence
github.com/reset
Chef Conf 2016

3. NEW SLIDE HERE
Chef Conf 2016

4. Chef Conf 2016

5. Great for infrastructure,
Good for applications
Chef Conf 2016

6. Application Automation Principles
4 Automation Needs To Live With Your Application
4 Build Failures over Runtime Failures
4 Choreography over Orchestration
Chef Conf 2016

7. Atomic, Isolated,
Repeatable & Auditable
Chef Conf 2016

8. Chef Conf 2016

9. Builder
SaaS To Build Your Packages
Chef Conf 2016

10. It's already live!
4 Web - https://app.habitat.sh
4 API - https://app.habitat.sh/v1
Chef Conf 2016

11. Why build Builder?
4 Public hosting for packages
4 Public builders for packages
4 History of where a package came from
Chef Conf 2016

12. Automatic Re-Build/
Publish On Dependency
Update
Chef Conf 2016

13. pkg_name=hab-builder-api
pkg_origin=core
pkg_version=0.7.0
pkg_maintainer="Jamie Winsor <reset@chef.io>"
pkg_license=('apachev2')
pkg_source=nosuchfile.tar.gz
pkg_bin_dirs=(bin)
pkg_deps=(core/glibc core/openssl core/coreutils core/gcc-libs
core/zeromq core/libsodium core/libarchive)
pkg_build_deps=(core/protobuf core/protobuf-rust core/coreutils
core/cacerts core/rust core/gcc core/pkg-config
core/node core/phantomjs)
pkg_expose=(9636)
srv_bin="bldr-api"
pkg_svc_run="bin/$srv_bin start -c ${pkg_svc_path}/config.toml"
Chef Conf 2016

14. Building Builder
4 Educate about Builder
4 For users
4 For collaborators
4 Show how Habitat helped to build Builder
Chef Conf 2016

15. It's about to get
technical
Chef Conf 2016

16. Service Oriented Design
4 Gateway nodes
4 Router nodes
4 Service nodes
Chef Conf 2016

17. Service Layout
Chef Conf 2016

18. Why Rust?
4 Systems programming language
4 Fast
4 Memory safety without a
virtual machine
4 Thread safety guarantees
4 Great tooling
4 Basically a modern C
Chef Conf 2016

19. Walkthrough: Starting a
new build
Chef Conf 2016

20. Hop 1/10: Client
4 Web Client
4 https://app.habitat.sh
4 Angular 2
4 HTTP Client
4 REST
4 application/json
Chef Conf 2016

21. Hop 2/10: Gateway
4 Public facing edge node
4 External authentication/
identification by OAuth/
Authorization header
4 Transform client-request into
net-request & forward to
router
4 Await reply if transactional
and send reply to web-client
Chef Conf 2016

22. Authorization
4 Read value of Authorization
header into protocol message
4 Forward protocol message to
any RouteSrv
4 RouteSrv forward request to
appropriate SessionSrv
Chef Conf 2016

23. Builder-Protocol/Net
Protobuf & ZeroMQ Sockets
Chef Conf 2016

24. Protobuf
4 Language agnostic DSL
4 Backwards compatibility between service versions
4 Composable, modular messages
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
}
Chef Conf 2016

25. ZeroMQ
4 Socket library for building distributed services
4 Not a message queue/message bus
4 Makes no assumptions for your protocol
4 Guarantees for multi-part message delivery
Chef Conf 2016

26. components/builder-protocol/
protocols/net.proto
message Msg {
// hint for decoders
required string message_id = 1;
// serialized msg body
required bytes body = 2;
// used by RouteSrv to know where to send msg
optional RouteInfo route_info = 3;
}
Chef Conf 2016

27. Authorization (2)
// components/builder-protocol/protocols/sessionsrv.proto
message SessionGet {
required string token = 1;
}
4 Create new SessionGet msg
4 Write value of Authorization header into token field
4 Forward SessionGet message to any RouteSrv
Chef Conf 2016

28. Hop 3/10: RouteSrv
// components/builder-protocol/protocols/net.proto
message RouteInfo {
// Determines destination service
required Protocol protocol = 1;
// Determines destination shard
optional uint64 hash = 2;
}
enum Protocol {
Net = 0;
RouteSrv = 1;
SessionSrv = 2;
VaultSrv = 3;
JobSrv = 4;
}
4 Read RouteInfo and forward to
appropriate service
Chef Conf 2016

29. Hop 4/10:
SessionSrv
1. Receive message from
RouteSrv
2. Dispatch SessionGet to handler
3. Handler reads token field of
message and looks in datastore
for session
Chef Conf 2016

30. Replying to Request
4 On Success: Session
4 On Failure: NetError
message Session {
required uint64 id = 1;
required string email = 2;
required string name = 3;
required string token = 4;
}
Chef Conf 2016

31. NetError: Transaction Failures
message NetError {
required ErrCode code = 1;
required string msg = 2;
}
enum ErrCode {
// other codes...
SESSION_EXPIRED = 8;
// more codes...
}
Chef Conf 2016

32. Dissecting The Message
Code: 8, Msg: rg:auth:1
4 Code: for the user
4 Msg: for the developer
4 service: rg
4 operation/function: auth
4 case: 1
Chef Conf 2016

33. Hop 5/10: RouteSrv
4 Receive transaction reply from
SessionSrv
4 Send response to appropriate
HTTP-Gateway
Chef Conf 2016

34. Hop 6a/10: Gateway
1. Receive reply from RouteSrv
2. Forward reply to client thread
3. Client thread creates
ProjectGet message from HTTP
params
Chef Conf 2016

35. Hop 6b/10: Gateway
// protocols/vaultsrv.proto
message ProjectGet {
required uint64 id = 1;
}
4 Send ProjectGet through
RouteSrv to VaultSrv
Chef Conf 2016

36. Actual Service Cluster
Chef Conf 2016

37. Chef Conf 2016

38. Builder Is Service Oriented & Sharded
4 Service goes down: partial outage for all users
4 Shard goes down: partial outage of service for
subset of users
4 Easily add new machines by re-balancing shards
Chef Conf 2016

39. Message Routing (cont.)
// components/builder-protocol/protocols/net.proto
message RouteInfo {
// Determines destination service
required Protocol protocol = 1;
// Determines destination shard
optional uint64 hash = 2;
}
4 Populate protocol with message's protocol
4 Populate hash with a message's RouteKey
Chef Conf 2016

40. Defining The RouteKey
Chef Conf 2016

41. Routable
/// Defines a contract for protocol messages to be routed through `RouteSrv`.
pub trait Routable: protobuf::Message + protobuf::MessageStatic {
/// Type of the route key
type H: RouteKey + fmt::Display;
/// Return a `RouteKey` for `RouteSrv` to know which key's
/// value to route on.
///
/// If `Some(T)`, the message will be routed by hashing the
/// value of the route key and modding it against the shard
/// count. This is known as "randomly deterministic routing".
///
/// If `None`, the message will be randomly routed to an available node.
fn route_key(&self) -> Option<Self::H>;
}
Chef Conf 2016

42. Protobuf Definition
message ProjectGet {
required uint64 id = 1;
}
Trait Implementation
impl Routable for ProjectGet {
type H = InstaId;
fn route_key(&self) -> Option<Self::H> {
Some(InstaId(self.get_id()))
}
}
Chef Conf 2016

43. InstaId
4 64-bit integer containing
4 create-time
4 sequence-id (generated by database, 0-1024)
4 shard (0-127)
4 Inspired by Instagram engineering1
1
http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram
Chef Conf 2016

44. pub const EPOCH_MS: u64 = 1460499133628;
pub const SHARD_COUNT: u32 = 128;
pub struct InstaId(pub u64);
impl InstaId {
pub fn generate(auto_id: u64) -> Self {
let time = Self::since_epoch();
let id = auto_id % 1024;
let shard_id = id % SHARD_COUNT as u64;
let mut iid: u64 = time << 23;
iid |= id << 13;
iid |= shard_id;
InstaId(iid)
}
pub fn since_epoch() -> u64 {
let timespec = time::get_time();
let sec: u64 = timespec.sec as u64 * 1000;
let nsec: u64 = timespec.nsec as u64 / 1000 / 1000;
(sec + nsec) - EPOCH_MS
}
}
Chef Conf 2016

45. Hop 7/10: VaultSrv
1. Receive message from
RouteSrv
2. Dispatch ProjectGet to handler
3. Read Project from database
4. Reply to HTTP-Gateway
through RouteSrv
Chef Conf 2016

46. What is VaultSrv?
4 Persistent storage for
4 Origin memberships
4 Origin invitations
4 Projects
Chef Conf 2016

47. Persisting Protobufs Into
The Database
Chef Conf 2016

48. Persistable
/// Defines a contract for protocol messages to be persisted to a datastore.
pub trait Persistable: protobuf::Message + protobuf::MessageStatic {
/// Type of the primary key
type Key: fmt::Display;
/// Returns the value of the primary key.
fn primary_key(&self) -> Self::Key;
/// Sets the primary key to the given value.
fn set_primary_key(&mut self, value: Self::Key) -> ();
}
Chef Conf 2016

49. Protobuf Definition
message Project {
required uint64 id = 1;
required uint64 owner_id = 2;
required uint64 origin_id = 3;
required string plan_path = 4;
oneof vcs {
VCSGit git = 5;
}
}
Trait Implementation
impl Persistable for Project {
type Key = u64;
fn primary_key(&self) -> Self::Key {
self.get_id()
}
fn set_primary_key(&mut self, value: Self::Key) {
self.set_id(value);
}
}
Chef Conf 2016

50. Hop 8/10: REST-
Gateway
1. Receive reply from RouteSrv
2. Create new JobSpec from
Session & Project
3. Send JobSpec through RouteSrv
to JobSrv
message JobSpec {
required uint64 owner_id = 1;
required vault.Project project = 2;
}
Chef Conf 2016

51. Hop 9/10: JobSrv
1. Receive message from
RouteSrv
2. Dispatch JobSpec to job_create
handler
3. Create Job from JobSpec
4. Persist Job to database/queue
Chef Conf 2016

52. What is JobSrv?
4 Store job history
4 Job is an entity created by a
project
4 Active jobs are placed in a
queue
4 Different qeueues for
different OSes
4 Job create success/failure
Chef Conf 2016

53. Hop 9b/10: JobSrv
1. Send Job through RouteSrv to
HTTP-Gateway
message Job {
required uint64 id = 1;
required uint64 owner_id = 2;
required JobState state = 3;
required vault.Project project = 4;
}
enum JobState {
Pending = 0;
Processing = 1;
Complete = 2;
Rejected = 3;
Failed = 4;
}
Chef Conf 2016

54. Hop 10/10: REST-
Gateway
1. Receive reply from RouteSrv
2. Forward reply to client thread
3. Client thread serialize Job into
JSON
4. Send 200 to client with encoded
Job
Chef Conf 2016

55. JSON Trait Implementation
impl ToJson for Job {
fn to_json(&self) -> Json {
let mut m = BTreeMap::new();
m.insert("id".to_string(), self.get_id().to_json());
m.insert("state".to_string(), self.get_state().value().to_json());
Json::Object(m)
}
}
JSON Response
{
"id": 62214184531664897,
"state": 0,
}
Chef Conf 2016

56. Worker Pool 11/10
Chef Conf 2016

57. Chef Conf 2016

58. Workers
4 (N) workers connected to (X) JobSrv
4 OS specific
4 Pops a message off a JobSrv queue
4 Builds are done within a studio on the worker
4 Finished builds published to public depot as project's
package ident
Chef Conf 2016

59. Habitat Builder Is Self Hosted
4 Habitat supervises all processes
4 Supervisor watches public depot (self) for package
updates
4 Builder builds it's own updates
4 Automatically updates when a build is published to
public depot
Chef Conf 2016

60. Chef Conf 2016

61. Chef Conf 2016

62. Where Is config/default.toml?
Chef Conf 2016

63. Builder Uses Native Habitat Config
[cfg]
http_addr = "0.0.0.0:9636"
[cfg.github]
client_id = "{clientid}"
client_secret = "{clientsecret}"
[[bind.router.members]]
alive = true
ip = "10.0.0.190"
port = "5562"
service = "hab-builder-router"
Chef Conf 2016

64. Director
Chef Conf 2016

65. Director systemd
[Unit]
Description=Habitat Director
[Service]
ExecStart=/bin/hab-director start -c /hab/etc/director/config.toml
[Install]
WantedBy=default.target
Chef Conf 2016

66. Director Config
[cfg.services.core.redis.${env}]
start = "--permanent-peer --peer ${peer_ip}:9000"
[cfg.services.core.hab-builder-api.${env}]
start = "--permanent-peer --bind database:redis.${env},router:hab-builder-router.${env}
--peer ${peer_ip}:9000"
Chef Conf 2016

67. Bind Is Service Discovery
Chef Conf 2016

68. Shard Allocation
Chef Conf 2016

69. Electing A Master Router
4 Automatically elected and configured by Habitat
through the --topology leader flag
4 Configured to expect (N) service nodes
4 Gives out shard assignments on service connect
Chef Conf 2016

70. Bring Your Own Worker
Chef Conf 2016

71. Habitat Hack Day
Thursday, 10AM-3PM
Chef Conf 2016