Vertically Scaled Design Patters

Vertically Scaled Design
Patterns
Jeff Malnick & Paul Ambrosini
Systems Engineering @ SRC:CLR
srcclr.com | @srcclr

How this is going down:
● History & Evolution
● Principles
● Notes & Examples
● Demo

History & Evolution

In the beginning there was
● AWS
● ELB
● Ansible
● SCP
● Tomcat
● We all start somewhere

Then there was

Paul heroically rolled it out in 2 weeks

Massive Gains
● Huge improvement
● Split up Frontend and Backend Code
● Teams deploying separately
● Productivity went up dramatically

Wasn’t perfect
● Not fully dynamic
● CI hooks not perfect
● Cookbook merging*
● Service health management lacking

Then we migrated...
● *hired Jeff
○ ‘ops role’
○ or ‘devops’

We got a shiny new CM system.

"Type" : "AWS::EC2::Instance",
"Properties" : {
"KeyName" : "secure",
"SecurityGroupIds" : ["sg-1234abcd","sg-5678efg"],
"SubnetId" : "subnet-1234abcd",
"UserData" : {
"Fn::Base64" : {
"Fn::Join" : ["", [
"#!/bin/sh -vn",
"ROLE='qa_nginx_static'n",
"MASTER='https://puppet.srcclr.com'n",
"D_ROLE=$(echo $ROLE | tr '_' '-')n",
"D_IP=$(hostname -I | tr '.' '-' | cut -d' ' -f1)n",
"HOSTNAME="$(echo $D_ROLE)-$(echo $D_IP)"n",
"echo "127.0.0.1 $HOSTNAME" >> /etc/hosts",
"apt-get updaten",
"echo $HOSTNAME > /etc/hostnamen",
"/bin/hostname $HOSTNAMEn",
"echo $ROLE > /etc/rolen",
"sudo curl -k $MASTER:8140/packages/current/install.bash | sudo bashn"
]]}
Cloud Formation for Bare Metal

Continuous Integration

# Executed in $WORKSPACE on jenkins
VERSION=$(some_mvn_command)
curl
-X POST
-d@-
puppet.myorg.com:1015/deploy <<EOF
{
"service": "analytics",
"version": "$VERSION",
"environment": "qa",
"role": "qa_service"
}
EOF
How we do CI

New CM was better but...

Shameless use of hooks
include http
http::listener {‘kick_haproxy’:
port => '1234',
routes => {
'kick_haproxy' => {
'method' => 'get',
'command' => "su - peadmin -c 'mco puppet runonce -F
role == haproxy_internal'"
},

“Service Discovery”
http { 'kick_haproxy':
ensure => get,
port => '1234',
fqdn => 'puppet.myorg.com'
}

sc_services::service { 'librarian':
deploy_stage => 'production',
version => $version,
enable_newrelic_apm => true,
loadbalancer_role => 'production_internal',
xmx_setting => '1024',
xms_setting => '1024',
proc_opts => {
'1' => {
'env_profile' => 'prod',
'port' => '11100',
'mgmt_port' => '11110',
'override' => $properties_file,
},
'2' => {
'env_profile' => 'prod',
'port' => '11101',
'mgmt_port' => '11111',
}
}
Still static...

deploy_stage => 'app',
proc_opts => {
'1' => {
'env_profile' => 'app',
'port' => '11100',
'mgmt_port' => '11110',
},
'2' => {
'port' => '11101',
'mgmt_port' => '11111',
}
}
Still static...
{Statically
assigning ports!

proc_opts => {
'1' => {
'port' => '11100',
'mgmt_port' => '11110',
},
'2' => {
'port' => '11101',
'mgmt_port' => '11111',
}
}
Still static...
What S3 Bucket
is this Jar from?

proc_opts => {
'1' => {
'port' => '11100',
'mgmt_port' => '11110',
},
'2' => {
'port' => '11101',
'mgmt_port' => '11111',
}
}
Still static...
“Service Discovery”
for HaProxy

proc_opts => {
'1' => {
'port' => '11100',
'mgmt_port' => '11110',
},
'2' => {
'port' => '11101',
'mgmt_port' => '11111',
}
}
Still static...
Service
statically
scheduled to
execute on a
node classified
by Puppet

“Scaling”
$ports = {
'librarian' => {
'svc' => '20060-20070',
'mgmt' => '20071-20080',
},
'notifications' => {
'svc' => '20000-20010',
'mgmt' => '20011-20020',
},
'cloud_agent' => {
'svc' => '31040-31050',
'mgmt' => '31051-31060',
},
'search' => {
'svc' => '41140-41150',
'mgmt' => '41151-41160',
},

“Scaling”
$ports = {
'librarian' => {
'svc' => '20060-20070',
'mgmt' => '20071-20080',
},
'notifications' => {
'svc' => '20000-20010',
'mgmt' => '20011-20020',
},
'cloud_agent' => {
'svc' => '31040-31050',
'mgmt' => '31051-31060',
},
'search' => {
'svc' => '41140-41150',
'mgmt' => '41151-41160',
}
...
{ Scaling to maximum of 10
instances per box.

Recap
● CM statically schedules services to run on node
● Can scale to maximum number of known ports
● This does “scale” vertically & horizontally...

A Standard Factory...
● Bare Metal Provisioning
● Configuration Management
● Remote Execution Framework

The Monolith…

As a framework it is:
● Hard to diagnose what is broken
● One part breaks, the entire thing breaks
● Unwieldy code bases
● Works for some, wasn’t for us

The Monolith Factory
● Statically schedules services
● Doesn’t require service discovery

The Micro-Service

As a framework it is:
● Atomic services
● Easy(er) to figure out what is broken
● Scale or add individual pieces

The Micro-Services Factory
● Dynamic task scheduling
● Service discovery required

The Scaling Problem...
● Service ‘x’ requires known port assignments
● Service ‘x’ configuration changes across env’s
● Service ‘x’ should be able to run on a box with
other instances of Service ‘x’ as well as Service ‘y’
and Service ‘z’

The Vertical Component
● Start locally, on a single box
● Find a solution that works vertically, master it
● Once the vertical solution is found, move
horizontally

Dynamic Task Scheduling
● Purpose built remote execution framework

What we really need...
● Purpose built remote execution framework
● … micro services are like streakers…

They don’t care where they’re running, they just want
to be exposed.
● Dynamic tasks scheduling
● … micro services are like streaker`

What we really need...
● Purpose built systems
● … CM does CM; SD does SD; divide and conquer

Configuration Management
● Is still important!
● Is still necessary!

Our Remote Execution Framework

Our Initial Deployment
● Apache Mesos
● Marathon

"Type" : "AWS::EC2::Instance",
"Properties" : {
"KeyName" : "secure",
"SecurityGroupIds" : ["sg-1234abcd","sg-5678efg"],
"SubnetId" : "subnet-1234abcd",
"UserData" : {
"Fn::Base64" : {
"Fn::Join" : ["", [
"#!/bin/sh -vn",
"ROLE='qa_nginx_static'n",
"MASTER='https://puppet.srcclr.com'n",
"D_ROLE=$(echo $ROLE | tr '_' '-')n",
"D_IP=$(hostname -I | tr '.' '-' | cut -d' ' -f1)n",
"HOSTNAME="$(echo $D_ROLE)-$(echo $D_IP)"n",
"echo "127.0.0.1 $HOSTNAME" >> /etc/hosts",
"apt-get updaten",
"echo $HOSTNAME > /etc/hostnamen",
"/bin/hostname $HOSTNAMEn",
"echo $ROLE > /etc/rolen",
"sudo curl -k $MASTER:8140/packages/current/install.bash | sudo bashn"
]]}
Design Pattern : Bare Metal

class roles::mesos_master {
include ::profiles::mesos::basic
include ::profiles::mesos::master
include ::profiles::docker::marathon
include ::profiles::haproxy::marathon_template
}
Design Pattern : Config Mgmt

Design Pattern : Application Layer
Mesos for Layer 7:
- Deploy application layer via Docker
- Ensure service ‘x’ is running ‘n’ number of
times

{
"container": {
"type": "DOCKER",
"docker": {
"forcePullImage": true,
"image": "malnick/microbrew",
Design Pattern : Service Discovery
Marathon for Service Discovery/Deployment:
- API to Apache Mesos
- Queried for IP/Port assignments of layer 7
resources to configure haproxy

LIVE DEMO
Are you sure you want to do this?
Pray to the demo gods

Jeff Malnick Paul Ambrosini
malnick at gmail dot com
github.com/malnick
@malnick
technoblogic.io
c at c4 dot vc
paul at srcclr dot com
github.com/cl4r1ty
@cl4r1ty
c4.vc

Vertically Scaled Design Patters

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (15)

Similar to Vertically Scaled Design Patters

Similar to Vertically Scaled Design Patters (20)

More from Jeff Malnick

More from Jeff Malnick (20)

Recently uploaded

Recently uploaded (20)

Vertically Scaled Design Patters