Automating Life in the Cloud - PuppetCamp Chicago '12

Automang Life in the Cloud Joshua Buss & Ma+hew Kemp

There are no Silver Bullets (but if you ﬁnd one let us know) Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 3

Virtual Machines Designing for the Cloud Have to go with what cloud provider oﬀers. Not always ideal for every workload. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 5

Scalability Designing for the Cloud Focus on scaling applica5ons horizontally. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 6

Service Oriented Architecture Applica5on Design Wikipedia Deﬁni5on: SOA as an architecture relies on service-‐orienta3on as its fundamental design principle. If a service presents a simple interface that abstracts away its underlying complexity, users can access independent services without knowledge of the services plaBorm implementa3on. Layman’s terms: A complex system is broken into simple components that are able to interact with each other (and possibly outside sources). Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 7

Case Study: Services Applica5on Design tagserve datahub database Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 8

Service Division of Labor Applica5on Design When should you split services up? Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 9

Backwards Compa5bility Applica5on Design Changes need to be allowed, but compa5bility needs to be maintained. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 10

Asynchronous vs Synchronous Applica5on Design Dependent on needs of business, SLAs and technology. Can introduce more complexity and moving pieces. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 11

Failures Happen Cloud Design Some5mes the failures are outside your control. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 12

Design for Failure Cloud Design Keep failures self contained. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 13

Case Study: Redundancy Applica5on Design Run a full stack in each region. tagserve datahub tagserve datahub database database Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 14

Inter-‐Region Communica5on Applica5on Design Need some data available in all regions, but keep inter-‐region communica5on to a minimum. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 15

Case Study: Cassandra Applica5on Design East West cassandra04 cassandra01 cassandra04 cassandra01 [192-‐255] [0-‐63] [193-‐0] [1-‐64] cassandra03 cassandra02 cassandra03 cassandra02 [128-‐191] [64-‐127] [129-‐192] [65-‐128] Key hashes to 157 Writes go here Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 16

Run5me Controls Applica5on Design Provide mul5ple modes of opera5on. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 17

Smooth Code Pushes Deployment Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 19

Mirror Environment Cutover Deployment Easy migra5ons and upgrade path. Can be more expensive. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 20

Rolling Deploy Deployment More complicated migra5ons and upgrades. Longer deploy window. Usually cheaper. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 21

Case Study: Fabric Rolling Deploy Deployment for region in regions: for app in apps: for server in region: if app on server: maintenance app scp new code to <d_tag> dir symlink app/current to app/<d_tag> restart app wait for healthy Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 22

Fabric vs Puppet Deployment Fabric is push, puppet is pull. Businesses dont move as fast as infrastructure changes, but conﬁgs have to stay up to date all the 5me. (/etc/hosts) (systempoller.py) (mashed_potatoes.env) (dataserver.war) puppet ===================================== fabric (real-‐time up-‐to-‐date) (moderately up-‐to-‐date) (weekly) Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 23

Consistency > * Puppet Need a rock-‐solid founda5on to deploy onto. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 24

Single Puppet Master Puppet Set environment per-‐instance: /etc/puppet/puppet.conf Symlink /etc/puppet/environments/ on master to various folders with read/write access by our main user. $ cd /etc/puppet/environments $ sudo ln –s ~/src/puppet/prod_stable $ sudo ln –s ~/src/puppet/stage_stable $ sudo ln –s ~/src/puppet/dev_test Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 25

Source Controlled Puppet Conﬁgs Puppet Each environment has its own branch. Make a new branch for every new feature. Merge into a test branch to test. Merge into stable. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 26

What is Zerg? Management + = Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 28

How to Reach Servers? Management DNS vs /etc/hosts Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 29

Case Study: /etc/hosts Management $ curl –s http://zerg/etchosts/us-‐west-‐1 # The following lines are desirable for IPv6 capable hosts" ::1 ip6-‐localhost ip6-‐loopback fe00::0 ip6-‐localnet ff00::0 ip6-‐mcastprefix ff02::1 ip6-‐allnodes ff02::2 ip6-‐allrouters ff02::3 ip6-‐allhosts 10.0.0.10 server01 # External: 123.123.123.123 10.0.0.11 server02 # External: 123.123.123.124 10.0.0.12 server03 # External: 123.123.123.125 10.0.0.13 server04 # External: 123.123.123.126 10.0.0.14 server05 # External: 123.123.123.127 10.0.0.15 server06 # External: 123.123.123.128 Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 30

Instance Crea5on Management Amazon API is easy ... but just crea5ng the instance is 10% of the work Gesng the right sotware. Surviving internal API failures. Staying hos5ng provider agnos5c. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 31

Case Study: Instance Crea5on Management ROLE_MAPPING = { "stage" : { "supercloud" : { "ops" : ["zerg"], "awesome" : ["awesome", "haproxy_awesome"], "shabang" : ["shabang", "mashed_potatoes", "haproxy_shabang"], "whistles" : ["gowhooo", "shabang", "thehardproblem", "redis"], "data" : ["dataserver", "dataleaf"], "nosql" : ["itshards", "devnull"], "lb" : ["haproxy"], "redis" : ["redis"], "graph" : ["graphite", "tattle"] }, "prod" : { "evenmoresupercloud" : { ... } } } Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 32

Case Study: Instance Configura5on Management /etc/instanceinfo filled out with values based on Zergs defini5ons: §  hostname §  environment §  region §  roles §  wheres my local zerg server? §  wheres my local graphite server? §  cloud meta info Confiden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 33

Loadbalancer Conﬁgura5on Management Update 5ming tricky to get right. Too important to leave completely autonomous. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 34

HAProxy Configura5on Workflow Management Zerg (genera5on) Git (ops) Script (human) Large changes to templates Git (puppet) (human) Server Server Server Confiden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 35

Applica5on Descrip5ons Management APP_DEFS : { "zerg" : {"type": "http", "healthcheck": {"port": 19999, "resource": "/zerghealth"}}, "awesome" : {"type": "http", "healthcheck": {"port": 20000, "resource": "/ahc"} }, "haproxy_awesome" : {"type": "http", "healthcheck": {"port": 20001, "resource": "/"}}, "shabang" : {"type": "http", "healthcheck": {"port": 20002, "resource": "/"}}, "mashed_potatoes" : {"type": "http", "healthcheck": {"port": 20003, "resource": "/"}}, "haproxy_shabang" : {"type": "http", "healthcheck": {"port": 20004, "resource": "/hc"}}, "gowhooo" : {"type": "http", "healthcheck": {"port": 20005, "resource": "/"}}, "thehardproblem" : {"type": "http", "healthcheck": {"port": 20006, "resource": "/"}}, "redis" : { "type": "tcp", "healthcheck": {"port": 20007, "resource": "/rhc"}}, "dataserver" : { "type": "http", "healthcheck": {"port": 20008, "resource": "/"}}, "itshards" : { "type": "http", "healthcheck": {"port": 20009, "resource": "/"}}, "devnull" : { "type": "http", "healthcheck": {"port": 200010, "resource": "/hc"}}, "graphite" : { "type": "http", "healthcheck": {"port": 80, "resource": "/composer"}} } Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 36

Case Study: HAProxy Conﬁgura5on Management $ curl -‐s http://zerg/haproxy/<env>/<region>/<service> global log 127.0.0.1 local0 log 127.0.0.1 local1 notice stats socket /tmp/haproxy blah blah defaults log global mode http blah blah frontend mashedpotatoes_vip bind *:30000 default_backend data backend mashedpotatoes blah blah options server shabang01 10.0.0.30:30001 check server shabang02 10.0.0.31:30001 check Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 37

The Manual Step Management $ ./update_haproxy.sh <env> <region> <service> ** Git is clean and in sync with origin.. now waiting for zerg http response.. [prod_stable f4qijo] [puppetry] Haproxy Auto-‐Commit for <env> <region> <service> 1 files changed, 2 insertions(+), 2 deletions(-‐) ** Template pulled and committed ** Here is the diff from origin to the new version: diff -‐-‐git a/modules/haproxy/templates/haproxy_<env>_<region>_<service>_cfg.erb b/modules/haproxy/templates/haproxy_<env>_<region>_<service>_cfg.erb -‐-‐-‐ a/modules/haproxy/templates/haproxy_prod_us-‐east-‐1_tagserve_cfg.erb +++ b/modules/haproxy/templates/haproxy_prod_us-‐east-‐1_tagserve_cfg.erb -‐ oldyuckyserver01 -‐ oldyuckyserver02 + fastwonderfulnewserver01 + fastwonderfulnewserver02 ** Do you want to push this change? (y/n) y blah blah successful git push message ** Commit successfully pushed to origin ** All done! Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 38

Trust is a Luxury Best Prac5ces Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 39

Uniform Environments Best Prac5ces Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 40

Why monitor? Monitoring How do you know whats going on? Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 42

What to monitor? Monitoring Iden5fy metrics that act as signals. Add alerts ater every incident. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 43

Data Collec5on Monitoring Need system level metrics and applica5on metrics to get full picture. Everything is in a diﬀerent format. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 44

Case Study: Metric Polling at BrightTag Monitoring datahub datahub redis tagserve redis tagserve mpoller cassandra haproxy cassandra haproxy mpoller mpoller graphite graphite carbon carbon mpoller Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 45

Graphite Monitoring Storage of historical metrics allows for trending and comparisons. Aggrega5on is performed on data retrieval via the webapp. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 46

Branches and Leaves Monitoring Expose a "metrics" service per region. Enables a ﬂexible topology. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 47

Real5me Numbers Across Regions Monitoring Requests are farmed out to each metrics service. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 48

Visualiza5on Monitoring Different visualiza5ons tell you different things. Confiden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 49

Red-‐Yellow-‐Green Monitoring Dashboards provide at-‐a-‐glance high level overviews. Conﬁden5al, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without BrightTag, Inc.s prior wri+en authoriza5on. © 2011 BrightTag, Inc. All Rights Reserved. 50

Automating Life in the Cloud - PuppetCamp Chicago '12

by Puppet Labs on Jul 25, 2012

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Automating Life in the Cloud - PuppetCamp Chicago ’12 Presentation Transcript