Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)

Application Deployment Orchestration
with Puppet and Jenkins
Anton Gurov, Chaminda Delpagodage
August 20, 2014

22
About Us
Chaminda Delpagodage
Paydiant Technical Operations Team
Release Engineering, Systems Administration, Automation
linkedin.com/in/chamindad
Anton Gurov
Paydiant Technical Operations Team
Infrastructure, Systems Administration, Security
linkedin.com/in/antongurov

33
Cloud-based mobile wallet solution
Open ecosystem for mobile payments, offers
and loyalty
Completely white-label
“Bank grade” platform of shared services
↘ SaaS
↘ Secure SDKs for iPhone and Android
Top tier investors and well capitalized

44
Paydiant Puppet Use
Puppet Enterprise (PE) users since day one
100% PE coverage of Paydiant platform
↘ PE handles everything after instance bootstrap
Multiple environments actively managed by PE
↘ 4 Puppet Masters in multiple datacenters and security zones
↘ 8 Environments
Licensed node count doubling every year
Estimated by
Year-End
0
100
200
300
400
500
600
700
800
900
Hosts
Nodes under
management
2011 2012 2013 2014 EST

55
Paydiant Puppet Use
‘11-12 – Bi-annual production platform releases
↘ Waterfall – major platform change
↘ Big outage – 1-2 days on the weekend
‘13-14 – Transition to daily/weekly non-
production and monthly production releases
↘ Agile – smaller platform changes
↘ Zero-downtime deployment
↘ 100% Production release success rate since inception
Heavy usage of Puppet Dashboard, Puppet APIs
and Jenkins

66
Puppet Dashboard as data repository
Why Dashboard?
↘ Visual, flexible, powerful (if used right)
↘ Allows for business data edits by teams
unfamiliar with Puppet
↘ Hiera not available at the time
Decided early on to keep Puppet
code and data separate
Came up with our own Dashboard
pattern – “Classes, Parameters
and Supergroups”
Puppet
Module
Code
PuppetDashboard
Business
Data
Puppet
Module
Parameters

77
Classes, Parameters and Supergroups pattern overview
class_C
supergroup_type_A
class_Bclass_A
parameters_X parameters_Y parameters_Z…
…
node 1 node 2 node X…node 4node 3
Groups
Nodes

88
Classes, Parameters and Supergroups pattern overview
class_C
supergroup_type_B
class_Bclass_A
parameters_X parameters_Y parameters_Z…
…
node 1 node 2 node X…node 4node 3
Groups
Nodes

99
class_B
def: default params
incl:
Class building block
class B
class A class B
class_A
def: default params
incl: class A
class_C
def: default params
incl: class C
class C
…
Classes
Groups
Group name prefixed with class_
Contains Puppet class and some default variables/parameters for the class

1010
Class building block - example

1111
Parameters building block
Group name prefixed with
parameters_
Only contains data and
data overrides
Arbitrary hierarchy levels
Allows for inheritance and
reuse
parameters_X_1
incl:
def: params overrides
def: additional params
parameters_X
def: default params
parameters_X
supergroup_A supergroup_B
parameters_X_2
incl:
def: params overrides
def: additional params
parameters_X
supergroup_C

1212
Parameters building block – inheritance example

1313
Supergroup building block == server “role”
Group name prefixed with
supergroup_
Contains all the “ingredients” for
the node to configure and define
itself
Node can belong to only one
supergroup (many-to-one)
supergroup_type_A
incl:
def: params overrides (if any)
def: additional params (if any)
class_B
class_A parameters_X
parameters_Z
node 1 node 2
Groups
Nodes
class_B
class_A
parameters_X
parameters_Z

1414
Supergroup building block - example
2-3 pages
condensed

1515
Classes, Parameters and Supergroups pattern Pros
All parameters and classes are visible on the Supergroup page
↘ See missing parameters (if inherited “SET ME!” from parent for example)
↘ See parameter clashes (Dashboard will warn if parameter is defined in 2 places)
↘ See exactly where parameter is defined
Allows teams unfamiliar with Puppet to make changes via Dashboard
Arbitrary data hierarchy/inheritance
Data reuse

1616
Classes, Parameters and Supergroups pattern Cons
Version control is difficult
↘ Have to resolve to group cloning/export/import (custom RAKE copy/clone command from Puppet support)
↘ Puppet roadmap to fix this
Dashboard UI could use some help
↘ Too much data on the screen sometimes
↘ Lack of sorting/grouping
Can’t store complex multi-line variables like text blobs

Zero-Downtime Deployment architecture …

v.1
Frontend
Load Balancer
FE-BFE-A
v.1
FE-BFE-B
v.1
Backend
Load Balancer
FE-BBE-A
v.1
FE-BBE-B
v.1
parameters_deployment-staging-FE-BankA
paydiant_deployment_bank=STAGING-FRONTEND-A
paydiant_app_operation_mode=LIVE
paydiant_app_version=1
v.1
High-level platform
representation
parameters_deployment-staging-BE-BankA
paydiant_deployment_bank=STAGING-BACKEND-A
parameters_deployment-staging-FE-BankB
paydiant_deployment_bank=STAGING-FRONTEND-B
parameters_deployment-staging-BE-BankB
paydiant_deployment_bank=STAGING-BACKEND-B

Frontend
Load Balancer
FE-BFE-A
v.1
FE-BFE-B
v.1
Backend
Load Balancer
FE-BBE-A
v.1
FE-BBE-B
v.1
Disable B(FE+BE)
v.1v.1
paydiant_app_operation_mode=MAINTENANCE

v.2a
Frontend
Load Balancer
FE-BFE-A
v.1
FE-BFE-B
v.1
Backend
Load Balancer
FE-BBE-A
v.1
FE-BBE-B
v.1
Run first phase of database changes
(i.e. adds new stuff &
migrate data)
v.2aDB changes Phase 1

Frontend
Load Balancer
FE-BFE-A
v.1
FE-BFE-B
v.2
Backend
Load Balancer
FE-BBE-A
v.1
FE-BBE-B
v.2
Upgrade B (FE+BE)
v.2av.2a

Frontend
Load Balancer
FE-BFE-A
v.1
FE-BFE-B
v.2
Backend
Load Balancer
FE-BBE-A
v.1
FE-BBE-B
v.2
Re-enable B (FE+BE)
v.2av.2a

Frontend
Load Balancer
FE-BFE-A
v.1
FE-BFE-B
v.2
Backend
Load Balancer
FE-BBE-A
v.1
FE-BBE-B
v.2
Disable A(FE+BE)
v.2av.2a

Frontend
Load Balancer
FE-BFE-A
v.2
FE-BFE-B
v.2
Backend
Load Balancer
FE-BBE-A
v.2
FE-BBE-B
v.2
Upgrade A (FE+BE)
v.2av.2a

Frontend
Load Balancer
FE-BFE-A
v.2
FE-BFE-B
v.2
Backend
Load Balancer
FE-BBE-A
v.2
FE-BBE-B
v.2
Re-enable A (FE+BE)
v.2av.2a

v.2
Frontend
Load Balancer
FE-BFE-A
v.2
FE-BFE-B
v.2
Backend
Load Balancer
FE-BBE-A
v.2
FE-BBE-B
v.2
Run second phase of database changes
(Cleanup old v.1 data)
v.2DB changes Phase 2

Details of the upgrade sequence …

v.1
Frontend
Load Balancer
FE-BFE-A
v.1
FE-BFE-B
v.1
Backend
Load Balancer
FE-BBE-A
v.1
FE-BBE-B
v.1
Putting a set of nodes into
maintenance mode

2929
Putting nodes into maintenance mode
Using LB node health check – http://nodeX:8080/healthcheck.jsp
Puppet ERB template for healthcheck.jsp content
…
…
…
Pseudo code:
Check if “maintenance mode” throw exception
else
If “module A” present
Check if module A is up
If “module B” present
Check if module B is up
…
Throw 503 if any exception caught

3030
Putting nodes into maintenance mode cont.
A parameter group controls the maintenance mode
E.g. Parameter group “parameters_deployment-staging-BankB”
controls “paydiant_app_operation_mode” for the nodes in set FE-B of
the Staging environment

3131
Putting nodes into maintenance mode cont.
Update group parameter using Rake API (as ‘puppet-dashboard’ user)
RACK_ENV=production /opt/puppet/bin/rake -s -X -f /opt/puppet/share/puppet-dashboard/Rakefile
nodegroup:variables [parameters_deployment-stagin-BankB, 'paydiant_app_operation_mode=MAINTENANCE’]
Puppet run-once using MCO (as ‘peadmin’ user)
mco puppet runonce --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B
While loop… check the health check page till all nodes return 503 (i.e. in
maintenance) status
mco shellcmd --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B --cmd=''curl --silent
http://localhost:8080/healthcheck/healthcheck.jsp

Frontend
Load Balancer
FE-BFE-A
v.1
FE-BFE-B
v.2
Backend
Load Balancer
FE-BBE-A
v.1
FE-BBE-B
v.2
Upgrading applications
on a set of nodes
v.2a

3333
Upgrading Application Version
Disable Puppet agent
mco puppet disable --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B
Stop Tomcat service
mco service tomcat stop --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B
Cleanup exploded Tomcat webapps directory (for sanity)
mco shellcmd --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B --cmd='find $TOMCAT_HOME/webapps/ -
maxdepth 1 -mindepth 1 -type d -exec rm -rf {} ;’

3434
Upgrading Application Version Cont.
Upgrade the application version
nodegroup:variables [parameters_deployment-stagin-BankB, ’paydiant_app_version=2’]
Re-enable Puppet
mco puppet enable --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B
Puppet run-once

Frontend
Load Balancer
FE-BFE-A
v.1
FE-BFE-B
v.2
Backend
Load Balancer
FE-BBE-A
v.1
FE-BBE-B
v.2
Taking a set of nodes out of
maintenance mode
v.2a

3636
Taking nodes out of maintenance mode
Update parameter using Rake API (as ‘puppet-dashboard’ user)
nodegroup:variables [parameters_deployment-staging-BankB, 'paydiant_app_operation_mode=LIVE’]
Puppet run-once using MCO (as ‘peadmin’ user)
While loop… check the health check page till all nodes return 200 (i.e. live)
status
mco shellcmd --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B --cmd=''curl --silent
http://localhost:8080/healthcheck/healthcheck.jsp

Frontend
Load Balancer
FE-BFE-A
v.1
FE-BFE-B
v.2
Backend
Load Balancer
FE-BBE-A
v.1
FE-BBE-B
v.2
Switching traffic to
upgraded stack
v.2a

Viewing transition in Splunk across multiple datacenters

4040
What is Jenkins
Tool to schedule and monitor the execution of repeated jobs

4141
Why Jenkins ?
Configurability
↘ Different types of input parameters
↘ Invoke shell scripts
↘ Post-build actions (automatic/manual)

4242
Why Jenkins ? cont.
Plugin support
↘ More than 600 plugins (https://wiki.jenkins-ci.org/display/JENKINS/Plugins)
↘ Eg. vSphere plugin (stop/start, snapshots, rollbacks…)
↘ Build pipeline plugin
↘ Parameterized remote trigger plugin

4343
Why Jenkins ? cont.
Keeps all your console logs at a single place
↘ No need to hunt for 10 log files on 5 different machines
↘ Visual representation of passed/failed/in-progress status, based on downstream shell scripts or other jobs

4444
Why Jenkins ? cont.
And it’s…

MCO
Rake API
DB FE-BFE-* FE-BBE-*
Source code,
liquibase
change sets

4646
Jenkins – Puppet Integration

4747
Jenkins – Puppet Integration cont.

4848

4949

5050
Jenkins invoke local bash scripts, which in turn use SSH to call;
↘ MCO (as ‘peadmin’ user on Puppet Master)
↘ Rake API (as ‘puppet-dashboard’ user on Puppet Master)
SSH login as ‘peadmin’ and ‘puppet-dashboard’ is password-less, using PKI
↘ Generate RSA keypair for the local Jenkins user, using ssh-keygen command
↘ Append the public key to ~/.ssh/authorized_keys file of ‘peadmin’ and ‘puppet-dashboard’ users, on Puppet Master
MCO special purpose sub commands we use;
↘ puppet
↘ service
↘ shellcmd* (ask your Puppet Enterprise Support for this custom MCO plugin)

5151
Links
Rake API: https://docs.puppetlabs.com/pe/latest/console_rake_api.html
MCO: https://docs.puppetlabs.com/mcollective/reference/basic/basic_cli_usage.html
Jenkins: http://jenkins-ci.org/
Liquibase: http://www.liquibase.org/documentation/index.html

5252
Recap/Takeaways…
Use Puppet Enterprise
↘ Support is awesome (Celia Cottle, Jay Wallace, Ken Johnson, Zachary Stern – you guys rock!)
↘ Gotten help and features from James Turnbull and Nigel Kersten with some early versions of PE
↘ Live management and Mcollective are essential for any self-respecting enterprise
Zero-downtime upgrades
↘ To Dashboard or not to Dashboard?
↘ Database update phases
↘ Managing LB health check monitors dynamically using Puppet
Automation baby steps – don’t boil the ocean
↘ Understand what you are doing before automating it - develop runbooks
↘ Identify manual steps and script some of them
↘ Add scripts to orchestration tool (Jenkins, ServiceNow, whatever else you use in-house)

Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)

Recommended

Recommended

More Related Content

Similar to Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)

Similar to Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate) (20)

More from Puppet

More from Puppet (20)

Recently uploaded

Recently uploaded (20)

Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)