This document discusses application deployment orchestration using Puppet and Jenkins at Paydiant. It describes how Puppet is used to manage the entire Paydiant platform infrastructure with 900+ nodes. The document outlines Paydiant's use of the Puppet Dashboard to store configuration data in a Classes, Parameters and Supergroups pattern. It also details Paydiant's zero-downtime deployment process which involves taking subsets of nodes in and out of maintenance mode while upgrading applications. Finally, it provides an overview of how Jenkins is used to automate and monitor repeated deployment jobs.
Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rak...Puppet
Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate) presented by Anton Gurov and Chaminda Delpagodage, Paydiant at Puppet Camp Boston 2014
Ims11 ims13 application programming enhancements - IMS UG May 2014 Sydney & ...Robert Hain
IMS 13 extends the ability to access your IMS databases with SQL. In this session, I detail how to set up your environment for the IMS Catalog, and write COBOL applications accessing your IMS databases with SQL. Further, I will show how to use C# on the distributed platform to access your IMS databases.
What’s New For SQL Optimization In DB2 9 And DB2 10 For z/OSSurekha Parekh
Abstract:
Reducing the total cost of ownership is everyone’s challenge, and it’s one of the key benefits of query optimization. Since its initial inception in DB2® for z/OS®, the cost-based Optimizer has continually evolved to deliver expert-based query and workload analysis and many other performance-enhancing functions. Based on the latest use by customers, the Optimizer delivers even more in DB2 10. This session will show you how the Optimizer in DB2 10 for z/OS can help reduce your total cost of ownership. We’ll discuss incremental query optimization enhancements available with the upcoming release of DB2 10 that build on DB2 9 enhancements, such as global query optimization. This session will cover the insight discovered in early usage, provide the reason for each enhancement and highlight the most important enhancements.
Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rak...Puppet
Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate) presented by Anton Gurov and Chaminda Delpagodage, Paydiant at Puppet Camp Boston 2014
Ims11 ims13 application programming enhancements - IMS UG May 2014 Sydney & ...Robert Hain
IMS 13 extends the ability to access your IMS databases with SQL. In this session, I detail how to set up your environment for the IMS Catalog, and write COBOL applications accessing your IMS databases with SQL. Further, I will show how to use C# on the distributed platform to access your IMS databases.
What’s New For SQL Optimization In DB2 9 And DB2 10 For z/OSSurekha Parekh
Abstract:
Reducing the total cost of ownership is everyone’s challenge, and it’s one of the key benefits of query optimization. Since its initial inception in DB2® for z/OS®, the cost-based Optimizer has continually evolved to deliver expert-based query and workload analysis and many other performance-enhancing functions. Based on the latest use by customers, the Optimizer delivers even more in DB2 10. This session will show you how the Optimizer in DB2 10 for z/OS can help reduce your total cost of ownership. We’ll discuss incremental query optimization enhancements available with the upcoming release of DB2 10 that build on DB2 9 enhancements, such as global query optimization. This session will cover the insight discovered in early usage, provide the reason for each enhancement and highlight the most important enhancements.
Patterns and Tools for Database Versioning, Migration, Data Loading and Test ...Alan Pinstein
Talk given at CodeWorks PHP Conference in Atlanta on 9/29/2009.
Covers MP "migrations for php" project as well as other best-practices for managing database migrations for PHP projects.
There are a number of ways to add custom meta boxes to WordPress admin, from coding against core functions through code API frameworks to GUI interfaces. All of these have a place, but which is the right one for your project, code style and coding level? In this talk I will provide code examples and illustrations of the code techniques for each of the methods.
Time permitting, I will also show you how to extend the frameworks.
Aimed at all levels of developer because of the range of options covered.
Wi-Fi: diagnosi lato client/edge
Sonde di Synthetic Monitoring permettono di avere il corretto Feedback necessario per la gestione di infrastrutture di rete complesse
Migrating to SharePoint 2013 - Business and Technical PerspectiveJohn Calvert
Many organizations still run versions of SharePoint prior to 2013, but the appetite and pressure to upgrade is increasing as business and IT better understand the new features and capabilities of SharePoint 2013 / Online combined with its sister products Office 2013 and Exchange 2013, or alternately Office 365. Drawing on experience from real client engagements in this presentation we examine the business and technical roadmap, and challenges in planning and executing a migration to SharePoint 2013.
Patterns and Tools for Database Versioning, Migration, Data Loading and Test ...Alan Pinstein
Talk given at CodeWorks PHP Conference in Atlanta on 9/29/2009.
Covers MP "migrations for php" project as well as other best-practices for managing database migrations for PHP projects.
There are a number of ways to add custom meta boxes to WordPress admin, from coding against core functions through code API frameworks to GUI interfaces. All of these have a place, but which is the right one for your project, code style and coding level? In this talk I will provide code examples and illustrations of the code techniques for each of the methods.
Time permitting, I will also show you how to extend the frameworks.
Aimed at all levels of developer because of the range of options covered.
Wi-Fi: diagnosi lato client/edge
Sonde di Synthetic Monitoring permettono di avere il corretto Feedback necessario per la gestione di infrastrutture di rete complesse
Migrating to SharePoint 2013 - Business and Technical PerspectiveJohn Calvert
Many organizations still run versions of SharePoint prior to 2013, but the appetite and pressure to upgrade is increasing as business and IT better understand the new features and capabilities of SharePoint 2013 / Online combined with its sister products Office 2013 and Exchange 2013, or alternately Office 365. Drawing on experience from real client engagements in this presentation we examine the business and technical roadmap, and challenges in planning and executing a migration to SharePoint 2013.
Automating it management with Puppet + ServiceNowPuppet
As the leading IT Service Management and IT Operations Management platform in the marketplace, ServiceNow is used by many organizations to address everything from self service IT requests to Change, Incident and Problem Management. The strength of the platform is in the workflows and processes that are built around the shared data model, represented in the CMDB. This provides the ‘single source of truth’ for the organization.
Puppet Enterprise is a leading automation platform focused on the IT Configuration Management and Compliance space. Puppet Enterprise has a unique perspective on the state of systems being managed, constantly being updated and kept accurate as part of the regular Puppet operation. Puppet Enterprise is the automation engine ensuring that the environment stays consistent and in compliance.
In this webinar, we will explore how to maximize the value of both solutions, with Puppet Enterprise automating the actions required to drive a change, and ServiceNow governing the process around that change, from definition to approval. We will introduce and demonstrate several published integration points between the two solutions, in the areas of Self-Service Infrastructure, Enriched Change Management and Automated Incident Registration.
Simplified Patch Management with Puppet - Oct. 2020Puppet
Does your company struggle with patching systems? If so, you’re not alone — most organizations have attempted to solve this issue by cobbling together multiple tools, processes, and different teams, which can make an already complicated issue worse.
Puppet helps keep hosts healthy, secure and compliant by replacing time-consuming and error prone patching processes with Puppet’s automated patching solution.
Join this webinar to learn how to do the following with Puppet:
Eliminate manual patching processes with pre-built patching automation for Windows and Linux systems.
Gain visibility into patching status across your estate regardless of OS with new patching solution from the PE console.
Ensure your systems are compliant and patched in a healthy state
How Puppet Enterprise makes patch management easy across your Windows and Linux operating systems.
Presented by: Margaret Lee, Product Manager, Puppet, and Ajay Sridhar, Sr. Sales Engineer, Puppet.
2. 2
About Us
Chaminda Delpagodage
Paydiant Technical Operations Team
Release Engineering, Systems Administration,
Automation
linkedin.com/in/chamindad
Anton Gurov
Paydiant Technical Operations Team
Infrastructure, Systems Administration, Security
linkedin.com/in/antongurov
3. 3
Cloud-based mobile wallet solution
Open ecosystem for mobile payments,
offers and loyalty
Completely white-label
“Bank grade” platform of shared services
↘ SaaS
↘ Secure SDKs for iPhone and Android
Top tier investors and well capitalized
4. 4
Paydiant Puppet Use
Puppet Enterprise (PE) users since day one
100% PE coverage of Paydiant platform
↘ PE handles everything after instance bootstrap
900
800
700
600
500
400
Multiple environments actively managed by PE
↘ 4 Puppet Masters in multiple datacenters and security zones
↘ 8 Environments
Licensed node count doubling every year
Nodes under
management
Estimated by
Year-End
300
200
100
0
Hosts
2011 2012 2013 2014 EST
5. 5
Paydiant Puppet Use
‘11-12 – Bi-annual production platform
releases
↘ Waterfall – major platform change
↘ Big outage – 1-2 days on the weekend
‘13-14 – Transition to daily/weekly non-production
and monthly production releases
↘ Agile – smaller platform changes
↘ Zero-downtime deployment
↘ 100% Production release success rate since inception
Heavy usage of Puppet Dashboard, Puppet
APIs and Jenkins
6. 6
Puppet Dashboard as data repository
Why Dashboard?
↘ Visual, flexible, powerful (if used right)
↘ Allows for business data edits by teams
unfamiliar with Puppet
↘ Hiera not available at the time
Decided early on to keep
Puppet code and data
separate
Came up with our own
Dashboard pattern – “Classes,
Parameters and Supergroups”
Puppet
Module
Code
Puppet
Dashboar
d
Business
Data
Puppet
Module
Parameter
s
7. 7
Puppet Dashboard as data repository
Classes, Parameters and Supergroups pattern overview
class_C
supergroup_type_A
class_A class_B
parameters_X parameters_Y … parameters_Z
…
node 1 node 2 node 3 node 4 … node X
Groups
Nodes
8. 8
Puppet Dashboard as data repository
Classes, Parameters and Supergroups pattern overview
class_C
supergroup_type_B
class_A class_B
parameters_X parameters_Y … parameters_Z
…
node 1 node 2 node 3 node 4 … node X
Groups
Nodes
9. 9
Puppet Dashboard as data repository
Class building block
Group name prefixed with class_
Contains Puppet class and some default variables/parameters for the
class
class A class B
class_B
def: default
params
incl:
class B
class_A
def: default
params
incl:
class A
class C
class_C
def: default
params
incl:
class C
…
Classes
Groups
11. 11
Puppet Dashboard as data repository
Parameters building block
Group name prefixed
with parameters_
Only contains data and
data overrides
Arbitrary hierarchy levels
Allows for inheritance
and reuse
parameters_X
def: default
params
parameters_X_1
incl:
def: params
overrides
def: additional
params
parameters_X
parameters_X_2
incl:
def: params
overrides
def: additional
params
supergroup_A supergroup_B
parameters_X
supergroup_C
12. 12
Puppet Dashboard as data repository
Parameters building block – inheritance example
13. 13
Puppet Dashboard as data repository
Supergroup building block == server “role”
Group name prefixed with
supergroup_
Contains all the “ingredients”
for the node to configure and
define itself
Node can belong to only one
supergroup (many-to-one)
class_B
parameters_Z
class_A parameters_X
supergroup_type_A
incl:
class_A
parameters_Z
def: params overrides (if any)
def: additional params (if any)
node 1 node 2
Groups
Nodes
class_B
parameters_X
14. 14
Puppet Dashboard as data repository
Supergroup building block - example
2-3 pages
condensed
15. 15 Classes, Parameters and Supergroups pattern
Pros
All parameters and classes are visible on the Supergroup page
↘ See missing parameters (if inherited “SET ME!” from parent for example)
↘ See parameter clashes (Dashboard will warn if parameter is defined in 2 places)
↘ See exactly where parameter is defined
Allows teams unfamiliar with Puppet to make changes via Dashboard
Arbitrary data hierarchy/inheritance
Data reuse
16. 16 Classes, Parameters and Supergroups pattern
Cons
Version control is difficult
↘ Have to resolve to group cloning/export/import (custom RAKE copy/clone command from Puppet support)
↘ Puppet roadmap to fix this
Dashboard UI could use some help
↘ Too much data on the screen sometimes
↘ Lack of sorting/grouping
Can’t store complex multi-line variables like text blobs
28. Frontend
Load
Balancer
v.1
FFEE--AB
v.1
FFEE--BB
v.1
Backend
Load
Balancer
BFEE--AB
v.1
BFEE--BB
v.1
Putting a set of nodes into
maintenance mode
29. 29
Putting nodes into maintenance mode
Using LB node health check – http://nodeX:8080/healthcheck.jsp
Puppet ERB template for healthcheck.jsp content
…
…
…
Pseudo code:
Check if “maintenance mode” throw exception
else
If “module A” present
Check if module A is up
If “module B” present
Check if module B is up
…
Throw 503 if any exception caught
30. 30
Putting nodes into maintenance mode cont.
A parameter group controls the maintenance mode
E.g. Parameter group “parameters_deployment-staging-BankB”
controls “paydiant_app_operation_mode” for the nodes in set FE-B
of the Staging environment
31. 31
Putting nodes into maintenance mode cont.
Update group parameter using Rake API (as ‘puppet-dashboard’
user)
RACK_ENV=production /opt/puppet/bin/rake -s -X -f /opt/puppet/share/puppet-dashboard/Rakefile
nodegroup:variables [parameters_deployment-stagin-BankB, 'paydiant_app_operation_mode=MAINTENANCE’]
Puppet run-once using MCO (as ‘peadmin’ user)
mco puppet runonce --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B
While loop… check the health check page till all nodes return 503
(i.e. in maintenance) status
mco shellcmd --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B --cmd=''curl --silent
http://localhost:8080/healthcheck/healthcheck.jsp
32. Frontend
Load
Balancer
FFEE--AB
v.1
FFEE--BB
v.2
Backend
Load
Balancer
BFEE--AB
v.1
BFEE--BB
v.2
Upgrading applications
on a set of nodes
v.2
a
35. Frontend
Load
Balancer
FFEE--AB
v.1
FFEE--BB
v.2
Backend
Load
Balancer
BFEE--AB
v.1
BFEE--BB
v.2
Taking a set of nodes out of
maintenance mode
v.2
a
36. 36
Taking nodes out of maintenance mode
Update parameter using Rake API (as ‘puppet-dashboard’ user)
RACK_ENV=production /opt/puppet/bin/rake -s -X -f /opt/puppet/share/puppet-dashboard/Rakefile
nodegroup:variables [parameters_deployment-staging-BankB, 'paydiant_app_operation_mode=LIVE’]
Puppet run-once using MCO (as ‘peadmin’ user)
mco puppet runonce --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B
While loop… check the health check page till all nodes return 200
(i.e. live) status
mco shellcmd --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B --cmd=''curl --silent
http://localhost:8080/healthcheck/healthcheck.jsp
40. 40
What is Jenkins
Tool to schedule and monitor the execution of repeated jobs
41. 41
Why Jenkins ?
Configurability
↘ Different types of input parameters
↘ Invoke shell scripts
↘ Post-build actions (automatic/manual)
42. 42
Why Jenkins ? cont.
Plugin support
↘ More than 600 plugins (https://wiki.jenkins-ci.org/display/JENKINS/Plugins)
↘ Eg. vSphere plugin (stop/start, snapshots, rollbacks…)
↘ Build pipeline plugin
↘ Parameterized remote trigger plugin
43. 43
Why Jenkins ? cont.
Keeps all your console logs at a single place
↘ No need to hunt for 10 log files on 5 different machines
↘ Visual representation of passed/failed/in-progress status, based on downstream shell scripts or other jobs
50. 50
Jenkins – Puppet Integration cont.
Jenkins invoke local bash scripts, which in turn use SSH to call;
↘ MCO (as ‘peadmin’ user on Puppet Master)
↘ Rake API (as ‘puppet-dashboard’ user on Puppet Master)
SSH login as ‘peadmin’ and ‘puppet-dashboard’ is password-less,
using PKI
↘ Generate RSA keypair for the local Jenkins user, using ssh-keygen command
↘ Append the public key to ~/.ssh/authorized_keys file of ‘peadmin’ and ‘puppet-dashboard’ users, on Puppet Master
MCO special purpose sub commands we use;
↘ puppet
↘ service
↘ shellcmd* (ask your Puppet Enterprise Support for this custom MCO plugin)
52. 52
Recap/Takeaways…
Use Puppet Enterprise
↘ Support is awesome (Celia Cottle, Jay Wallace, Ken Johnson, Zachary Stern – you guys rock!)
↘ Gotten help and features from James Turnbull and Nigel Kersten with some early versions of PE
↘ Live management and Mcollective are essential for any self-respecting enterprise
Zero-downtime upgrades
↘ To Dashboard or not to Dashboard?
↘ Database update phases
↘ Managing LB health check monitors dynamically using Puppet
Automation baby steps – don’t boil the ocean
↘ Understand what you are doing before automating it - develop runbooks
↘ Identify manual steps and script some of them
↘ Add scripts to orchestration tool (Jenkins, ServiceNow, whatever else you use in-house)