Your SlideShare is downloading. ×
0
PuppetConf 2013
CHALLENGES
• Rapid growth
– Number of servers being provisioned
– New services being provided
• Manual configurations
• Drift
• Silos...
Objectives
• Push button configuration
• Self service for customers
• Visibility into state of configurations
– Reports
• Simplify kn...
• Support workstations/dev/test/prod
• Allow adhoc manual changes*
• Do not slow down development teams
• Include legacy s...
Application
Middleware
OS
Server
Storage
Network
IAAS
PAAS
SAAS
Puppet
Puppet Versions
• Puppet 3.0.0
• Puppet Dashboard 1.2.12
• PuppetDB 1.0.1
• MCollective 2.2.1
• ActiveMQ 5.5
The Data
• Inherent Facts
– Facter
– Exists simply because the node exists.
• Derived Facts
– Business rules applied to inherent fa...
Assigned Facts
• espn_role
– Identifes the role class to be applied
• espn_cluster
– Identifies nodes in the cluster
• esp...
What to manage? Everything!
Role
• Aligns with business and or IT needs
• Defines everything on a node
• Used for classification of nodes
• Exactly on...
Group everything by roles
AA BB CC
Release Environment
• Not puppet environments
• Determined by application development
and release cycle
• espn_env
Group by release environment
AA BB CC
ProdProd
QAQA
DevDev
WorkWork
Clusters
• Always identical
• Configured to interact with one another
• espn_cluster
Group by clusters
AA BB CC
ProdProd
QAQA
DevDev
WorkWork
Hiera hierarchy in theory
hostname/%{hostname}
cluster/%{espn_cluster_name}
role_env/%{espn_role}_%{espn_env}
role/%{espn_...
Hiera hierarchy in practice
role_env/%{espn_role}_%{espn_env}
role/%{espn_role}
env/%{espn_env}
network/%{espn_network}
de...
Classification
Application
Middleware
OS
Server
Storage
Network
IAAS
PAAS
SAAS
Role
Profiles
Releases
Role
Profiles
Releases Resources
Resources
Classes Resources
Resource Hierarchy
Resources
• The building blocks
• Everything managed is a resource
• Defined in modules, used by profiles
• Two resources ...
Modules
• Isolate resources within the module
• Never reference another module
• No organizational specific logic
• init.p...
R RR RR RR R
R RR RR RR R
R RR RR RR R
R RR RR RR R
R RR RR RR R
Resources by module
R RR RR RR R
R RR RR RR R
R RR RR RR R
R RR RR RR R
R RR RR RR R
Never cross modules
Role
Profiles
Releases Resources
Resources
Classes Resources
Resource Hierarchy
Profile
• Defines the platform
• Cross module references
• Enforces dependencies between modules
• Class parameters preven...
Example Profile
class profile::jboss_eap_6(
$java_version=‘jdk_1_7_u10’,
$mod_cluster_version=‘1.2.3’,
$httpd_version=‘2.2...
Role
Profiles
Releases Resources
Resources
Classes Resources
Resource Hierarchy
Release
• Special type of profile
• Knows how to install on top of a profile
• Deploys resources from an “artifact hash”
•...
Artifact Hash
• Defines abstract resources in a release
• Contract between developers and
operational groups
• Profile agn...
Example Artifact Hash
applications
‘jee-app.ear’:
version: 1.2.3
espn.war:
version: 1.0.0
datasources:
datasource1:
driver...
Example Release
Class release::studio_record{
artifact_hash = hiera(‘artifacts’,undef,”release/${espn_release_id}”)
#modif...
Role
Profiles
Releases Resources
Resources
Classes Resources
Resource Hierarchy
Example Role
class role::studio_record{
include profile::base
include profile::jboss_eap_6
include profile::mysql_5
includ...
site.pp
# assigned facts retrieved from external datasources
$espn_cluster_nodes = espn_cluster_nodes()
$espn_release_id =...
Defining a Node
MCollective
Colonol John Boyd
• Military Strategist
• OODA Loop
Observations
Decision
(Hypothesis)
Action
(Test)
Observe Orient Decide Act
Feedback
Feedback
Unfolding
Circumstances
Outsi...
OODA Loop
"Time is the dominant parameter. The
pilot who goes through the OODA cycle in
the shortest time prevails because...
Observations
Decision
(Hypothesis)
Action
(Test)
Observe Orient Decide Act
Feedback: Puppet runs on dev/test puppet enviro...
Observations
Decision
(Hypothesis)
Action
(Test)
Observe Orient Decide Act
Feedback: Puppet runs on dev/test puppet enviro...
Noop Puppet Runs
• Hourly cron job
– “mco puppet runall 20 –noop”
• Dashboard displays unresponsive nodes
– no_longer_repo...
Observations
Decision
(Hypothesis)
Action
(Test)
Observe Orient Decide Act
Feedback: Puppet runs on dev/test puppet enviro...
Configuration Changes
• Always noop first
• Always target nodes with filters
• Always use tags
• Validate changes then app...
Observations
Decision
(Hypothesis)
Action
(Test)
Observe Orient Decide Act
Feedback: Puppet runs on dev/test puppet enviro...
20/20 Hindsight
Do Differently?
• Implement MCollective first
– Security
– Sub-collectives
– Availability
• Plan for developer dashboards
...
Nice to have?
• Puppet runs that span multiple nodes
– Allocate disk on node A, create shared
filesystem on node B
• Resou...
Questions?
Upcoming SlideShare
Loading in...5
×

Case Study: Green Field Implementation of Puppet 3.0 at ESPN

1,993

Published on

At the end of 2012 ESPN undertook an effort to modernize its deployment and maintenance of linux based platform services. ESPN faced a challenging problem in that hundreds of servers needed to be puppetized yet the largest cluster of identical servers was only eight servers. Therefore having a puppet environment that was flexible, consistent, simple to understand and data driven was critical to success. This session looks at the architectural decisions made by ESPN while performing a green field implementation of Puppet 3.0 and reflects on the resulting good and bad of those decisions.

Ben Schofield
Senior Application Architect, ESPN
Ben Schofield is the middleware architect for ESPN. With 11 years of IT experience working for Fortune 200 companies in the retail, insurance, financial and media industries, Ben has seen the good, bad and ugly of IT operations and management. He brings a unique perspective on how a well designed devops team with the right mind set can help large IT departments reduce costs and decrease time to market.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,993
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
43
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Welcome everyone Ask for show of hands about managers and those looking to implement puppet for the first time. Timeline: Challenges Objectives Data Classification MCollective
  • 1 new server per business day in that last 12 months Silos – request and wait processes
  • adhoc manual changes helps with troubleshooting return service activies apply changes to legacy servers without breaking them
  • How we wanted to use puppet initially IAAS not in scope Developed an ISO with kernel and puppet rpm’s to do kickstart provisioning which runs puppet auto
  • ----- Meeting Notes (8/13/13 08:56) ----- adhoc manual changes helps with troubleshooting return service activies dynamic changed to elasticity ephemeral apply changes to legacy servers without breaking them
  • Talk about Data first and the Classification How we think about the data in terms of puppet How we apply the right data to the right nodes
  • Derived and assigned facts are not puppet terminology
  • Glance over this slide Will go into detail in the coming slides
  • How we apply resources
  • Classification from a bottom up perspective
  • ----- Meeting Notes (8/13/13 08:56) ----- give example reit
  • Class parameters should be declared as defaults in hiera. They are added here for clarity. Not showing all the jboss resources used to configur secruity and common services
  • Jms queues and topic Any resource application server resource for example: virtual hosts
  • Releases can deploy same artifact hash to multiple different types of profiles. Allows swapping out vendor software
  • ----- Meeting Notes (8/13/13 08:56) ----- put top down slide before this slide
  • Future plan is to add IAAS assigned facts to do provisioning and replace kickstart IAAS assigned facts Availability zones cpu/mem/disk
  • ----- Meeting Notes (8/13/13 08:56) ----- missing arrows on this slide talk about devops team and dashboard
  • Facts + mcollective + rules engine = skynet
  • Transcript of "Case Study: Green Field Implementation of Puppet 3.0 at ESPN"

    1. 1. PuppetConf 2013
    2. 2. CHALLENGES
    3. 3. • Rapid growth – Number of servers being provisioned – New services being provided • Manual configurations • Drift • Silos • Traditional clouds not reasonable
    4. 4. Objectives
    5. 5. • Push button configuration • Self service for customers • Visibility into state of configurations – Reports • Simplify knowledge transfer • Data driven • Elastic • Build/rebuild a node at anytime
    6. 6. • Support workstations/dev/test/prod • Allow adhoc manual changes* • Do not slow down development teams • Include legacy servers
    7. 7. Application Middleware OS Server Storage Network IAAS PAAS SAAS Puppet
    8. 8. Puppet Versions • Puppet 3.0.0 • Puppet Dashboard 1.2.12 • PuppetDB 1.0.1 • MCollective 2.2.1 • ActiveMQ 5.5
    9. 9. The Data
    10. 10. • Inherent Facts – Facter – Exists simply because the node exists. • Derived Facts – Business rules applied to inherent facts. – Puppet custom facts • Assigned Facts – Exist because we deem it to be true. – Top or node scoped variables
    11. 11. Assigned Facts • espn_role – Identifes the role class to be applied • espn_cluster – Identifies nodes in the cluster • espn_env – workstation, dev, test, qa, prod • espn_owner – change notifications – security delegation – licensing
    12. 12. What to manage? Everything!
    13. 13. Role • Aligns with business and or IT needs • Defines everything on a node • Used for classification of nodes • Exactly one role per node • Includes profiles and releases • espn_role
    14. 14. Group everything by roles AA BB CC
    15. 15. Release Environment • Not puppet environments • Determined by application development and release cycle • espn_env
    16. 16. Group by release environment AA BB CC ProdProd QAQA DevDev WorkWork
    17. 17. Clusters • Always identical • Configured to interact with one another • espn_cluster
    18. 18. Group by clusters AA BB CC ProdProd QAQA DevDev WorkWork
    19. 19. Hiera hierarchy in theory hostname/%{hostname} cluster/%{espn_cluster_name} role_env/%{espn_role}_%{espn_env} role/%{espn_role} env/%{espn_env} network/%{espn_network} os/%{operationsystem} default
    20. 20. Hiera hierarchy in practice role_env/%{espn_role}_%{espn_env} role/%{espn_role} env/%{espn_env} network/%{espn_network} defaults
    21. 21. Classification
    22. 22. Application Middleware OS Server Storage Network IAAS PAAS SAAS Role Profiles Releases
    23. 23. Role Profiles Releases Resources Resources Classes Resources Resource Hierarchy
    24. 24. Resources • The building blocks • Everything managed is a resource • Defined in modules, used by profiles • Two resources never manage the same configuration
    25. 25. Modules • Isolate resources within the module • Never reference another module • No organizational specific logic • init.pp is a minimal installer • Reusability is key
    26. 26. R RR RR RR R R RR RR RR R R RR RR RR R R RR RR RR R R RR RR RR R Resources by module
    27. 27. R RR RR RR R R RR RR RR R R RR RR RR R R RR RR RR R R RR RR RR R Never cross modules
    28. 28. Role Profiles Releases Resources Resources Classes Resources Resource Hierarchy
    29. 29. Profile • Defines the platform • Cross module references • Enforces dependencies between modules • Class parameters prevent hiera overrides
    30. 30. Example Profile class profile::jboss_eap_6( $java_version=‘jdk_1_7_u10’, $mod_cluster_version=‘1.2.3’, $httpd_version=‘2.2.22’, ){ include “java::${java_version}” include directories::middleware class{‘mod_cluster’: version => $mod_cluster_version,} class{‘httpd’: version => $httpd_version,} class{‘jboss_eap_6’: java_home => getvar(“java::${java_version}::home”),} }
    31. 31. Role Profiles Releases Resources Resources Classes Resources Resource Hierarchy
    32. 32. Release • Special type of profile • Knows how to install on top of a profile • Deploys resources from an “artifact hash” • Cleans up removed artifacts • Driven by versioned release id – espn_release_id
    33. 33. Artifact Hash • Defines abstract resources in a release • Contract between developers and operational groups • Profile agnostic
    34. 34. Example Artifact Hash applications ‘jee-app.ear’: version: 1.2.3 espn.war: version: 1.0.0 datasources: datasource1: driver_name: ”oracle” libraries: utility.jar: version: 1.0.0
    35. 35. Example Release Class release::studio_record{ artifact_hash = hiera(‘artifacts’,undef,”release/${espn_release_id}”) #modify the artifact hash so it can be used with create_resources #set organization specific parameters such URL’s to the artifact repo resources {‘jboss7_datasource: purge => true,} create_resources(jboss7_datasource, $artifact_hash[‘datasources’]) resources {‘jboss7_deployment: purge => true,} create_resources(jboss7_deployment, $artifact_hash[‘applications’]) }
    36. 36. Role Profiles Releases Resources Resources Classes Resources Resource Hierarchy
    37. 37. Example Role class role::studio_record{ include profile::base include profile::jboss_eap_6 include profile::mysql_5 include release::studio_record }
    38. 38. site.pp # assigned facts retrieved from external datasources $espn_cluster_nodes = espn_cluster_nodes() $espn_release_id = hiera(‘espn_release_id’) # single assigned fact drives 100% of classification node default { include “role::${espn_role}” }
    39. 39. Defining a Node
    40. 40. MCollective
    41. 41. Colonol John Boyd • Military Strategist • OODA Loop
    42. 42. Observations Decision (Hypothesis) Action (Test) Observe Orient Decide Act Feedback Feedback Unfolding Circumstances Outside Information Unfolding Interaction With Environment Cultural Traditions Analysis & Synthesis Previous Experience New Information Genetic Heritage Feedback Implicit Guidance & Control Implicit Guidance & Control
    43. 43. OODA Loop "Time is the dominant parameter. The pilot who goes through the OODA cycle in the shortest time prevails because his opponent is caught responding to situations that have already changed.“ Harry Hillaker (chief designer of the F-16)
    44. 44. Observations Decision (Hypothesis) Action (Test) Observe Orient Decide Act Feedback: Puppet runs on dev/test puppet environments Feedback Unfolding Circumstances Outside Information Unfolding Interaction With Environment Cultural Traditions Analysis & Synthesis Previous Experience New Information Genetic Heritage Feedback Implicit Guidance & Control Implicit Guidance & Control
    45. 45. Observations Decision (Hypothesis) Action (Test) Observe Orient Decide Act Feedback: Puppet runs on dev/test puppet environments Feedback: (Test) Noop puppet run on production Unfolding Circumstances Outside Information Unfolding Interaction With Environment Cultural Traditions Analysis & Synthesis Previous Experience New Information Genetic Heritage Feedback Implicit Guidance & Control Implicit Guidance & Control
    46. 46. Noop Puppet Runs • Hourly cron job – “mco puppet runall 20 –noop” • Dashboard displays unresponsive nodes – no_longer_reporting_cutoff
    47. 47. Observations Decision (Hypothesis) Action (Test) Observe Orient Decide Act Feedback: Puppet runs on dev/test puppet environments Feedback: (Test) Noop puppet run on production Unfolding Circumstances Outside Information Unfolding Interaction With Environment Puppet Dashboard Cultural Traditions Analysis & Synthesis Previous Experience New Information Genetic Heritage Feedback: (Action) No-noop puppet run on production Implicit Guidance & Control Implicit Guidance & Control
    48. 48. Configuration Changes • Always noop first • Always target nodes with filters • Always use tags • Validate changes then apply –no-noop mco puppet runonce --no-noop -C jboss-eap-6 --tag initscript --tag jboss-eap-6
    49. 49. Observations Decision (Hypothesis) Action (Test) Observe Orient Decide Act Feedback: Puppet runs on dev/test puppet environments Feedback: (Test) Noop puppet run on production Unfolding Circumstances Outside Information Cultural Traditions Analysis & Synthesis Previous Experience New Information Genetic Heritage Feedback: (Action) No-noop puppet run on production Implicit Guidance & Control: MCollective Agent Plugins Implicit Guidance & Control Unfolding Interaction With Environment Puppet Dashboard
    50. 50. 20/20 Hindsight
    51. 51. Do Differently? • Implement MCollective first – Security – Sub-collectives – Availability • Plan for developer dashboards • Implement Custom ENC
    52. 52. Nice to have? • Puppet runs that span multiple nodes – Allocate disk on node A, create shared filesystem on node B • Resources automatically tagged with catalog unique identifiers • Role based access control for dashboard
    53. 53. Questions?
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×