• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Case Study: Green Field Implementation of Puppet 3.0 at ESPN
 

Case Study: Green Field Implementation of Puppet 3.0 at ESPN

on

  • 1,946 views

At the end of 2012 ESPN undertook an effort to modernize its deployment and maintenance of linux based platform services. ESPN faced a challenging problem in that hundreds of servers needed to be ...

At the end of 2012 ESPN undertook an effort to modernize its deployment and maintenance of linux based platform services. ESPN faced a challenging problem in that hundreds of servers needed to be puppetized yet the largest cluster of identical servers was only eight servers. Therefore having a puppet environment that was flexible, consistent, simple to understand and data driven was critical to success. This session looks at the architectural decisions made by ESPN while performing a green field implementation of Puppet 3.0 and reflects on the resulting good and bad of those decisions.

Ben Schofield
Senior Application Architect, ESPN
Ben Schofield is the middleware architect for ESPN. With 11 years of IT experience working for Fortune 200 companies in the retail, insurance, financial and media industries, Ben has seen the good, bad and ugly of IT operations and management. He brings a unique perspective on how a well designed devops team with the right mind set can help large IT departments reduce costs and decrease time to market.

Statistics

Views

Total Views
1,946
Views on SlideShare
1,696
Embed Views
250

Actions

Likes
0
Downloads
27
Comments
0

1 Embed 250

http://puppetlabs.com 250

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Welcome everyone Ask for show of hands about managers and those looking to implement puppet for the first time. Timeline: Challenges Objectives Data Classification MCollective
  • 1 new server per business day in that last 12 months Silos – request and wait processes
  • adhoc manual changes helps with troubleshooting return service activies apply changes to legacy servers without breaking them
  • How we wanted to use puppet initially IAAS not in scope Developed an ISO with kernel and puppet rpm’s to do kickstart provisioning which runs puppet auto
  • ----- Meeting Notes (8/13/13 08:56) ----- adhoc manual changes helps with troubleshooting return service activies dynamic changed to elasticity ephemeral apply changes to legacy servers without breaking them
  • Talk about Data first and the Classification How we think about the data in terms of puppet How we apply the right data to the right nodes
  • Derived and assigned facts are not puppet terminology
  • Glance over this slide Will go into detail in the coming slides
  • How we apply resources
  • Classification from a bottom up perspective
  • ----- Meeting Notes (8/13/13 08:56) ----- give example reit
  • Class parameters should be declared as defaults in hiera. They are added here for clarity. Not showing all the jboss resources used to configur secruity and common services
  • Jms queues and topic Any resource application server resource for example: virtual hosts
  • Releases can deploy same artifact hash to multiple different types of profiles. Allows swapping out vendor software
  • ----- Meeting Notes (8/13/13 08:56) ----- put top down slide before this slide
  • Future plan is to add IAAS assigned facts to do provisioning and replace kickstart IAAS assigned facts Availability zones cpu/mem/disk
  • ----- Meeting Notes (8/13/13 08:56) ----- missing arrows on this slide talk about devops team and dashboard
  • Facts + mcollective + rules engine = skynet

Case Study: Green Field Implementation of Puppet 3.0 at ESPN Case Study: Green Field Implementation of Puppet 3.0 at ESPN Presentation Transcript

  • PuppetConf 2013
  • CHALLENGES
  • • Rapid growth – Number of servers being provisioned – New services being provided • Manual configurations • Drift • Silos • Traditional clouds not reasonable
  • Objectives
  • • Push button configuration • Self service for customers • Visibility into state of configurations – Reports • Simplify knowledge transfer • Data driven • Elastic • Build/rebuild a node at anytime
  • • Support workstations/dev/test/prod • Allow adhoc manual changes* • Do not slow down development teams • Include legacy servers
  • Application Middleware OS Server Storage Network IAAS PAAS SAAS Puppet
  • Puppet Versions • Puppet 3.0.0 • Puppet Dashboard 1.2.12 • PuppetDB 1.0.1 • MCollective 2.2.1 • ActiveMQ 5.5
  • The Data
  • • Inherent Facts – Facter – Exists simply because the node exists. • Derived Facts – Business rules applied to inherent facts. – Puppet custom facts • Assigned Facts – Exist because we deem it to be true. – Top or node scoped variables
  • Assigned Facts • espn_role – Identifes the role class to be applied • espn_cluster – Identifies nodes in the cluster • espn_env – workstation, dev, test, qa, prod • espn_owner – change notifications – security delegation – licensing
  • What to manage? Everything!
  • Role • Aligns with business and or IT needs • Defines everything on a node • Used for classification of nodes • Exactly one role per node • Includes profiles and releases • espn_role
  • Group everything by roles AA BB CC
  • Release Environment • Not puppet environments • Determined by application development and release cycle • espn_env
  • Group by release environment AA BB CC ProdProd QAQA DevDev WorkWork
  • Clusters • Always identical • Configured to interact with one another • espn_cluster
  • Group by clusters AA BB CC ProdProd QAQA DevDev WorkWork
  • Hiera hierarchy in theory hostname/%{hostname} cluster/%{espn_cluster_name} role_env/%{espn_role}_%{espn_env} role/%{espn_role} env/%{espn_env} network/%{espn_network} os/%{operationsystem} default
  • Hiera hierarchy in practice role_env/%{espn_role}_%{espn_env} role/%{espn_role} env/%{espn_env} network/%{espn_network} defaults
  • Classification
  • Application Middleware OS Server Storage Network IAAS PAAS SAAS Role Profiles Releases
  • Role Profiles Releases Resources Resources Classes Resources Resource Hierarchy
  • Resources • The building blocks • Everything managed is a resource • Defined in modules, used by profiles • Two resources never manage the same configuration
  • Modules • Isolate resources within the module • Never reference another module • No organizational specific logic • init.pp is a minimal installer • Reusability is key
  • R RR RR RR R R RR RR RR R R RR RR RR R R RR RR RR R R RR RR RR R Resources by module
  • R RR RR RR R R RR RR RR R R RR RR RR R R RR RR RR R R RR RR RR R Never cross modules
  • Role Profiles Releases Resources Resources Classes Resources Resource Hierarchy
  • Profile • Defines the platform • Cross module references • Enforces dependencies between modules • Class parameters prevent hiera overrides
  • Example Profile class profile::jboss_eap_6( $java_version=‘jdk_1_7_u10’, $mod_cluster_version=‘1.2.3’, $httpd_version=‘2.2.22’, ){ include “java::${java_version}” include directories::middleware class{‘mod_cluster’: version => $mod_cluster_version,} class{‘httpd’: version => $httpd_version,} class{‘jboss_eap_6’: java_home => getvar(“java::${java_version}::home”),} }
  • Role Profiles Releases Resources Resources Classes Resources Resource Hierarchy
  • Release • Special type of profile • Knows how to install on top of a profile • Deploys resources from an “artifact hash” • Cleans up removed artifacts • Driven by versioned release id – espn_release_id
  • Artifact Hash • Defines abstract resources in a release • Contract between developers and operational groups • Profile agnostic
  • Example Artifact Hash applications ‘jee-app.ear’: version: 1.2.3 espn.war: version: 1.0.0 datasources: datasource1: driver_name: ”oracle” libraries: utility.jar: version: 1.0.0
  • Example Release Class release::studio_record{ artifact_hash = hiera(‘artifacts’,undef,”release/${espn_release_id}”) #modify the artifact hash so it can be used with create_resources #set organization specific parameters such URL’s to the artifact repo resources {‘jboss7_datasource: purge => true,} create_resources(jboss7_datasource, $artifact_hash[‘datasources’]) resources {‘jboss7_deployment: purge => true,} create_resources(jboss7_deployment, $artifact_hash[‘applications’]) }
  • Role Profiles Releases Resources Resources Classes Resources Resource Hierarchy
  • Example Role class role::studio_record{ include profile::base include profile::jboss_eap_6 include profile::mysql_5 include release::studio_record }
  • site.pp # assigned facts retrieved from external datasources $espn_cluster_nodes = espn_cluster_nodes() $espn_release_id = hiera(‘espn_release_id’) # single assigned fact drives 100% of classification node default { include “role::${espn_role}” }
  • Defining a Node
  • MCollective
  • Colonol John Boyd • Military Strategist • OODA Loop
  • Observations Decision (Hypothesis) Action (Test) Observe Orient Decide Act Feedback Feedback Unfolding Circumstances Outside Information Unfolding Interaction With Environment Cultural Traditions Analysis & Synthesis Previous Experience New Information Genetic Heritage Feedback Implicit Guidance & Control Implicit Guidance & Control
  • OODA Loop "Time is the dominant parameter. The pilot who goes through the OODA cycle in the shortest time prevails because his opponent is caught responding to situations that have already changed.“ Harry Hillaker (chief designer of the F-16)
  • Observations Decision (Hypothesis) Action (Test) Observe Orient Decide Act Feedback: Puppet runs on dev/test puppet environments Feedback Unfolding Circumstances Outside Information Unfolding Interaction With Environment Cultural Traditions Analysis & Synthesis Previous Experience New Information Genetic Heritage Feedback Implicit Guidance & Control Implicit Guidance & Control
  • Observations Decision (Hypothesis) Action (Test) Observe Orient Decide Act Feedback: Puppet runs on dev/test puppet environments Feedback: (Test) Noop puppet run on production Unfolding Circumstances Outside Information Unfolding Interaction With Environment Cultural Traditions Analysis & Synthesis Previous Experience New Information Genetic Heritage Feedback Implicit Guidance & Control Implicit Guidance & Control
  • Noop Puppet Runs • Hourly cron job – “mco puppet runall 20 –noop” • Dashboard displays unresponsive nodes – no_longer_reporting_cutoff
  • Observations Decision (Hypothesis) Action (Test) Observe Orient Decide Act Feedback: Puppet runs on dev/test puppet environments Feedback: (Test) Noop puppet run on production Unfolding Circumstances Outside Information Unfolding Interaction With Environment Puppet Dashboard Cultural Traditions Analysis & Synthesis Previous Experience New Information Genetic Heritage Feedback: (Action) No-noop puppet run on production Implicit Guidance & Control Implicit Guidance & Control
  • Configuration Changes • Always noop first • Always target nodes with filters • Always use tags • Validate changes then apply –no-noop mco puppet runonce --no-noop -C jboss-eap-6 --tag initscript --tag jboss-eap-6
  • Observations Decision (Hypothesis) Action (Test) Observe Orient Decide Act Feedback: Puppet runs on dev/test puppet environments Feedback: (Test) Noop puppet run on production Unfolding Circumstances Outside Information Cultural Traditions Analysis & Synthesis Previous Experience New Information Genetic Heritage Feedback: (Action) No-noop puppet run on production Implicit Guidance & Control: MCollective Agent Plugins Implicit Guidance & Control Unfolding Interaction With Environment Puppet Dashboard
  • 20/20 Hindsight
  • Do Differently? • Implement MCollective first – Security – Sub-collectives – Availability • Plan for developer dashboards • Implement Custom ENC
  • Nice to have? • Puppet runs that span multiple nodes – Allocate disk on node A, create shared filesystem on node B • Resources automatically tagged with catalog unique identifiers • Role based access control for dashboard
  • Questions?