Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Puppet ENC – a ServiceNow Scoped Application; Richard Romanus

45 views

Published on

Puppet Camp America West; 25 June 2020

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Puppet ENC – a ServiceNow Scoped Application; Richard Romanus

  1. 1. Puppet ENC – A ServiceNow Scoped Application Richard Romanus
  2. 2. Richard Romanus Sr. Software Engineer IaaS Automated Solutions T-Mobile • 2013 University of Washington - BSCSSE • 2015 – Present (5yr) T-Mobile - Software Engineer • 2009 – 2014 (5yr) Ericsson - Switch Technician (@ Sprint) • 1999 – 2009 (10yr) Sprint - Switch Technician • 1996 – 1999 (3yr) AirTouch Cellular - NOC Technician • 1990 – 1994 (4yr) US Navy (DDG-996) - Electronic Warfare Technician [EW] About Me
  3. 3. The Problem Richard Romanus Data Islands Business Data Configuration Data
  4. 4. The Problem Richard Romanus Configuration Management - FACTs - Hiera - Params
  5. 5. The Problem Richard Romanus Configuration Management - FACTs - Hiera - Params Business Data - Compliance Data - Business Information - Location Information
  6. 6. The Problem Richard Romanus Configuration Management - FACTs - Hiera - Params Business Data - Compliance Data - Business Information - Location Information
  7. 7. The Solution Richard Romanus Configuration Management - FACTs - Hiera - Params Business Data - Compliance Data - Business Information - Location Information
  8. 8. The Solution • A custom Puppet Node Classifier (PNC) Ruby script on every compile master • A custom External Node Classifier (ENC) JavaScript that interfaces with the companies global CMDB tables and with our scoped application tables • Multiple scoped Application Tables to store the Puppet Role and Profile data • Standard REST API calls to communicate between the PNC and ENC Richard Romanus ServiceNow Puppet Master Application Tables ENC PNC REST JavaScript Ruby
  9. 9. ServiceNow Puppet ENC The Architecture Puppet Master Scoped Application Tables ENC PNC 1. Check for Overrides 2. Get FACTs from Puppet DB 3. Get Puppet data from Puppet Console 4. Get Business Data from ENC 5. Merge values and Return Servername environment: production classes: puppet_roles::common: {} parameters: tmosn_role_type: database tmosn_business_app_name: Puppet Enterprise tmosn_sox: 'no' tmosn_pci: 'no' tmosn_usedfor: Development tmosn_install_status: installed tmosn_location: Seattle Global Tables (Global Space) (Scoped Application Space) Server Puppet Agent
  10. 10. The Architecture (ver 1.0) Puppet ENC External Node Classifier Servers Puppet Agent Code Base Puppet Database last_puppet_run Server FACTS Puppet Compile Masters Puppet Node Classifier Latest Code Releases Puppet Console Puppet Cluster FACTs FACTs FACTs Puppet Specific Classes ENC Endpoints GET ENC Server Table ENC Role Table ServiceNow Global Server Table Global Business Data Table ENC Environments Table (Global Space) Global Relationship Tables (Scoped Application Space) JavaScript RUBY Richard Romanus
  11. 11. The Results Pros: • It worked at scale! • Mostly Stable • Good performance Richard Romanus Cons: • Very limited • No overriding • No table ACLs • No API
  12. 12. The Results • Implemented Roles & Profiles • The ServiceNow Global Business information provided immediate default Role classification. • However, limited access to the role field in ServiceNow, limited its usability. Richard Romanus puppet roles | +- manifests | | | +- apps | | | | | +- <BUSINESS_NAME#1> | | | +- default.pp | | | +- <role1>.pp | | | +- <role2>.pp . | | . | +- <BUSINESS_NAME#2> . | +- default.pp | +- <role1>.pp | +- common.pp
  13. 13. The Results Richard Romanus Puppet ENC External Node Classifier ENC Endpoint GET ServiceNow Environments Environments production test_env Roles puppet puppet splunk compiler database master puppet_roles::puppet::compiler puppet_roles::puppet::database puppet_roles::splunk::master splunk . . . database . . . puppet_roles::splunk::database . . . Business Name Role Puppet Classes Servers serverabc serverxyz server123 production production production server456 . . . test_env . . . Server Name Env compiler database database master . . . Role puppet puppet splunk splunk . . . Business . . . server123 environment: production classes: puppet_roles::apps::splunk::database: {} parameters: tmosn_role_type: database tmosn_business_app_name: Splunk tmosn_sox: 'no' tmosn_pci: 'no' tmosn_usedfor: Development tmosn_install_status: installed tmosn_location: Seattle puppet roles | +- manifests | | | +- apps . | | . | +- puppet . | | +- default.pp | | +- compiler.pp | | +- database.pp | | | +- splunk | +- default.pp | +- database.pp | +- master.pp | +- common.pp class puppet_roles::apps::splunk::database { include puppet_profile::apps::splunk } Puppet Compile Masters PNC
  14. 14. The Results Richard Romanus Configuration ManagementBusiness Data Puppet ENC - Version 1.0
  15. 15. The Results Richard Romanus Configuration ManagementBusiness Data Puppet ENC - Version 1.0 - Automated Patching User Interface - Automated Patching Workflow - FACTs - Ops Server Check Data - Puppet Last Run DataBusiness Data
  16. 16. Puppet ENC – Version 2.0 Richard Romanus Configuration ManagementBusiness Data Puppet ENC - Version 2.0 - Automated Patching User Interface - Automated Patching Workflow - FACTs - Ops Server Check Data - Puppet Last Run DataBusiness Data Server Status Data
  17. 17. Version 2.0 Solution: • Updated the PNC and ENC to gather and store the of Last Puppet Run data and Operations Server Check data in the ENC tables • Created a new ‘Puppet Patch Ready’API to determine if a server is ready for patching • Created a wrapper script that runs the Operations Server Check script and saves the results to a FACT file. Richard Romanus Problem: • Make automated patching more stable and reliable.
  18. 18. The Architecture – Version 2.0 Puppet ENC External Node Classifier Servers Puppet Agent Code Base Puppet Database last_puppet_run Server FACTS Puppet Compile Masters Puppet Node Classifier Latest Code Releases Puppet Console Puppet Cluster FACTs FACTs FACTs Puppet Specific Classes ENC Endpoints POST ENC Server Table ENC Role Table ServiceNow Global Server Table Global Business Data Table ENC Environments Table Global Relationship Tables JavaScript RUBY Patch Ready Endpoint Automated Patching ServiceNow Automated Patching Workflow JavaScript External Container API Web Interface Sinatra GET puppet last run data [new] ops server check data [new] Overrides File RUBY run_ops_chk_script.rb ops_server_chk.sh /etc/puppetlabs/facter/facts.d/ tmo_ops_chk.json tmo_ops_chk GET
  19. 19. The Results Pros: • Gathers last Puppet Run data • Gathers Ops Server Check status • Provides internal API for checking on server status • Provides an external API for user interface with ENC • Provides a method for overriding ENC values in the PNC Richard Romanus Cons: • Still no table ACLs • PNC becoming very customized • Default Role entry is manual • No auto deletion of decom servers • No caching of ServiceNow data (Defaults can be dangerous!!!)
  20. 20. The Results Time saved with Automated Patching: 20min/server x 10k server = +3k hours +3k hours of work saved per Quarter +13k hours of work saved per Year Richard Romanus
  21. 21. Timeline Richard Romanus • Release: Dec. 2018 • Provide Business Data in ServiceNow to Puppet • Implemented Roles/Profiles • Release: Aug. 2019 • Provide Server Status Data from Puppet, to ServiceNow • Provided a new ‘Patch Ready’ API for checking server status • Release: July 2020 • Update model for gathering Business Data from ServiceNow • Improved the storing, filtering, and formatting of the Ops Server Check data
  22. 22. What’s Next – Ver 3.0 Richard Romanus • Improved stability and performance of both ENC and PNC • ACLs!!! (Finally) • Auto default role entry for every Business Server type in the ENC Role table • Store Operations ‘Server Check’ data into its own table • Improved PNC handling of overrides
  23. 23. What’s Next Richard Romanus • Capture specified server FACTs and store them in ServiceNow. • A User Interface (UI) page for users to access ENC settings/data (role, env, etc. ) • A UI page for users to see other server FACTS (software version, status, etc.)
  24. 24. Challenges Richard Romanus • Developing in ServiceNow for the first time. (with JavaScript) • Working with non-standard data and fields • Providing some of the Puppet Console functionality with the ENC
  25. 25. Closing Richard Romanus Sr. Engineer, Software – IaaS Automated Solutions Rick.Romanus@T-Mobile.com Richard Romanus Thank You!!!
  26. 26. Q & A Richard Romanus Q & A

×