Puppeting in a Highly Regulated Industry


Published on

"Puppeting in a Highly Regulated Industry" by Marinus Damm of PGE at Puppet Camp Portland 2014.

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Puppeting in a Highly Regulated Industry

  1. 1. Puppe%ng   in  a     Highly  Regulated  Industry   Marinus  Damm   marinus.damm@pgn.com  
  2. 2. Every  business  is  regulated…   •  Labor  regs                minimum  wage,  paid  sick  leave,  hours  and  breaks   •  Money  regs                income  tax  withholding,  accoun%ng  prac%ces  (SOX)   •  Safety  regs              protec%ve  equipment,  training,  repor%ng  accidents   •  Licensing  regs                  business  license,  HAZMAT,  serving  liquor  
  3. 3. “Highly  Regulated”   as  it  pertains  to  system  administra#on   Ac#ve  Monitoring   Level  of  Detail  of  regs  
  4. 4. Ac#ve  Monitoring   Level  of  Detail  of  regs   Is  a  Policy  in  place?         Are  Procedures  to  implement  that  in  place?         Do  employees  receive  Training  on  P&P?         Can  you  Prove  that  P&P  are  followed?  
  5. 5. Ac#ve  Monitoring   Level  of  Detail  of  regs   •  •  •  •  •  Separa%on  of  du%es   Data  access   System  access  %meouts   Least  privilege   Passwords   •  “Passwords  shall  be  at  least  eight  characters  in  length,  and  shall   include  at  least  one  uppercase  character,  one  lowercase  character,   one  numeral,  and  one  special  character.”  
  6. 6. Who  Is  The  Boss?   FERC:  Federal  Energy  Regulatory  Commission     and  its  designee         NERC:  North  American  Electric  Reliability  Corp.    
  7. 7. Power  Flow   to  Congress   to  FERC   to  NERC   United States Constitution   Art. 1, Sec. 8 “to regulate commerce among the several states” Do  this,  or  else.   Or  else  what?      $$  Fines,  baby…  fines.    
  8. 8. Power  Surge   •  Used  to  be  that  NERC  made  sugges%ons  only     •  As  electric  power  suppliers  were  deregulated,   the  need  for  predictable  delivery  increased     •  In  2006,  FERC  designated  NERC  as  the  na%onal   ‘Electric  Reliability  Organiza%on’         •  NERC’s  sugges%ons  are  now  Standards.    
  9. 9. How  Can  Companies  Get  On  Track?   Obviously  all  these  NERC  P&Ps  will            massively  increase  produc%vity….    or  not     So  how  do  we  deal  with  the  new  strictures?       à  We  need  a  framework!   Anybody  got  one?    
  10. 10. Coincidentally,  on  a  Parallel  Track…     Aber  the  …  excesses  …  of  the  dot-­‐com  era,  the   business  side  wanted  to  rein  in  IT   Information Technology Infrastructure Library (ITIL)
  11. 11. Two  Tracks  Align   •  The  FERC  Reliability  Standards,  plus   •  The  MBAs’  counteradack  on  Techies                                                                                                              gave  us     *            CHANGE  MANAGEMENT  
  12. 12. Change  Management   The objective of change management ... is to ensure that standardized methods and procedures are used for efficient and prompt handling of all changes to control IT infrastructure, in order to minimize the number and impact of any related incidents upon service. from  the  wikipedia  ar#cle      
  13. 13. Simplified  Example  of  Change  Flow   1.  2.  3.  4.  5.  6.  7.  8.  9.  Sysadmin  writes  proposal  for  new  sehng   Different  sysadmin  or  manager  agrees   Sysdmin  becomes  Change  Owner,  engages  CM  tool:   Describe  business  effects  of  doing/not  doing   Iden%fy  systems/services/apps/users  affected   Design  the  procedure  (including  verifica%on  and  backout  plan)   Design  and  execute  a  test  plan   Change  Owner  schedules  %me  for  change   Every  affected  IT  group  assesses  change  impact   Every  affected  system/service/app/user  reviews  change  and  authorizes   Change  Board  considers  all  imminent  changes,  weighs  risks  and   conflicts,  approves  change  for  implementa%on   Change  Owner  executes  procedure  at  scheduled  %me   Change  Owner  completes  change  record  
  14. 14. And  that's  just  for  the  kiddie  systems   The  systems  handling  the  power  grid  proper  are  a  whole  'nuther  animal.    
  15. 15. Ques%ons?  Correc%ons?  
  16. 16. Why  do  we  puppet  the  way  we  do?   Power History Accountability
  17. 17. Why  do  we  puppet  the  way  we  do?   Power History Accountability
  18. 18. Why  do  we  puppet  the  way  we  do?   Power History Accountability
  19. 19. PGE  –  PEC   (Puppet  Enterprise  Components)   puppet  master   hdp   unk   puppet  console   DB   webservice   dev   puppet  db   tst   prd   webservice   database   All  three  are  VMs,  2  cores/8GB  ram,  RHEL  
  20. 20. PGE  –  Puppet  Environments   Every  node  is  in  one  and  only  one  environment.       The  puppetmaster  has  three  parallel  directory  structures:        /etc/puppetlabs/puppet/environments/[dev|tst|prd] unk   dev       The  directories  are  all  clones  of  a  single  git  repo,  and  pull   from  that  remote  repo  for  manifest  and  module  updates.   tst   prd  
  21. 21. PGE  -­‐  Promo%on   dev  –      80  or  so  systems      deploy,  then  watch  puppet  reports  to  verify        what’s  changing...        and  that  things  don’t  keep  changing.                                     tst  –      around  100  systems    Collect  ‘test  results’  for  inclusion  in  the  Change         dev   tst   change  management  bar   prd  –    around  120  systems    Several  new  or  revised  modules  are  promoted        as  a  single  change  –  an  ‘OS  Release’   prd  
  22. 22. PGE  -­‐  Keeping  Tabs  With  Custom  Facts   •  third-­‐party  sobware     •  locate  inconsistency     •  feed  our  manifests  and  templates  
  23. 23. PGE  -­‐  Custom  Facts  Defined   # synergy_status.rb Facter.add("synergy_installed") do setcode do File.executable?("/usr/bin/syninfo") end end Facter.add("synergy_joined") do confine :synergy_installed => true setcode do domain = Facter::Util::Resolution.exec(‘syninfo --domain') domain.eql?(“it.pgn.com") end end Facter.add("synergy_status") do setcode do if Facter.value(:synergy_installed) if Facter.value(:synergy_joined) Facter::Util::Resolution.exec(‘syninfo --mode') else "Not_Joined" end else "Not_Installed" end end end
  24. 24. PGE  -­‐  Custom  Facts  Realized   synergy_installed => true synergy_joined => true synergy_status => connected These  are  reportable/searchable  via  PuppetDB.  
  25. 25. PGE  -­‐  Custom  Facts  Available   #!/bin/bash FACT=$1 VALUE=$2 curl -X GET -H "Accept: application/json" --cacert /home/marinus/puppetInventory/ca.pem --cert /home/marinus/puppetInventory/cert.pem --key /home/marinus/puppetInventory/private.pem 'https://puppetdb:8081/v2/facts/'${FACT} --data-urlencode 'query=["not", ["=", "value", "'${VALUE}'"]]' Just  show  me  systems  where  Synergy  is  not  ‘connected’:            /facts_without_value.sh synergy_status connected
  26. 26. PGE  -­‐  Really  Simple  Modules   •  A  few  module-­‐level  variables      probably  set  from  facts  or  literals,  not  computed     •  A  File  resource      usually  a  .conf  file     •  A  Service  resource      subscribed  to  the  file  resource  
  27. 27. PGE  -­‐  Really  Similar  Modules   •  If  you’ve  seen  one,  you’ve  seen  ‘em  all     •  Every  file’s  content  comes  from  a  template      even  if  there’s  no  variability     •  puppet-­‐lint      helps  us  enforce  textual  appearance    
  28. 28. PGE  –  Common  Module  Layout   class synergy { if $::synergy_installed != 'true' { warning('This node does not have Synergy installed') } else { $os = $::operatingsystem $filegroup = $os ? { /AIX/ => 'system', /RedHat/ => 'root', default => 'unk', } File { ensure => file, mode => '0644', owner => 'root', group => $filegroup, } file { '/etc/synergy/gid.ignore': content => template ("synergy/gid.ignore.${os}.erb"), } file { '/etc/synergy/synergy.conf': content => template ("synergy/synergy.conf.${os}.erb"), } service { 'synergy': ensure => running, enable => true, subscribe => File['/etc/synergy/synergy.conf'], } } }
  29. 29. coda   Puppet  Enterprise     gives  us  Power    lets  us  deal  with  our  History    eases  Accountability   Marinus  Damm   marinus.damm@pgn.com  
  30. 30. PGE  Service  Territory   St. Helens PGE SERVICE TERRITORY 30 Scapoose 26 Banks North Plains 5 30 Fairview 47 84 26 84 10 Milwaukie Tigard King City Scholls YAMHILL CO Tualatin Carlton 47 99W Dayton 212 Sandy 219 26 Eagle Creek 26 Zigzag 211 99E Government Camp Estacada Canby Barlow 26 HOOD RIVER CO WASCO CO Aurora St. Paul 224 Mulino 99E Hubbard 18 99W Brightwood 211 Oregon City Willsonville 26 224 213 MARION CO YAMHILL CO Lafayette McMinnville YAMHILL CO Newberg Dundee Carver West Linn 5 99W MULTNOMAH CO CLACKAMAS CO Johnson City 43 205 Yamhill VE R CO MA H CO Boring Lake Oswego Rivergrove 47 219 Happy Valley 205 217 WASHINGTON CO NO 99E Beaverton 210 T UL 219 84 30 26 10 Gaston Troutdale Gresham RI Portland 8 Wood Village M 405 OD Hillsboro 8 Cornelius HO Forest Grove 221 Colton 211 Woodburn 211 5 AC CO A M KA O N C MA 18 Willamina Molalla CL Amity Sheridan RI 213 Gervais S O Mt. Angel Grand Ronde 99E 214 213 Marquam Scotts Mills Keizer 22 Silverton 221 213 99W 22 Salem 5 214 WASHINGTON COUNTY 214 MULTNOMAH COUNTY YAMHILL COUNTY Counties Columbia CLACKAMAS COUNTY Washington Multnomah POLK COUNTY MARION COUNTY Yamhill Clackamas Marion Polk Turner •  About  a  million  points  of  delivery   •  1400  servers  (Windows  &  UNIX)   •  Sixty  people  in  IT  Infrastructure            …  and  nice  benefits