Puppe%ng	
  
in	
  a	
  	
  
Highly	
  Regulated	
  Industry	
  

Marinus	
  Damm	
  
marinus.damm@pgn.com	
  
Every	
  business	
  is	
  regulated…	
  
•  Labor	
  regs	
  

	
  	
  	
  	
  	
  	
  	
  minimum	
  wage,	
  paid	
  si...
“Highly	
  Regulated”	
  
as	
  it	
  pertains	
  to	
  system	
  administra#on	
  

Ac#ve	
  Monitoring	
  
Level	
  of	
...
Ac#ve	
  Monitoring	
  
Level	
  of	
  Detail	
  of	
  regs	
  
Is	
  a	
  Policy	
  in	
  place?	
  
	
  	
  	
  
Are	
  ...
Ac#ve	
  Monitoring	
  
Level	
  of	
  Detail	
  of	
  regs	
  
• 
• 
• 
• 
• 

Separa%on	
  of	
  du%es	
  
Data	
  acces...
Who	
  Is	
  The	
  Boss?	
  
FERC:	
  Federal	
  Energy	
  Regulatory	
  Commission	
  
	
  
and	
  its	
  designee	
  	
...
Power	
  Flow	
  

to	
  Congress	
  

to	
  FERC	
  

to	
  NERC	
  

United States
Constitution	
  
Art. 1, Sec. 8
“to r...
Power	
  Surge	
  
•  Used	
  to	
  be	
  that	
  NERC	
  made	
  sugges%ons	
  only	
  
	
  
•  As	
  electric	
  power	
...
How	
  Can	
  Companies	
  Get	
  On	
  Track?	
  
Obviously	
  all	
  these	
  NERC	
  P&Ps	
  will	
  	
  
	
  	
  	
  	...
Coincidentally,	
  on	
  a	
  Parallel	
  Track…	
  
	
  
Aber	
  the	
  …	
  excesses	
  …	
  of	
  the	
  dot-­‐com	
  e...
Two	
  Tracks	
  Align	
  
•  The	
  FERC	
  Reliability	
  Standards,	
  plus	
  
•  The	
  MBAs’	
  counteradack	
  on	
...
Change	
  Management	
  
The objective of change management ... is to
ensure that standardized methods and
procedures are ...
Simplified	
  Example	
  of	
  Change	
  Flow	
  
1. 
2. 
3. 

4. 
5. 
6. 
7. 
8. 
9. 

Sysadmin	
  writes	
  proposal	
  f...
And	
  that's	
  just	
  for	
  the	
  kiddie	
  systems	
  

The	
  systems	
  handling	
  the	
  power	
  grid	
  proper...
Ques%ons?	
  Correc%ons?	
  
Why	
  do	
  we	
  puppet	
  the	
  way	
  we	
  do?	
  

Power

History



Accountability
Why	
  do	
  we	
  puppet	
  the	
  way	
  we	
  do?	
  

Power

History



Accountability
Why	
  do	
  we	
  puppet	
  the	
  way	
  we	
  do?	
  

Power

History



Accountability
PGE	
  –	
  PEC	
  

(Puppet	
  Enterprise	
  Components)	
  
puppet	
  master	
  
hdp	
  
unk	
  
puppet	
  console	
  
D...
PGE	
  –	
  Puppet	
  Environments	
  
Every	
  node	
  is	
  in	
  one	
  and	
  only	
  one	
  environment.	
  
	
  
	
 ...
PGE	
  -­‐	
  Promo%on	
  
dev	
  –	
  	
  	
  80	
  or	
  so	
  systems	
  
	
  	
  deploy,	
  then	
  watch	
  puppet	
 ...
PGE	
  -­‐	
  Keeping	
  Tabs	
  With	
  Custom	
  Facts	
  
•  third-­‐party	
  sobware	
  
	
  
•  locate	
  inconsisten...
PGE	
  -­‐	
  Custom	
  Facts	
  Defined	
  
# synergy_status.rb
Facter.add("synergy_installed") do
setcode do
File.executa...
PGE	
  -­‐	
  Custom	
  Facts	
  Realized	
  
synergy_installed => true
synergy_joined => true
synergy_status => connected...
PGE	
  -­‐	
  Custom	
  Facts	
  Available	
  
#!/bin/bash
FACT=$1
VALUE=$2
curl -X GET -H "Accept: application/json" 
--c...
PGE	
  -­‐	
  Really	
  Simple	
  Modules	
  
•  A	
  few	
  module-­‐level	
  variables	
  

	
  	
  probably	
  set	
  f...
PGE	
  -­‐	
  Really	
  Similar	
  Modules	
  
•  If	
  you’ve	
  seen	
  one,	
  you’ve	
  seen	
  ‘em	
  all	
  
	
  
• ...
PGE	
  –	
  Common	
  Module	
  Layout	
  
class synergy {
if $::synergy_installed != 'true' {
warning('This node does not...
coda	
  
Puppet	
  Enterprise	
  	
  
gives	
  us	
  Power	
  
	
  lets	
  us	
  deal	
  with	
  our	
  History	
  
	
  ea...
PGE	
  Service	
  Territory	
  

St. Helens

PGE SERVICE TERRITORY
30

Scapoose

26

Banks

North Plains
5
30

Fairview

4...
Puppeting in a Highly Regulated Industry
Upcoming SlideShare
Loading in...5
×

Puppeting in a Highly Regulated Industry

579

Published on

"Puppeting in a Highly Regulated Industry" by Marinus Damm of PGE at Puppet Camp Portland 2014.

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
579
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
20
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Puppeting in a Highly Regulated Industry

  1. 1. Puppe%ng   in  a     Highly  Regulated  Industry   Marinus  Damm   marinus.damm@pgn.com  
  2. 2. Every  business  is  regulated…   •  Labor  regs                minimum  wage,  paid  sick  leave,  hours  and  breaks   •  Money  regs                income  tax  withholding,  accoun%ng  prac%ces  (SOX)   •  Safety  regs              protec%ve  equipment,  training,  repor%ng  accidents   •  Licensing  regs                  business  license,  HAZMAT,  serving  liquor  
  3. 3. “Highly  Regulated”   as  it  pertains  to  system  administra#on   Ac#ve  Monitoring   Level  of  Detail  of  regs  
  4. 4. Ac#ve  Monitoring   Level  of  Detail  of  regs   Is  a  Policy  in  place?         Are  Procedures  to  implement  that  in  place?         Do  employees  receive  Training  on  P&P?         Can  you  Prove  that  P&P  are  followed?  
  5. 5. Ac#ve  Monitoring   Level  of  Detail  of  regs   •  •  •  •  •  Separa%on  of  du%es   Data  access   System  access  %meouts   Least  privilege   Passwords   •  “Passwords  shall  be  at  least  eight  characters  in  length,  and  shall   include  at  least  one  uppercase  character,  one  lowercase  character,   one  numeral,  and  one  special  character.”  
  6. 6. Who  Is  The  Boss?   FERC:  Federal  Energy  Regulatory  Commission     and  its  designee         NERC:  North  American  Electric  Reliability  Corp.    
  7. 7. Power  Flow   to  Congress   to  FERC   to  NERC   United States Constitution   Art. 1, Sec. 8 “to regulate commerce among the several states” Do  this,  or  else.   Or  else  what?      $$  Fines,  baby…  fines.    
  8. 8. Power  Surge   •  Used  to  be  that  NERC  made  sugges%ons  only     •  As  electric  power  suppliers  were  deregulated,   the  need  for  predictable  delivery  increased     •  In  2006,  FERC  designated  NERC  as  the  na%onal   ‘Electric  Reliability  Organiza%on’         •  NERC’s  sugges%ons  are  now  Standards.    
  9. 9. How  Can  Companies  Get  On  Track?   Obviously  all  these  NERC  P&Ps  will            massively  increase  produc%vity….    or  not     So  how  do  we  deal  with  the  new  strictures?       à  We  need  a  framework!   Anybody  got  one?    
  10. 10. Coincidentally,  on  a  Parallel  Track…     Aber  the  …  excesses  …  of  the  dot-­‐com  era,  the   business  side  wanted  to  rein  in  IT   Information Technology Infrastructure Library (ITIL)
  11. 11. Two  Tracks  Align   •  The  FERC  Reliability  Standards,  plus   •  The  MBAs’  counteradack  on  Techies                                                                                                              gave  us     *            CHANGE  MANAGEMENT  
  12. 12. Change  Management   The objective of change management ... is to ensure that standardized methods and procedures are used for efficient and prompt handling of all changes to control IT infrastructure, in order to minimize the number and impact of any related incidents upon service. from  the  wikipedia  ar#cle      
  13. 13. Simplified  Example  of  Change  Flow   1.  2.  3.  4.  5.  6.  7.  8.  9.  Sysadmin  writes  proposal  for  new  sehng   Different  sysadmin  or  manager  agrees   Sysdmin  becomes  Change  Owner,  engages  CM  tool:   Describe  business  effects  of  doing/not  doing   Iden%fy  systems/services/apps/users  affected   Design  the  procedure  (including  verifica%on  and  backout  plan)   Design  and  execute  a  test  plan   Change  Owner  schedules  %me  for  change   Every  affected  IT  group  assesses  change  impact   Every  affected  system/service/app/user  reviews  change  and  authorizes   Change  Board  considers  all  imminent  changes,  weighs  risks  and   conflicts,  approves  change  for  implementa%on   Change  Owner  executes  procedure  at  scheduled  %me   Change  Owner  completes  change  record  
  14. 14. And  that's  just  for  the  kiddie  systems   The  systems  handling  the  power  grid  proper  are  a  whole  'nuther  animal.    
  15. 15. Ques%ons?  Correc%ons?  
  16. 16. Why  do  we  puppet  the  way  we  do?   Power History Accountability
  17. 17. Why  do  we  puppet  the  way  we  do?   Power History Accountability
  18. 18. Why  do  we  puppet  the  way  we  do?   Power History Accountability
  19. 19. PGE  –  PEC   (Puppet  Enterprise  Components)   puppet  master   hdp   unk   puppet  console   DB   webservice   dev   puppet  db   tst   prd   webservice   database   All  three  are  VMs,  2  cores/8GB  ram,  RHEL  
  20. 20. PGE  –  Puppet  Environments   Every  node  is  in  one  and  only  one  environment.       The  puppetmaster  has  three  parallel  directory  structures:        /etc/puppetlabs/puppet/environments/[dev|tst|prd] unk   dev       The  directories  are  all  clones  of  a  single  git  repo,  and  pull   from  that  remote  repo  for  manifest  and  module  updates.   tst   prd  
  21. 21. PGE  -­‐  Promo%on   dev  –      80  or  so  systems      deploy,  then  watch  puppet  reports  to  verify        what’s  changing...        and  that  things  don’t  keep  changing.                                     tst  –      around  100  systems    Collect  ‘test  results’  for  inclusion  in  the  Change         dev   tst   change  management  bar   prd  –    around  120  systems    Several  new  or  revised  modules  are  promoted        as  a  single  change  –  an  ‘OS  Release’   prd  
  22. 22. PGE  -­‐  Keeping  Tabs  With  Custom  Facts   •  third-­‐party  sobware     •  locate  inconsistency     •  feed  our  manifests  and  templates  
  23. 23. PGE  -­‐  Custom  Facts  Defined   # synergy_status.rb Facter.add("synergy_installed") do setcode do File.executable?("/usr/bin/syninfo") end end Facter.add("synergy_joined") do confine :synergy_installed => true setcode do domain = Facter::Util::Resolution.exec(‘syninfo --domain') domain.eql?(“it.pgn.com") end end Facter.add("synergy_status") do setcode do if Facter.value(:synergy_installed) if Facter.value(:synergy_joined) Facter::Util::Resolution.exec(‘syninfo --mode') else "Not_Joined" end else "Not_Installed" end end end
  24. 24. PGE  -­‐  Custom  Facts  Realized   synergy_installed => true synergy_joined => true synergy_status => connected These  are  reportable/searchable  via  PuppetDB.  
  25. 25. PGE  -­‐  Custom  Facts  Available   #!/bin/bash FACT=$1 VALUE=$2 curl -X GET -H "Accept: application/json" --cacert /home/marinus/puppetInventory/ca.pem --cert /home/marinus/puppetInventory/cert.pem --key /home/marinus/puppetInventory/private.pem 'https://puppetdb:8081/v2/facts/'${FACT} --data-urlencode 'query=["not", ["=", "value", "'${VALUE}'"]]' Just  show  me  systems  where  Synergy  is  not  ‘connected’:            /facts_without_value.sh synergy_status connected
  26. 26. PGE  -­‐  Really  Simple  Modules   •  A  few  module-­‐level  variables      probably  set  from  facts  or  literals,  not  computed     •  A  File  resource      usually  a  .conf  file     •  A  Service  resource      subscribed  to  the  file  resource  
  27. 27. PGE  -­‐  Really  Similar  Modules   •  If  you’ve  seen  one,  you’ve  seen  ‘em  all     •  Every  file’s  content  comes  from  a  template      even  if  there’s  no  variability     •  puppet-­‐lint      helps  us  enforce  textual  appearance    
  28. 28. PGE  –  Common  Module  Layout   class synergy { if $::synergy_installed != 'true' { warning('This node does not have Synergy installed') } else { $os = $::operatingsystem $filegroup = $os ? { /AIX/ => 'system', /RedHat/ => 'root', default => 'unk', } File { ensure => file, mode => '0644', owner => 'root', group => $filegroup, } file { '/etc/synergy/gid.ignore': content => template ("synergy/gid.ignore.${os}.erb"), } file { '/etc/synergy/synergy.conf': content => template ("synergy/synergy.conf.${os}.erb"), } service { 'synergy': ensure => running, enable => true, subscribe => File['/etc/synergy/synergy.conf'], } } }
  29. 29. coda   Puppet  Enterprise     gives  us  Power    lets  us  deal  with  our  History    eases  Accountability   Marinus  Damm   marinus.damm@pgn.com  
  30. 30. PGE  Service  Territory   St. Helens PGE SERVICE TERRITORY 30 Scapoose 26 Banks North Plains 5 30 Fairview 47 84 26 84 10 Milwaukie Tigard King City Scholls YAMHILL CO Tualatin Carlton 47 99W Dayton 212 Sandy 219 26 Eagle Creek 26 Zigzag 211 99E Government Camp Estacada Canby Barlow 26 HOOD RIVER CO WASCO CO Aurora St. Paul 224 Mulino 99E Hubbard 18 99W Brightwood 211 Oregon City Willsonville 26 224 213 MARION CO YAMHILL CO Lafayette McMinnville YAMHILL CO Newberg Dundee Carver West Linn 5 99W MULTNOMAH CO CLACKAMAS CO Johnson City 43 205 Yamhill VE R CO MA H CO Boring Lake Oswego Rivergrove 47 219 Happy Valley 205 217 WASHINGTON CO NO 99E Beaverton 210 T UL 219 84 30 26 10 Gaston Troutdale Gresham RI Portland 8 Wood Village M 405 OD Hillsboro 8 Cornelius HO Forest Grove 221 Colton 211 Woodburn 211 5 AC CO A M KA O N C MA 18 Willamina Molalla CL Amity Sheridan RI 213 Gervais S O Mt. Angel Grand Ronde 99E 214 213 Marquam Scotts Mills Keizer 22 Silverton 221 213 99W 22 Salem 5 214 WASHINGTON COUNTY 214 MULTNOMAH COUNTY YAMHILL COUNTY Counties Columbia CLACKAMAS COUNTY Washington Multnomah POLK COUNTY MARION COUNTY Yamhill Clackamas Marion Polk Turner •  About  a  million  points  of  delivery   •  1400  servers  (Windows  &  UNIX)   •  Sixty  people  in  IT  Infrastructure            …  and  nice  benefits  
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×