Puppet
and
Telefónica I+D

Puppet Camp Barcelona 2013

Xavi Carrillo
01
Introduction
01
     Introduction


             Who am I?

            § Puppet user for the last 2 years
            § Working at Telefonica I+D for 4 months
            § Telefonica Digital curently involved in a process
              of modernization
            § Helping on that by puppetizingTelefónica I+D




                                                                  3
01
     What do we do at I+D?




                             4
01
     Initiatives




                   5
02
The Problem Domain
02
     Environment

                                 x7




                   Development



                     Staging




                    Production




                                      7
02
     The problem domain




     § Machine Setups: manually (howtos,etc)
     § Environments: docs, wikis.
     § Releases: Hand over an RPM…

     § At least one SysAdmin per initiative




                                               8
02
      The needs




     • Infrastructure for many sysadmins, initiatives and environments.
     • Each initiatives’ setups are isolated.
     • Some manifests are common.
     • User-friendly repo management.
     • Updates must be automatic, with some degree of control.
                                                                      9
03
The Solution
03
     The Solution




                    11
03
     Git Frontends




                     Internal Github

                     § Licencing
                     § Proprietary
                       (can’t add post-receive hooks)




                                                        12
03
     Git Frontends


         Git + Gitolite or Gitosis + Gitweb

            § Takes time (install, make it work, learn)
            § Not the most user friendly combo.
            § Ugly




                                                          13
03
     Gitlab



         § Open Source
         § Free
         § Easy to install
         § Easy to administrate
         § LDAP integration
         § Owner of repo can manage
           team members
         § Looks great!



                                      14
04
Implementation
04
     Products + Environments



        •   Puppet Environments = Initiative + Environment

        •   Git repos = Initiative + ‘_ puppet’
              cdn_puppet
              m2m_puppet

        •   Git branches = development, staging and production


        •   Puppet Agent: --environment cdn_dev or m2m_sta




                                                                 16
04
     Puppet Master
                                   MANDATORY MODULES :
           /ETC/PUPPET                ntp, yum, snmp, ssh…
            |__ MODULES
            |__MANIFESTS           AVAILABLE MODULES:
               (default node)         mysql, stlib, apache, etc.



           /VAR/LIB/PUPPET/INITIATIVES
                  |__CDN_DEV <-- repo ‘cdn_puppet’,
                  | |__MODULES    branch ‘development’
                 | |__MANIFESTS
                 |    |__NODES

                  |__CDN_STA <-- repo ‘cdn_puppet’,
                                 branch ‘stagement’
                    |__MODULES
                    |__MANIFESTS
                       |__NODES


                                                                   17
04
     Hooks

        Pre-receive            Post-receive
        § Puppet Style Guide   § Updates the puppet master
        § Puppet parser




                                                             18
04
     Hooks
                  Commit goes through




                                           YES?

                             pre-receive          post-receive



                                                       Update the
     push
                                                       initiative’s
     origin                                            manifests
     develop




      sysadmins




                                                                      19
04
     The Big Picture
                                  CDN

               cdn_dev

              m2m_sta
                               DEVELOPMENT



                                 M2M

                   sysadmins



                                STAGING
                         M2M




                         CDN




                                          20
04
     Yeah!


             •   Different sysadmins can configure their
                 systems in several environments in a
                 centralized way
             •   All machines in a controlled state
             •   Inventory
             •   Changes are tracked (commits)
             •   No more how-tos: a git repository is all you
                 need
             •   Deploying new environments is much
                 easier
             •   Reduce duplication of work (Apache,
                 MySQL, etc)



                                                         21
04
     Work in progress




                  Production is not puppetized yet




                                                     22
05
Final notes
05
     Evangelization




            •   puppetusers@tid.es
            •   Separate data from code (Hiera)
            •   Always have the future in mind
                (parametrized classes, OS changes, multiple networks, etc.)


                                                                              24
05
     Vagrant


     Vagrant::Config.run do |config|
       config.vm.box = “RH5.8_2”
       config.vm.box_url = http://xxx/rh58_base.box
       config.vm.host_name = “m2mvagrant2”

       config.vm.provision :puppet_server do |puppet|
          puppet.puppet_server = "puppet.aislada“
          puppet.options = ["--environment","m2m_dev"]
       end
     end




                                                         25
05
     Mcollective

     $ mco puppet runall 10

     2013-03-13 14:26:50: Running all nodes with a concurrency of 10
     2013-03-13 14:26:50: Discovering enabled Puppet nodes to manage


     $ mco package status kernel
            Summary of Ensure:

                 2.6.32-220.el6 = 147
              2.6.32-279.19.1.el6 = 19
              2.6.32-220.7.1.el6 = 3




                                                                       26
05
       Mcollective – Detect performance issues
     $ mco puppet summary

                Total resources: ▇▁▁▁▁▁▁▁▁▁
                                  ▁▁▁▁▁▁▁▁▁▂min: 66.0 max: 4.7k
            Out Of Sync resources: ▇▁▁▁▁▁▁▁▁▁
                                     ▁▁▂▁▁▁▁▁▁▁min: 0.0 max: 7.0
               Failed resources: ▇▁▁▁▁▁▁▁▁▁
                                  ▁▁▁▁▁▁▁▁▁▁min: 0.0 max: 1.0
              Changed resources: ▇▁▂▁▁▁▁▁▁▁
                                    ▁▁▁▁▁▁▁▁▁▁min: 0.0 max: 6.0
      Config Retrieval time (seconds): ▇▁▁▁▁▁▁▁▁▁
                                        ▁▁▁▁▁▁▁▁▁▁min: 0.7 max: 1.2
          Total run-time (seconds): ▇▁▁▂▁▁▁▁▁▁
                                     ▁▁▁▁▁▁▁▁▁▁min: 1.2 max: 10.9
       Time since last run (seconds): ▄▃▂▅ ▃▇▂ ▂▂
                                        ▂▃▄▄▆▃▁▄▂▁▅min: 31.0 max: 1.8k



     $ mco find -S "resource().total_time > 10"
     node-demo-fe-01




                                                                         27
06
Future
06
     Future




              §   ENC
              §   Hiera
              §   Semi-automatic VM provisioning
              §   PuppetDB and Puppet Inventory
              §   Automated monitoring with Nagios (exported resources)




                                                                          29
07
Thanks and
Questions
06
     Thanks


              § Puppet Labs
              § Telefónica I+D
              § Thank you!

              § We’re hiring!

               xavi.carrillo@gmail.com

                  epgbcn3@tid.es




                                         31
06
     Questions?
     … and hopefully, answers




                                32
Puppet and Telefonica R&D

Puppet and Telefonica R&D

  • 1.
    Puppet and Telefónica I+D Puppet CampBarcelona 2013 Xavi Carrillo
  • 2.
  • 3.
    01 Introduction Who am I? § Puppet user for the last 2 years § Working at Telefonica I+D for 4 months § Telefonica Digital curently involved in a process of modernization § Helping on that by puppetizingTelefónica I+D 3
  • 4.
    01 What do we do at I+D? 4
  • 5.
    01 Initiatives 5
  • 6.
  • 7.
    02 Environment x7 Development Staging Production 7
  • 8.
    02 The problem domain § Machine Setups: manually (howtos,etc) § Environments: docs, wikis. § Releases: Hand over an RPM… § At least one SysAdmin per initiative 8
  • 9.
    02 The needs • Infrastructure for many sysadmins, initiatives and environments. • Each initiatives’ setups are isolated. • Some manifests are common. • User-friendly repo management. • Updates must be automatic, with some degree of control. 9
  • 10.
  • 11.
    03 The Solution 11
  • 12.
    03 Git Frontends Internal Github § Licencing § Proprietary (can’t add post-receive hooks) 12
  • 13.
    03 Git Frontends Git + Gitolite or Gitosis + Gitweb § Takes time (install, make it work, learn) § Not the most user friendly combo. § Ugly 13
  • 14.
    03 Gitlab § Open Source § Free § Easy to install § Easy to administrate § LDAP integration § Owner of repo can manage team members § Looks great! 14
  • 15.
  • 16.
    04 Products + Environments • Puppet Environments = Initiative + Environment • Git repos = Initiative + ‘_ puppet’ cdn_puppet m2m_puppet • Git branches = development, staging and production • Puppet Agent: --environment cdn_dev or m2m_sta 16
  • 17.
    04 Puppet Master MANDATORY MODULES : /ETC/PUPPET ntp, yum, snmp, ssh… |__ MODULES |__MANIFESTS AVAILABLE MODULES: (default node) mysql, stlib, apache, etc. /VAR/LIB/PUPPET/INITIATIVES |__CDN_DEV <-- repo ‘cdn_puppet’, | |__MODULES branch ‘development’ | |__MANIFESTS | |__NODES |__CDN_STA <-- repo ‘cdn_puppet’, branch ‘stagement’ |__MODULES |__MANIFESTS |__NODES 17
  • 18.
    04 Hooks Pre-receive Post-receive § Puppet Style Guide § Updates the puppet master § Puppet parser 18
  • 19.
    04 Hooks Commit goes through YES? pre-receive post-receive Update the push initiative’s origin manifests develop sysadmins 19
  • 20.
    04 The Big Picture CDN cdn_dev m2m_sta DEVELOPMENT M2M sysadmins STAGING M2M CDN 20
  • 21.
    04 Yeah! • Different sysadmins can configure their systems in several environments in a centralized way • All machines in a controlled state • Inventory • Changes are tracked (commits) • No more how-tos: a git repository is all you need • Deploying new environments is much easier • Reduce duplication of work (Apache, MySQL, etc) 21
  • 22.
    04 Work in progress Production is not puppetized yet 22
  • 23.
  • 24.
    05 Evangelization • puppetusers@tid.es • Separate data from code (Hiera) • Always have the future in mind (parametrized classes, OS changes, multiple networks, etc.) 24
  • 25.
    05 Vagrant Vagrant::Config.run do |config| config.vm.box = “RH5.8_2” config.vm.box_url = http://xxx/rh58_base.box config.vm.host_name = “m2mvagrant2” config.vm.provision :puppet_server do |puppet| puppet.puppet_server = "puppet.aislada“ puppet.options = ["--environment","m2m_dev"] end end 25
  • 26.
    05 Mcollective $ mco puppet runall 10 2013-03-13 14:26:50: Running all nodes with a concurrency of 10 2013-03-13 14:26:50: Discovering enabled Puppet nodes to manage $ mco package status kernel Summary of Ensure: 2.6.32-220.el6 = 147 2.6.32-279.19.1.el6 = 19 2.6.32-220.7.1.el6 = 3 26
  • 27.
    05 Mcollective – Detect performance issues $ mco puppet summary Total resources: ▇▁▁▁▁▁▁▁▁▁ ▁▁▁▁▁▁▁▁▁▂min: 66.0 max: 4.7k Out Of Sync resources: ▇▁▁▁▁▁▁▁▁▁ ▁▁▂▁▁▁▁▁▁▁min: 0.0 max: 7.0 Failed resources: ▇▁▁▁▁▁▁▁▁▁ ▁▁▁▁▁▁▁▁▁▁min: 0.0 max: 1.0 Changed resources: ▇▁▂▁▁▁▁▁▁▁ ▁▁▁▁▁▁▁▁▁▁min: 0.0 max: 6.0 Config Retrieval time (seconds): ▇▁▁▁▁▁▁▁▁▁ ▁▁▁▁▁▁▁▁▁▁min: 0.7 max: 1.2 Total run-time (seconds): ▇▁▁▂▁▁▁▁▁▁ ▁▁▁▁▁▁▁▁▁▁min: 1.2 max: 10.9 Time since last run (seconds): ▄▃▂▅ ▃▇▂ ▂▂ ▂▃▄▄▆▃▁▄▂▁▅min: 31.0 max: 1.8k $ mco find -S "resource().total_time > 10" node-demo-fe-01 27
  • 28.
  • 29.
    06 Future § ENC § Hiera § Semi-automatic VM provisioning § PuppetDB and Puppet Inventory § Automated monitoring with Nagios (exported resources) 29
  • 30.
  • 31.
    06 Thanks § Puppet Labs § Telefónica I+D § Thank you! § We’re hiring! xavi.carrillo@gmail.com epgbcn3@tid.es 31
  • 32.
    06 Questions? … and hopefully, answers 32