From Pain to Gain: A Puppet Unit Testing Story</b
•
1 like•312 views
This document summarizes a presentation given by Jordan Moldow and Nadeem Ahmad at PuppetConf 2016 titled "From Pain To Gain: A Puppet Unit Testing Story". The presentation discusses the challenges the team at Box faced with manually testing Puppet configurations and their process of implementing unit testing to improve the testing workflow. Some of the key challenges addressed included long wait times for manual tests, inability to test all configurations, and lack of repeatability. The team started with basic unit testing using rspec-puppet but encountered issues with undefined facts, default attributes, and production-only functions. They created the box_spec_helper tool to address these problems and enable tests to run automatically and in parallel. This significantly reduced test
Recommended
More Related Content
Viewers also liked
Viewers also liked (17)
Similar to From Pain to Gain: A Puppet Unit Testing Story</b
Similar to From Pain to Gain: A Puppet Unit Testing Story</b (20)
More from Puppet
More from Puppet (20)
Recently uploaded
Recently uploaded (20)
From Pain to Gain: A Puppet Unit Testing Story</b
- 1. Jordan Moldow and Nadeem Ahmad PuppetConf 2016 From Pain To Gain: A Puppet Unit Tes4ng Story
- 7. Characteris8cs of spaghe6 monoliths • Inter-connected modules • No specifica4ons • Organic growth • Business logic everywhere
- 9. Puppet @ Box Agenda Why unit tesBng? GeFng started Making it work ConBnuous integraBon 1 2 3 4 5 Rolling it out 6
- 13. Why unit tes4ng? Our current tesBng process and how we wanted to improve
- 14. Manual Puppet tes8ng @ Box ~/puppet $ git commit ~/puppet $ git push origin $BRANCH $ ssh test.server.com user@test $ sudo puppet agent –-test --environment=$BRANCH
- 15. Manual Puppet tes8ng @ Box 1. Wait for catalog to compile 2. Wait for no-op run to complete 3. Review and accept no-op results 4. Wait for Puppet run to complete 5. Review state of machine Running Puppet on a test machine
- 16. Manual Puppet tes8ng @ Box 3:00:00 Time spent by engineers
- 17. Other issues with manual monolith tes8ng • Cannot manually test every affected role • Tests are not easily repeatable Module M Module C Module B Module A Role R3 Role R2 Role R1 Cluster 1 Cluster 2 Cluster 3
- 18. Approaches to Puppet tes8ng 1. Manual 2. Acceptance (Beaker) 3. Unit/Integra4on
- 19. Project goals 1. Tests can be wriPen for most files 2. Tests run automa8cally 3. Tests run quickly 4. Tests fail if something is broken 5. Failing tests are reported to code review 6. Developers write tests with their code
- 20. Gefng started First experiment with rspec-puppet
- 21. Tools we used • puppet (3.6.2) • puppetlabs_spec_helper (0.10.2) • rspec (3.1.0) – Ruby spec tes4ng framework • rspec-puppet (2.3.0) – Extension to rspec for tes4ng Puppet modules
- 22. Basic tes8ng Rakefile – Command definiBons
- 24. Basic tes8ng spec/spec_helper.rb – Configure rspec-puppet
- 26. Running tests ~/puppet $ rake spec Run all tests in a directory ~/puppet $ rake spec SPEC=spec/classes/foo Run all tests in a file ~/puppet $ rake spec SPEC=spec/classes/foo/bar_spec.rb Run all tests
- 27. Running tests
- 28. Basic tes8ng • This works! • …on some simple manifests • For technical benefits, this needs to work on most manifests • For cultural change, it needs to work AND be fric4onless
- 29. Making it work Problems we encountered and our soluBons to them
- 31. Problem #1: Undefined facts • rspec-puppet invokes Puppet with a bare minimum of facts provided • Fact values default to undef in manifests • Manifests won’t compile as expected
- 33. Problem #2: Default required aPributes • Resource collectors sefng default aFributes • In produc4on, exec commands don't need absolute path • But tests will fail without it
- 35. Problem #3: Produc8on-only func8ons • Some custom func4ons may only work in produc4on, e.g. ‒ Querying a produc4on data store ‒ Accessing files that only exist on Puppet masters • Any manifest that uses such a func4on cannot be tested ‒ Nor can any manifests that includes such a manifest
- 36. Solu8on #3: Globally-stubbed func8ons • Produc4on-only func4ons should always be stubbed • Any test can be wriFen without test-specific stubs • All other func4ons, classes, types should act as they would in prod
- 37. Solu8on #3: Globally-stubbed func8ons • module_path can include mul4ple directories • Custom func4ons cannot be redefined; first defini4on wins • Added extra tree to front of module_path • Added new modules with redefined func4ons Stubbing via module paths
- 44. Other big enhancements • Simple test auto-generator • Accumula4ng fact, param defini4ons from mul4ple blocks ‒ Including shared contexts • Emula4ng other opera4ng systems ‒ We were restricted on what hardware we could run tests
- 47. First Itera8on of CI Define small beta group of Puppet developers
- 48. First Itera8on of CI Run unit tests for their code reviews, report feedback
- 49. First Itera8on of CI Run unit tests for their code reviews, report feedback
- 55. Unlocking Paralleliza8on Find individual test unit ~/puppet $ find spec -name "*_spec.rb" spec/classes/os_hardening/init_spec.rb spec/classes/rsyslog/init_spec.rb spec/classes/rsyslog/package_spec.rb spec/classes/dashboard/sync_hive_query_runner_spec.rb spec/classes/pod/delivery_pipeline_spec.rb spec/classes/yum/centos_extras_spec.rb spec/classes/dev_env/repo/devtools_for_windows_spec.rb spec/classes/dev_env/repo/devtools_spec.rb spec/classes/snmp/server_spec.rb spec/classes/bind/staging_spec.rb spec/classes/bind/base_spec.rb …
- 56. Running tests ~/puppet $ rake spec Run all tests in a directory ~/puppet $ rake spec SPEC=spec/classes/foo Run all tests in a file ~/puppet $ rake spec SPEC=spec/classes/foo/init_spec.rb Run all tests
- 57. WTF??
- 58. Unlocking Paralleliza8on rake spec is actually a combinaBon of three subcommands: ~/puppet $ rake spec_prep Creates fixtures needed by all tests
- 59. Unlocking Paralleliza8on rake spec is actually a combinaBon of three subcommands: ~/puppet $ rake spec_clean Deletes all test fixtures
- 60. Unlocking Paralleliza8on rake spec is actually a combinaBon of three subcommands: ~/puppet $ rake spec_standalone Runs the actual test (assumes test fixtures are present)
- 63. Workflow $ git push post-receive Git hook Git Server ClusterRunner Master Jenkins Master
- 65. Rolling It Out
- 68. Conclusion
- 71. Ques8ons? • box_spec_helper ‒ hFps://github.com/jmoldow/box_spec_helper • Contact ‒ Box ‒ oss@box.com ‒ opensource.box.com ‒ www.box.com/careers/ ‒ Nadeem ‒ nadeem@box.com ‒ github.com/nadeemahmad ‒ Jordan ‒ jmoldow@box.com ‒ github.com/jmoldow
- 72. Using the Template and Guidelines • Refer to this document for rules and recommenda4ons for the treatment of graphic elements • Always start a new presenta4on using the Box corporate template and make edits from there; this will ensure you always have the correct styles, color paleFe and fonts in use • For layout inspira4on, please refer to the key slides presenta4on
- 73. Color and Font Themes How to Save Themes… Start by opening the Box Corporate Template and clicking on the Design tab. Save the colors from the template to your system by clicking Colors and Create New Theme Colors and save as Box. Do the same thing for the template fonts. Create New Theme Fonts and save as Box.
- 74. Bullet Slide Treatment (Calibri Regular 24pt) • Bullets are sentence case (Calibri Regular 20pt) ‒ Second-line bullets are Calibri 16pt ‒ Third-line bullets are Calibri 14pt • Limit the number of bullets on a slide • Text highlights are Calibri Bold, but not underlined • Try not to go below the recommended font sizes
- 75. Bullet Slide With Sub-8tle Treatment • Try to keep your use of bullet slides to a minimum • Be crea4ve and think visually • If you need to source something copy and paste the text box at the boFom les onto your page SubBtle Text Placeholder Title Case (Calibri 20pt) Source: Gray Calibri Regular 10pt
- 76. Centered Bullet Slide Treatment • Bullets are sentence case (Calibri Regular 20pt) ‒ Second-line bullets are Calibri 16pt ‒ Third-line bullets are Calibri 14pt • Limit the number of bullets on a slide • Text highlights are Calibri Bold, but not underlined • Try not to go below the recommended font sizes
- 77. Centered Bullet Slide With Sub-8tle Treatment • Try to keep your use of bullet slides to a minimum • Be crea4ve and think visually • If you need to source something copy and paste the text box at the boFom les onto your page SubBtle Text Placeholder Title Case (Calibri 22pt) Source: Gray Calibri Regular 10pt
- 78. Segue Slide Title Goes Here and Can Be a Few Lines in Length SubBtle goes here in sentence case
- 79. Segue Slide Title Goes Here and Can Be a Few Lines in Length SubBtle goes here in sentence case
- 80. Segue Slide Title Goes Here and Can Be a Few Lines in Length SubBtle goes here in sentence case
- 81. Segue Slide Title Goes Here and Can Be a Few Lines in Length SubBtle goes here in sentence case
- 82. Segue Slide Title Goes Here and Can Be a Few Lines in Length SubBtle goes here in sentence case
- 83. Segue Slide Title Goes Here and Can Be a Few Lines in Length SubBtle goes here in sentence case
- 84. Segue Slide Title Goes Here and Can Be a Few Lines in Length SubBtle goes here in sentence case
- 85. Enter Btle for secBon one here and use sentence case Agenda Slide Design Enter Btle for secBon two here and use sentence case Enter Btle for secBon three here and use sentence case Enter Btle for secBon four here and use sentence case Enter Btle for secBon five here and use sentence case 1 2 3 4 5
- 86. Enter Btle for secBon one here and use sentence case Agenda Slide Design With Highlight for Each Sec8on Enter Btle for secBon two here and use sentence case Enter Btle for secBon three here and use sentence case Enter Btle for secBon four here and use sentence case Enter Btle for secBon five here and use sentence case 1 2 3 4 5
- 87. Enter Btle for secBon one here and use sentence case Agenda Slide Design With Highlight for Each Sec8on Enter Btle for secBon two here and use sentence case Enter Btle for secBon three here and use sentence case Enter Btle for secBon four here and use sentence case Enter Btle for secBon five here and use sentence case 1 2 3 4 5
- 88. Enter Btle for secBon one here and use sentence case Agenda Slide Design With Highlight for Each Sec8on Enter Btle for secBon two here and use sentence case Enter Btle for secBon three here and use sentence case Enter Btle for secBon four here and use sentence case Enter Btle for secBon five here and use sentence case 1 2 3 4 5
- 89. Enter Btle for secBon one here and use sentence case Agenda Slide Design With Highlight for Each Sec8on Enter Btle for secBon two here and use sentence case Enter Btle for secBon three here and use sentence case Enter Btle for secBon four here and use sentence case Enter Btle for secBon five here and use sentence case 1 2 3 4 5
- 90. Enter Btle for secBon one here and use sentence case Agenda Slide Design With Highlight for Each Sec8on Enter Btle for secBon two here and use sentence case Enter Btle for secBon three here and use sentence case Enter Btle for secBon four here and use sentence case Enter Btle for secBon five here and use sentence case 1 2 3 4 5
- 91. Enter 4tle for sec4on one here and use sentence case Agenda Slide Color Coding Op8on Use different colors to color code each secBon/segue slide Enter 4tle for sec4on two here and use sentence case Enter 4tle for sec4on three here and use sentence case Enter 4tle for sec4on four here and use sentence case Enter 4tle for sec4on five here and use sentence case 1 2 3 4 5
- 92. Chart Slide With Mul8ple Colors Sub-Btle or Chart Title Here in Title Case 76 60 44 57 65 86 72 0 20 40 60 80 100 Category 1 Axis Title Goes Here in Title Case Calibri 16pt Series 1 Series 2 Series 3 Series 4 Series 5 Series 6 Series 7
- 93. Chart Slide With Mul8ple Data Bars Within Each Category Sub-Btle or Chart Title Here in Title Case 76 75 69 88 60 65 57 63 44 50 62 72 0 20 40 60 80 100 Category 1 Category 2 Category 3 Category 4 Axis Title Goes Here in Title Case Calibri 16pt Series 1 Series 2 Series 3
- 96. Process Diagram Treatment See Style Page for More Color OpBons 96 1 2 3 4 Suppor4ng text goes here under the number Suppor4ng text goes here under the number Suppor4ng text goes here under the number Suppor4ng text goes here under the number
- 97. Process Diagram Treatment You Can Use the Blue Bar to House Heavy Content Title Goes Here • One bullet • Two bullet • Three bullet Title Goes Here • One bullet • Two bullet • Three bullet Title Goes Here • One bullet • Two bullet • Three bullet Title Goes Here • One bullet • Two bullet • Three bullet 1 2 3 4
- 98. Box With Bullet Treatment • One bullet • Two bullet • Three bullet • One bullet • Two bullet • Three bullet • One bullet • Two bullet • Three bullet Title Here Title Here Title Here
- 99. Table Layout Treatment SubBtle Text Placeholder Title Case (Calibri 20pt) Header Header Header Header Title One Informa4on Informa4on Informa4on Title Two Informa4on Informa4on Informa4on Title Three Informa4on Informa4on Informa4on Title Four Informa4on Informa4on Informa4on Title Five Informa4on Informa4on Informa4on Title Six Informa4on Informa4on Informa4on Title Seven Informa4on Informa4on Informa4on
- 109. Image Treatment With Cap8on Layout How to Add Your Own Photos and Crop Properly… Insert one of the Picture with Cap4on layouts. Click on the icon at the center to add your image. Your image will populate the container but you will likely need to adjust the crop. Click on the image and the format tab and then select crop.
- 110. Image Treatment With Cap8on Layout (Con8nued) How to Add Your Own Photos and Crop Properly… Click on the grayed out por4on of the image and drag to the les or right un4l you are happy with the crop. Click out of the cropped image to set it and your image is ready to go.
- 111. Image With Bullets Treatment Insert Your Own Photo, See Slide 39 for InstrucBons • Bullet text here • Some more bullet text • Another short bullet • Last bullet goes here
- 112. Logo With Bullets Treatment Insert Your Chosen Logo to the Lek of the Line • Bullet text here • Some more bullet text • Another short bullet • Last bullet goes here
- 113. Image Treatment for Two Images Sub-Btle or Secondary InformaBon Here
- 114. Image Treatment for Three Images Sub-Btle or Secondary InformaBon Here
- 115. Mul8ple Image Treatment Sub-Btle or Secondary InformaBon Here
- 116. Mul8ple Image Treatment How to Insert MulBple Images… This layout will appear. Next, click the icon to insert an image. Navigate to your image and insert it. Repeat for all the image boxes. For instruc4ons on how to crop your image see page 39. Click the New Slide tab and select Mul4ple Image Layout.
- 118. Screen Shot Treatment With Browser Window How to Drop in Your Screen Shot… The browser window is like a frame so anything you drop behind it will show through. Drop in your screen shot, go to the format menu and crop it to show only what you want. Last, be sure to right click on your screen shot and send to back so it tucks behind the browser window frame.
- 119. Color PalePe Hex# 22A7F0 Hex# 2A62B9 Hex# 2D3D4F Hex# BCC1C7 Hex# 3FB87F Hex# 955CA5 Hex# F7931D Hex# E33D55 Hex# 74808C Primary Secondary
- 121. Video in Your Presenta8on How to Insert Video and Set Your Playback SeFngs… To insert a video click insert on the menu bar Next select video and navigate to your video. Once your video is on the slide and you click on it, you will get a new menu tab. You can now make changes to both the format and playback of your video. Use these sefngs for video playback: Start Automa4cally, Play Full Screen, and Hide While Not Playing. This will ensure a consistent look within all Box presenta4ons.