Test and behavior driven infrastructure provides a framework to cope with an ever changing environment. TDI allows operations the benefits as TDD (Test Driven Development) and BDD (Behavior Driven Development), minimizing risk, building confidence, and maintaining focus. I will look at problems commonly seen in traditional system administration. I will cover configuration management and go over a continuous integration workflow in TDI, including the methodology, components, and stages.
2. Hello!
I AM CARLOS MEZA
Sysadmin that wants to
do things better.
Carlos@digitalr00ts.com
@digitalr00ts
3. ROADMAP
My journey learning TDI
1. The Promise of TDI
2. Configuration
Management
3. Principles of Test
and Behavior Driven
Development
4. Continuous
Integration
5. Test Driven
Infrastructure
6. Continuous Delivery
7. Continuous
Deployment
8. Challenges
15. TEST
DRIVEN DEVELOPMENT
A framework to
manage evolving environments with
highly reliable and maintainable code.
○Maintains Scope
○Faster Feedback
○Reduces Risk and
Builds Trust
16. BEHAVIOR
DRIVEN DEVELOPMENT
Facilitates collaboration between
stakeholders and developers by describing
requirements as system behaviors.
○Business Values Prioritized
○Tests are Documentation
○Improves Communication
17. SERVERSPEC EXAMPLE
describe port(80) do
it { should be_listening }
end
describe command("lsb_release -d") do
it { should return_stdout /wheezy/ }
end
20. CONTINUOUS INTEGRATION
A practice with short development cycles,
automated testing and
code integration
○Detect Issues Sooner
○Faster to Implement
21. TESTING
Unit Test
Verifies that individual parts of your code
continue to work (Pre-Convergence)
Integration Test
Verifies the end state of the system post
integration (Post-Convergence)
31. PRINCIPLES AND PRACTICES
○Document - Requirements
○Test - Describe behavior
○Script - Code to pass test
○Audit - Version control
○Continuous - Automatically test and
integrate changes
42. CONTINUOUS DELIVERY
A practice of delivering every change to a
production-like environment.
○Increases Ability to Adapt
○Increases Reliability
○Lowers Risk
○Faster to Production
61. SOURCES AND
MORE INFORMATION
○5 Things About Configuration
Management Your Boss Needs To Know
https://www.scriptrock.com/blog/5-configuration-management-boss
○Overview of Test Driven Infrastructure
with Chef
https://www.chef.io/blog/2015/04/21/overview-of-test-driven-
infrastructure-with-chef/
○Agile DevOps: Test-driven infrastructure
http://www.ibm.com/developerworks/library/a-devops5/
62. SOURCES AND
MORE INFORMATION
○Guide to Agile Practices -
http://guide.agilealliance.org
○ThoughtWorks Insights on CD
https://www.thoughtworks.com/insights/continuous-delivery
HELLO
I am Carlos
A sysadmin trying to do things better
mix environmentLinux manage Windows
ROADMAP - my journey
working on deploying
what i have learned
TDI incorporates several concepts
no empirical definitions, general idea
Slides will be available.
The Promise of TDI
Configuration Management
Principles of Test and Behavior Driven Development
Continuous Integration
Infrastructure
Continuous Delivery
Continuous Deployment
Challenges
1. TEST DRIVEN INFRASTRUCTURE
what is
why worth effort
WHY TDI
Test driven infrastructure (TDI) DI applies agile development principles and practices to infrastructure.
Bringing Benefits
minimize risk
build confidence in code and process
and efforts stay on task, outcome focused
It does this with...
WHY TDI
-Verifiable Behavior - check results against requirements - Consistency
-Increased Code Quality - change are incremental, easier to improve, increasing reliability - Confidence
-Safe Refactoring - applying good practices, sped up - Speed
-Improved Design - Adaptability - being able to adopt enables design improvements
Efficiencies = low I.T. costs
SYSADMIN ANTI-PATTERNS
Issues w/ system deployment and maintenance
Lack of processInefficient / chaotic, difficult to repeat, cannot verify, does not scale
Manual process (docs)repeatable but prone to errors / inconsistencies, documentation outdated. introduces risk = deteriorating trust
Poor codemyriad of scripts, difficult to understand, unmaintainable
configuration management can help
2. CONFIGURATION MANAGEMENT
translates infrastructure into code
CONFIGURATION MANAGEMENT
Configuration management ensures that systems are in aknown, good and trusted state.
Visibility of configAuditable/trackable - version control
Reliabilitymaintaining state. Predictable
ProductivityFaster resolutions, Fewer outages = less firefighting = I.T. cost reduction
Confidence to deploy and scale fastermore efficient change management.
CHEF WORKFLOW
This is from Chef but the idea is similar for other CM.
Top: Repository (version control) with infrastructure code
Blue: Dev station change configurations and related attributes
Orange: CM server. Policies and config to enforce
Right: devices/nodes CM deploys to
CM OPTIONS
Puppet/Chef
PuppetNondeterministic run sequence
Salt/Ansibleattuned for Sysadmins
CHEF:
History Windows
Large Community
Out of the box tools
Innovative
Improving workflow
Developing and adopting tools
CHEF RECIPE EXAMPLE
Remove package
package 'explicit_action' do
action :remove
end
Remove package with version attribute
package 'with_attributes' do
version '1.0.0'
action :remove
end
CM SHORTCOMINGS
does not prevent bad infrastructure code that ispoorly created and maintained.
Incomplete Code (To Dos, Sloppy Quick Fixes)
Unclear Code (Duplicate or Contradicting)
Unintended Side Effects that could catastrophic = lack of confidence
CM on its own is not a complete solution
3. TEST and BEHAVIOR DRIVEN DEVELOPMENT
to the rescue
TEST DRIVEN DEVELOPMENT
A framework for managing evolving environments thatfacilitates the creation of highly reliable and maintainable code.
Maintain Scope:Test are written first.specific requirements.
Tests provide faster feedback
Tests reduce risk help builds trust in the process and code
BEHAVIOR DRIVEN DEVELOPMENT
BDD is an extension of TDD
Facilitates collaboration between stakeholders and developers by describing requirements as system behaviors.
business values are prioritized because the requirements correlated to business outcomes
Tests are describing the systembecoming documentation
Communications are Improved asdocumentation is more readable for experts across different domains.
Acceptance test
SERVERSPEC EXAMPLE
Example of test
Serverspec allows writing of
readable,
to validate that a server is correctly configure
--
describe port(80) do
it { should be_listening }
end
describe command("lsb_release -d") do
it { should return_stdout /wheezy/ }
end
RED, GREEN, REFACTOR
mnemonic for remembering the basic TDD cycle
RED: write a "single" unit test describing a requirement.make it fail.this is documentation
GREEN: write "just enough" code to pass test
REFACTOR:cleanup codechange implementation, not behavior
Rise and Repeat
4. CONTINUOUS INTEGRATION
CI is a practice from XP (a type of agile development), tospeed up development.
Automating TDD
CONTINUOUS INTEGRATION
A practice w/
short development cycles
automated testing
code integration
Detect Issues SoonerImmediate feedback with automated testingaka Failing Fast
Chages Faster to Implementchanges are small, verified and integrated automatically
TESTING
2 notably types of tests
(there are more types)
Unit TestVerifies that individual parts of your codebase
independent of other changes(Pre-Convergence, before integration)
Integration Test
Post integration verificationend state of the system is as expected(Post-Convergence)
CI WORKFLOW
1-Commit Code
CI WORKFLOW
2-CI server fetches changes
CI WORKFLOW
3-Build
4-Test
5-Pass/Fail
CI WORKFLOW
6-Feedback with notifications of results
CI SOFTWARE OPTIONS
Jenkins is probably the mostpopular tool for this task.I currently using
Travis CI is what I initially started with.it’s hosted platform is free for opensource project and easily integrates with GitHub.
Go is a tool opensourced by ThoughtWorks and looks interesting and I intend to investigate
CODE REVIEW
Just because we are moving towards automating all the the things, Code review should not be overlooked.
The creators of cyanide and happiness seem to agree.
CODE REVIEW
Peer review of source code to
improve software quality.
Ensuring readability and maintainability
Knowledge sharing of the code, new team members up to speedExposure to different approaches
TOOL ASSISTED OPTIONS
act as gatekeepers
More popular options
Garret
Review Board
Phabricator
They assist with
comments/discussion
auditing/tracking
difs
5. TEST DRIVEN INFRASTRUCTURE
Agile Infrastructure
PRINCIPLES AND PRACTICES
Document specifications and requirementsComponents and versionhow tos install
TestWrite tests to describe those requirements
ScriptCode to pass test.
Audit and track code w/Version control to track changes.
Continuous processAutomatically test every change and integrate
http://www.ibm.com/developerworks/library/a-devops5/
TDI WORKFLOW
There is not a definitive way to implement.
I am not going to say this is the “right” way to do it.
just how I’m doing it
Fork/Branch from version control
TDI WORKFLOW
Code:
creating test
code to pass test
TDI WORKFLOW
Commit and
git hook will trigger
TDI WORKFLOW
call to CI (Jenkins)
TDI WORKFLOW
Jenkins sends jobs to slaves withTest Kitchen (a chef tool)
TDI WORKFLOW
Jenkins slaves spins up environment
for integration testing.
VMs, containers, etc
TDI WORKFLOW
Feedback
Jenkins notifies us whether or not
if the job passed
email, chat, etc
TDI WORKFLOW
If it passesJenkins will issue a pull request
TDI WORKFLOW
Code Review
manual committedanother team member.
and code is integrate
6. CONTINUOUS DELIVERY
Delivering code changes to staging environments
CONTINUOUS DELIVERY
A practice of delivering every change to a production-like environment.
Increases Ability to Adaptby providing more feedback
Increases Reliabilityas effect of every incremental change can be seen
Lowers Riskas a result
Faster to Production
CD WORKFLOW
Passes code review andpull request committed
CD WORKFLOW
Test triggered using git hooks
CD WORKFLOW
Slaves
Jenkins sends jobs to slaveswith Test Kitchen (a chef tool)
CD WORKFLOW
Integration Test
Jenkins slaves spins up environment forintegration testing of master branch.VMs, containers, etc
CD WORKFLOW
Feedback
Jenkins notifies uswhether or not the job passed.email, chat, etc
CD WORKFLOW
Push to Chef staging environment
If passed
7. CONTINUOUS DEPLOYMENT
Automating to Production
the holy grail
CONTINUOUS DEPLOYMENT
Automating acceptance testing to promote code throughpre-productions environments to production(i.e. dev->staging->prod).
acceptance testdetermine if the requirements of a specification or contract are met
Provides
Quicker Return on Investment
Faster Client Feedback
CD WORKFLOW
Continuous delivery can automate the process through staging environments like development and QA.
CD WORKFLOW
The difference between continuous delivery and continuous deployment is
CD WORKFLOW
Continuous deploymentautomates the push to production.
CD WORKFLOWAs long
final step is to production is manualit is continuous delivery
few implement actual continuous deployment
where its 100% automated
TOOL OPTIONS for CD
Jenkins plugins:
-Workflow
-Pipelines
Native Chef solutions Delivery Truck
in Go pipelines are first class built-in concepts
Pipelines are a series of stages with visibility, feedback, and continuous deployment
8. CHALLENGES
Why this wont work and
will fail
CHALLENGES
Cultural shiftRequires:
workflow changes
time investment
Adoption Not doing proper practicing
Writing too many tests at once
Writing tests that are too large or coarse-grained
Not writing tests
CHALLENGES
Poor Testing
Not knowing/understanding what to test for
Overlooking what needs to be tested
Writing overly trivial tests or writing tests for trivial code