1.
Continuous Integration and Test
Driven Dev for Network Engineers
www.Nexusis.com 1 877.286.3987
Colin McNamara CCIE #18233
@colinmcnamara

2.
www.Nexusis.com 2 877.286.3987

3.
Agenda
 What drives my perspective
 Why TDD / CI for Network Engineers
 CI / TDD overview
 Network Change Management Use Case
www.Nexusis.com 3 877.286.3987

4.
Who am I?
 Chief Cloud Architect::Nexus
 OpenStack ATC/Ambassador
 OpenStack Core Reviewer
 Manage DevOps Team
 Manage Agile BPO Team
 Beard Growth Specialist
 CCIE 18233, RHCE, ETC, ETC
 @colinmcnamara
 www.colinmcnamara.com
www.Nexusis.com 4 877.286.3987

5.
What do
we do?
50%
upstream
For Core Dev’s
Software Development, Systems
Integration, Network Design
DenicaCloud
Rosetta
Aviator
More
Project Denica
Toolchain for Deploying CI systems
and the associated tooling into and
along with various cloud platforms
Core Contributions
and solutions in
OpenStack
OpenDaylight
Open vSwitch
Puppet
Agile / Lean Software
Development Practices
Extensive experience
with the folliwng:
• SDN Development for
Multi Vendor Integrations
• Agile Process Consulting
• Cloud Platform
Development
• Systems Integration
• Upstream contributions
• DevOps Consulting

6.
That all boils down to.. We make unicorn farts
www.Nexusis.com 6 877.286.3987

7.
www.Nexusis.com 7 877.286.3987
Why TDD / CI for
Networking?

8.
Our industry is changing
www.Nexusis.com 8 877.286.3987

9.
Four closely related items
www.Nexusis.com 9 877.286.3987

10.
Agile infrastructure evolution
www.Nexusis.com 10 877.286.3987
Classic Engineering
 12 + month dev times
 QA pushed till last
 Large amounts of bugs
 Maps easily to manual
silo’d process’s
 ITIL Aligned
AGILE
 2 Week Dev times
 Integrated QA
 Small bug counts
 Does not map into silo’d
process’s
 Cloud Aligned

11.
Speed Racer (Agile Infrastructure Developer)
www.Nexusis.com 11 877.286.3987
• Integrated tools
• Integrated test QA
• Software as a manufacturing line
• Code rolls from Step to Step
• Need infrastructure as code…

12.
DevOps is the “New”
www.Nexusis.com 12 877.286.3987
Network Ops

13.
Core SDN truth
www.Nexusis.com 13 877.286.3987
ONE change
In one FILE
Can CREATE a system
Or DESTROY a system

14.
What if you could actually KNOW it would work
www.Nexusis.com 14 877.286.3987
 Test EVERY network
change made
 Map each application to
the network component
underneath
 Automatically run tests with
reports of results

15.
www.Nexusis.com 15 877.286.3987

16.
www.Nexusis.com 18 877.286.3987
Agile Development Continuous
Integration for Network
Engineers
Technical items you need to know

17.
DevOps Workflows
www.Nexusis.com 19 877.286.3987

18.
Complex View
www.Nexusis.com 20 877.286.3987

19.
Simple CI System Walk Through
www.Nexusis.com 21 877.286.3987
SVC Only
SVC + Network
“Virtualized”
Production
System

20.
Obligatory Tools Discussion
www.Nexusis.com 22 877.286.3987

21.
Git / Gerrit
www.Nexusis.com 23 877.286.3987

22.
www.Nexusis.com 24 877.286.3987

23.
Integration Opportunity for CI Pipelines
www.Nexusis.com 25 877.286.3987
 VIRL(CMS) / JunoSphere
 Functional Test
 Network OS level testing
 MATE / WAE
 Stubs / Modeling
 Integration Test through API’s
 Tail-F, Schprokits, Puppet/Chef, Ansible
 Config Distribution
 Unified Data Models

24.
www.Nexusis.com 26 877.286.3987
Use Case
Change and Release Management

25.
Change and Release Management
www.Nexusis.com 27 877.286.3987

26.
Change and Release Management
Jenkins
Test Harnesses
Governance
Tooling
Git
Gerrit
Templates and
EPG’s
Git
Gerrit
www.Nexusis.com 28 877.286.3987
Gerrit
Gerrit
Jenkins
Test Harnesses
Vagrant Ant
Puppet
Jenkins
Test Harnesses
Vagrant
Puppet
Git
Gerrit
Puppet
Other
Git
Gerrit
Test Harnesses
Puppet
Other
Jenkins
Test Harnesses
Puppet
Jenkins Other
Test Harnesses
Vagrant
Puppet
Gerrit
Governance
Tooling

27.
Maturity Levels – how to move forward
Level0 Level1 Level2 Level3 Level4
www.Nexusis.com 29 877.286.3987
• Self Service Builds
• Nightly Builds
• Build Artifacts are
stored
• Self Services deploy
to test
• Auto deploy of
builds to 1st env
• Mostly standard
deploys
• Significant Test
execution at build
time
• Visibility: Team
• Latest reports always
accessible
• Build on commit
• Dependency Repo
• Secured Configs
• Self-Service deploy
to test and prod
• Standard Process
across all
environments
• Some Static analysis
• Automated Funct
tests run nightly
• Visibility: Cross
siloe’d team
(DevOps)
• Historical reports
available
• Triggered Builds
• Build Cluster
• Test Gated
Automated
Promotions
• Database
Deployments
• Multi-tier SOA
• High Code Coverage
• Security Scans
• Risk based manual
• testing
• Report trending
• Build from
Snapshots
• Gated Commit
• Continuous
Deployment to
production
• 100% Coverage
• Gross Silo Analysis
Management QA Operations Development
• Build Value Stream
Maps
• Build Scripts
• Source Control
• Deployment Scripts
• Source Control
• Build Test
Automation
• Source Control
• Visibility: Siloed
Individuals
• Tool Generating
Reporting

28.
How to get there – Level 0
www.Nexusis.com 30 877.286.3987
Development
•Create standards for service chains (cfg,controler)
•Configuration Templates , puppet modules
•Use source control - Git
•Build VSM of specific IT process
Operations
•Script Deployments – Expect, Simple Puppet
•Use source control – Git
QA
•Build new services in lab / cloud / virt environment – VIRL /
GNS3 / VMware / OpenStack
•Simple test harness built (ping)
Management
•Individual Kanban Boards / Ticket tracking
• Build Value Stream
Maps
• Build Scripts
• Source Control
• Deployment Scripts
• Source Control
• Build Test
Automation
• Source Control
• Visibility: Siloed
Individuals
• Tool Generating
Reporting
Level1 Level2
• Self Service
Builds
• Nightly Builds
• Build Artifacts
are stored
• Self Services
deploy to test
• Auto deploy of
builds to 1st env
• Mostly standard
deploys
• Significant Test
execution at
build time
• Visibility: Team
• Latest reports
always
accessible
• Build on commit
• Dependency
Repo
• Secured Configs
• Self-Service
deploy to test
and prod
• Standard
Process across
all environments
• Some Static
analysis
• Automated
Funct tests run
nightly
• Visibility: Cross
siloe’d team
(DevOps)
• Historical
reports available
Management QA Operations Development
Level0

29.
How to get there – Level 1
www.Nexusis.com 31 877.286.3987
Development
• Ci System Implemented
• Scripts build virtual env on demand
• Results from built system get stored
Operations
• Deployment via Config Mgmt tooling
•Governance / CI system deployed
•Ops review of config diffs
QA
•CI System Implemented
•Common functional tests created
•Common port / path tests created
Management
• Visualize flow – Kanban Board for team
• Initial ScrumBan Process’s Mapped
• Build Value Stream
Maps
• Build Scripts
• Source Control
• Deployment Scripts
• Source Control
• Build Test
Automation
• Source Control
• Visibility: Siloed
Individuals
• Tool Generating
Reporting
Level1 Level2
• Self Service Builds
• Nightly Builds
• Build Artifacts are
stored
• Self Services deploy
to test
• Auto deploy of
builds to 1st env
• Mostly standard
deploys
• Significant Test
execution at build
time
• Visibility: Team
• Latest reports always
accessible
• Build on commit
• Dependency
Repo
• Secured Configs
• Self-Service
deploy to test
and prod
• Standard
Process across
all environments
• Some Static
analysis
• Automated
Funct tests run
nightly
• Visibility: Cross
siloe’d team
(DevOps)
• Historical
reports available
Management QA Operations Development
Level0

30.
How to get there – Level 2
www.Nexusis.com 32 877.286.3987
Development
• Ci Systems integrated with Governance + Src Control
• Dendancy Mapping (Mvn, Ant, EPG’s)
• Keys, Passwords as variables, stored securely
Operations
• CI Chain Integrated w/ release
• Governance tool used for Change Control Board
• Puppet + Vagrant + Scripts abstracted from Dev, test prod
QA
• CI System Implemented
• CI system running Spec tests / Flake8 etc
• System building environment and running tests emulating applications
Management
• Roll up reports of multiple team boards
• Burndown charts stored
• Card movements logged
• DevOps changes mapped to existing process’s
• Build Value Stream
Maps
• Build Scripts
• Source Control
• Deployment Scripts
• Source Control
• Build Test
Automation
• Source Control
• Visibility: Siloed
Individuals
• Tool Generating
Reporting
Level1 Level2
• Self Service Builds
• Nightly Builds
• Build Artifacts are
stored
• Self Services deploy
to test
• Auto deploy of
builds to 1st env
• Mostly standard
deploys
• Significant Test
execution at build
time
• Visibility: Team
• Latest reports always
accessible
• Build on commit
• Dependency Repo
• Secured Configs
• Self-Service deploy
to test and prod
• Standard Process
across all
environments
• Some Static analysis
• Automated Funct
tests run nightly
• Visibility: Cross
siloe’d team
(DevOps)
• Historical reports
available
Management QA Operations Development
Level0

31.
www.Nexusis.com 33 877.286.3987
Take Aways

32.
Key Points
 TDD increases quality / decreases risk / increases velocity
 Network Operations and Design benefit from Agile
Methodologies
 Defining and executing on a maturity model necessary to
succeed.
www.Nexusis.com 34 877.286.3987

33.
www.Nexusis.com 35 877.286.3987

34.
www.Nexusis.com 36 877.286.3987
@colinmcnamara
www.colinmcnamara.com