Webinar: Network Automation [Tips & Tricks]

v
Network Automation: Tips & Tricks
A Black & White Presentation
David Barroso, Dinesh G Dutt
August 30, 2016

Agenda
Why Automate ?
The Work Before Automation
How Does Automating Routers Differ From Servers
Tips & Tricks
Summary
August 30, 2016 cumulusnetworks.com 2

Audience Assumptions
Interested in network automation, but stumped or
daunted
Small to mid size enterprises
 1-32 racks
 10-800 servers
Unfamiliar with programming
Use Ansible, but the ideas apply to Puppet, Chef,
Salt etc.

• Network Systems Engineer at Fastly
• Previously:
- NetworkEngineer at Spotify
- NetworkEngineer at NTT
- Network & Systems Engineer at Atlas IT
• Creator of:
- N.A.P.A.L.M.
- SDN InternetRouterTwitter |Linkedin |Github
@dbarrosop

WHY AUTOMATE?
Even when you are a small shop

#1 CONSISTENCY
It's not only about automation, it's about consistent configurations,
workflows and change control

Consistent errors introduced by bugs are easier to identify and
fix than random errors introduced by humans

#2 SCALABILITY
If it works for 1 device it works for N devices (-ish)

#3 FAST ITERATION
Small and incremental changes are easier to perform when you can
focus on the changes and the outcome
vs
where to apply them and how

#4 FOR "FUN"
It's more interesting than provisioning a VLAN or an IP
for the gazillionth time

•Prelude to Automation

Reduce Clutter
https://www.flickr.com/photos/rubbermaid/commons.wikimedia.org

Exploit order and regularity of network
 Same ports across all boxes connected to uplink ports
 Same host connected to pair of leaves on same port on both
leaves
From this order and regularity emerge simple
patterns
Automate Patterns
Patterns
Spine01 Spine02
1 2 3 1 2 3 1 2 3
Spine03

Principles of Simplifying Configuration
Cookie cutter configuration a.k.a substitutability
 As little node-specific variation as possible
• Nothing more than a single IP address, node name, for
example
 As little duplication of information as possible
• Specifying IP addresses on interfaces AND in OSPF/BGP
network statements
 As much configuration as necessary, not more

How Automating Switches/Routers Differs From Servers
 Scale
 Interfaces
 VLANs
 Multiple pieces of
information have to be
configured AND
coordinated across
devices:
 IP addresses on interfaces
(common subnet)
 If BGP, ASN number of self
and peer
L1 L2 L16
S1 S2 S4S3
10.1.1.1
10.1.1.0
10.1.4.33
10.1.4.32

How Automating Switches/Routers Differs From Servers
Duplication of Information
 Eg.: IP address specified on interface, in network statements, in
BGP neighbor statements
Complex Configuration
 Multiple protocols

•Tips & Tricks

#0: Simplicity vs Flexibility

#!/usr/bin/python
print “Hello Mr. Barroso”
#!/usr/bin/python
name = “Mr. Barroso”
print “Hello %s” % name
#!/usr/bin/python
class Person(object):
'''class storing attributes of a person'''
first_name = ''
last_name = ''
def __init__(self, name, surname):
self.first_name = name
self.last_name = surname
def greet(self):
print "Hello Mr. %s" % self.last_name
if __name__ == '__main__':
name_input = input('Enter <first name> <last name>:
')
print name_input
prenom, nom = name_input.split()
persona = Person(prenom, nom)
persona.greet()

Real Life Example of a Customer
 1. Push device-
specific files
(glorified file copy)
 2. Look at patterns
and create templates
 3. Automate more of
the tasks
 4. Add Ansible roles,
fully automated
L1 L2 L16
S1 S2 S4S3

Start with Automating Simple Tasks
Adding or removing users
Adding additional interesting packages
 bwm-ng or scamper for example
Convert ad-hoc command into playbook
cumulus@dinesh-ubuntu ~/w/a/playbook> ansible leaf-1 -s -m apt -a 'name=bwm-ng state=installed'
- hosts: all
tasks:
- name: Install bwm-ng on all hosts
apt: name=‘bwm-ng’ state=installed update_cache=yes

•#1: Pick Simple, Consistent
Toolchain

Some Tools Fit Better With Some Languages Than Others
Puppet & Chef have Ruby as base language
Ansible users tend to use Python
Mixing Python & Ruby tool chains requires multiple
language skills, can be more maintenance
 For example, Serverspec and other such validation tools will be
natural for Puppet/Chef shops, but will require adding Ruby
skills to Ansible shops
Cumulus Linux is Linux, so any tool works out of the
box, no assembly required

•#2: Use Unnumbered Interfaces

Use of Unnumbered in DC
Unnumbered Interfaces are those without a global IP
address of their own
Interface IP addresses are never advertised inside
the DC
 Reduces IP address requirements
 Reduces FIB & RIB sizes
 Reduces attack vector
 Automation simplification: Single IP address to configure per
node

•#3: Use Interface Names Instead of
IP Addresses

Why ?
Names are easier to spot errors with compared to IP
addresses
Unchanged configuration on renumbering
With unnumbered interfaces, interfaces have no IP
addresses anyway

OSPF: Avoid “network” Statements, Use “ip ospf area” under
“interface:
interface swp1
ip ospf area 0.0.0.0
interface swp2
…
inerface swp17
!
router ospf
ospf router-id 10.0.0.17
S1
interface swp1
interface swp2
…
inerface swp17
!
router ospf
S4
interface swp1
interface swp2
…
inerface swp4
!
router ospf
L1
interface swp1
interface swp2
…
inerface swp4
!
router ospf
L16
SPINE
LEAF L1 L2 L16
S1 S2 S4S3
10.1.1.1
10.1.1.0
10.1.4.33
10.1.4.32

Traditional BGP Configuration
August 30, 2016 29
router bgp 64501
bgp log-neighbor-changes
bgp router-id 10.0.0.1
!
neighbor 10.1.1.1 remote-as 65000
router bgp 64502
!
router bgp 65000
!
…
router bgp 65000
!
…
router bgp 64516
!
L1 L2 L16
S1
S4
cumulusnetworks.com
SPINE
LEAF L1 L2 L16
S1 S2 S4S3
10.1.1.1
10.1.1.0
10.1.4.33
10.1.4.32

BGP Unnumbered Configuration
August 30, 2016 30
router bgp 64501
!
neighbor swp1 remote-as external
router bgp 64502
!
router bgp 64516
!
router bgp 65000
!
…
router bgp 65000
!
…
L1 L2 L16
S1
S4
cumulusnetworks.com
SPINE
LEAF L1 L2 L16
S1 S2 S4S3

•#4: A Host by any Name…

Some Characteristics of a Hostname
Pick a base hostname that reflects the key role of the
device:
 leaf, tor, spine, etc.
Assign a unique number to device instance to
construct unique hostname
 leaf-1 (for leaf in rack-1), spine-1, etc.
Add prefixes to make it globally unique:
 dc-ny-tor-1, dc-sf-tor-1, dc-sf-tor-2 etc.

Generate Unique ID from Hostname
An example to simulate thinking, but this is not
unlike how some customers have deployed it
- hosts: all
any_errors_fatal: true
vars_files:
- properties.yml
tasks:
- name: Get my node ID
set_fact:
my_node_id: "{{ inventory_hostname.split('-')[1] }}"

Use Hostname to Derive Loopback IP address
 Poor man’s IPAM
 Uses jinja2’s ipsubnet
filter
 Spine addresses are
assigned from one end and
the leaf addresses from the
other end
 leaf-1 gets ipsubnet(32, 1),
leaf-2 gets ipsubnet(32, 2)
etc.
 spine-1 => ipsubnet(32, -
2), spine-2 => ipsubnet(32,
-3) etc.
- hosts: all
vars_files:
- properties.yml
tasks:
- name: Get my node ID
set_fact:
my_node_id: "{{ inventory_hostname.split('-')[1] }}"
- name: Get loopback IP for leaves
set_fact:
my_ip: "{{ lo_ip_subnet|ipsubnet(32, (my_node_id|int)) }}"
when: "{{ 'leaf' in group_names }}"
- name: Get loopback IP for spines
set_fact:
my_ip: "{{ lo_ip_subnet|ipsubnet(32, -(my_node_id|int)-1) }}"
when: "{{ 'spine' in group_names }}"

Use Hostname to Derive ASN
Same style as IP,
but for ASN
 leaf-1 => base_asn +
1
 leaf-2 => base_asn +
2
 ….
- hosts: all
vars_files:
- properties.yml
tasks:
- name: Get my ASN for spines
set_fact:
my_asn: "{{ bgp_spine_asn }}"
- name: Get my ASN for leaves
set_fact:
my_asn: "{{ (bgp_leaf_asn_base | int) + (my_node_id | int) }}"

Use Hostname to Derive CLAG Configuration
- name: Construct MLAG Local IP
set_fact:
my_clag_ip: |
{% if (my_node_id|int %2) == 1 %}
169.254.1.1/30
{%else%}
169.254.1.2/30
{%endif%}
when: "{{ 'leaf' in group_names and dual_attach_hosts }}"
- name: Construct MLAG Peer IP
set_fact:
my_clag_peer_ip: |
{% if (my_node_id|int % 2) == 1 %}
169.254.1.2
{%else%}
169.254.1.1
{%endif%}
when: "{{ 'leaf' in group_names and dual_attach_hosts }}"
- name: Construct CLAG SysMAC
set_fact:
my_clag_sys_mac: |
{% if (my_node_id|int % 2) == 1 %}
{{ "%s%02d"|format(clag_base_sys_mac,
(my_node_id|int)) }}
{%else%}
{{ "%s%02d" | format(clag_base_sys_mac,
(my_node_id|int - 1)) }}
{%endif%}
when: "{{ 'leaf' in group_names and dual_attach_hosts
}}"
- name: Construct CLAG Priority
set_fact:
my_clag_prio: |
{% if (my_node_id|int % 2) == 1 %}
4096
{%else%}
8192
{%endif%}
when: "{{ 'leaf' in group_names and dual_attach_hosts
}}"
Using modulo with my_node_id to derive unique information per pair

•#5: Validate Your Input Before
Applying

Use Ansible’s Validate Option Rigorously
- hosts: all
vars_files:
- properties.yml
tasks:
- name: Add ISL interfaces to interfaces file for OSPF
blockinfile:
dest: /etc/network/interfaces
marker: "#{mark} {{ item }} ANSIBLE MANAGED BLOCK"
block: |
auto {{ item }}
iface {{ item }} inet static
address {{ my_ip }}/32
validate: ifup -s -a -i %s
become: true
with_items: ansible_interfaces
when: "{{ (protocol == 'ospf') and
(('spine' in group_names and item|match(spine_to_leaf_ports)) or
('leaf' in group_names and item|match(leaf_to_spine_ports))) }}"
tags:
- ifconfig

Quagga Validation Needs a Little Love
- name: Push out quagga config checker
copy:
dest: /tmp/chk-quagga.sh
mode: 0700
content: |
#!/bin/sh
sudo chmod 644 $1
sudo vtysh -C -f $1
become: yes
- name: Add logging and base config
blockinfile:
dest: /etc/quagga/Quagga.conf
create: yes
owner: quagga
marker: "!{mark} base config ANSIBLE MANAGED BLOCK"
block: |
!
log file /var/log/quagga/quagga.log
log timestamp precision 6
!
validate: sudo /tmp/chk-quagga.sh %s
become: true

•#6: Automate Configuration
Verification

Validate Configuration Using Appropriate Tool
Most people use manual configuration to validate
configuration
Puppet & Chef come with their configuration
validation tools
With Ansible, use playbooks to validate
configuration
August 30, 2016 cumulusnetworks.com

Validate BGP Configuration
- name: Get bgp summary
command: vtysh -c 'sh ip bgp summary json'
register: cmd_out
become: true
- name: Get the peer count
set_fact:
peer_count: "{{ ((cmd_out.stdout|from_json).totalPeers) }}"
- name: Get the peer list
set_fact:
bgp_peers: "{{ (cmd_out.stdout|from_json).peers }}“
- name: Validate peer count matches the expected number of leaves
assert: { that: '(peer_count|int) == num_leaves' }
- name: Validate peer count matches the expected number of spines
assert: { that: '(peer_count|int) == num_spines' }
- name: Verify all BGP sessions are in established state
assert: { that: 'bgp_peers[item]["state"] == "Established"' }
with_items: "{{ bgp_peers }}"

Validate CLAG Configuration
---
- hosts: 'leaf*'
vars_files:
- properties.yml
gather_facts: false
tasks:
- name: Get clagctl output
command: clagctl -j
register: cmd_out
- name: Get the status
set_fact:
clag_status: "{{ (cmd_out.stdout|from_json).status }}"
- name: Get the Individual Bond status
set_fact:
clag_ifs: "{{ (cmd_out.stdout|from_json).clagIntfs }}"
- name: Verify CLAG Peer is up and alive
assert: { that: 'clag_status["peerAlive"] == true' }
- name: Verify all bonds are dual attached
assert: { that: 'clag_ifs[item]["status"] == "dual"' }
with_items: "{{ clag_ifs }}"
vagrant@leaf-1:~$ clagctl -j
{
"clagIntfs": {
"bond-swp5": {
"clagId": 5,
"status": "single"
},
"bond-swp6": {
"clagId": 6,
"status": "single"
}
},
"status": {
"backupActive": false,
"backupIp": "",
"ourId": "08:00:27:7f:06:83",
"ourPriority": 4096,
"ourRole": "primary",
"peerAlive": true,
"peerId": "08:00:27:70:33:61",
"peerIf": "peer-link.4094",
"peerIp": "169.254.1.2",
"peerPriority": 8192,
"peerRole": "secondary",
"sysMac": "44:38:39:ff:00:01"
}
}

•#7: Stage & Test Your Changes in
Virtual Land First

Vagrant & VX
Vagrant is a simple, effective tool to build networks –
complete with switches and hosts – in virtual land,
using VMs
 Comes with support for multiple hosts, and some switches
 Runs on OS X and Linux
VX is Cumulus Linux as a VM
Use Vagrant + VX + host boxes to spin up your entire
network (on a powerful server) or a section of it (on
your laptop) and test changes here before pushing it
out

Commit/Rollback in the Age of Automation
Master state is in the playbooks (or recipes), not the
device specific configuration themselves
Use source control (git is easy to get) to manage
playbook versions
Ansible’s validate ensures commands don’t fail due
to syntactic errors
Verifying Configuration ensures final state is as
desired
Testing changes in virtual land ensures you don’t
hose the box

Get Git

•#8: Separate Data From Code

Store Fabric-wide Data, Abstract Details
# Keep the numSpines and spine2leaf_ports consistent
# For example, if num_spines was 3, leaf_to_spine_ports would
# be "swp[1-3]".
num_spines: 4
num_leaves: 9
hosts_per_leaf: 1
# Keep the num_leaves and spine_to_leaf_ports consistent
# For example, with 4 leaves, spine_to_leaf_ports could be
# "swp[1-4]". The first server port is after the last ISL port
# For dual-attach ports, the number of leaves have to be even
spine_to_leaf_ports: "swp[1-9]"
leaf_to_spine_ports: "swp[1-4]"
leaf_to_server_ports: "swp[5-6]"
################### Routing Protocol Used ###########################
protocol: 'bgp'
################### SVI and Host IP CONFIGURATION ###################
lo_ip_subnet: '10.254.0.0/24'
################### Dual-Attach Server CONFIGURATION ################
# Update the clag-peer ports based on how many server ports are there
dual_attach_hosts: true
clag_peer_ports: "swp[7-8]"

#9 OWN YOUR CONFIGURATION
Let your configuration management system be the source of truth,
don't let your devices dictate your fate

❌ Traditional NOS that requires
you to tell it how to get there
---
# Add these VLANs
vlans_i_want:
- id: 200
name: prod
- id: 300
name: pre
- id: 400
name: dev
# Remove these VLANs
vlans_i_dont_want:
- id: 23
name: asd
# What happens with unknown VLANs?
✅ Modern OS that is able to understand
what you want and how to get there by itself
---
# Only VLANs specified here will be allowed
vlans:
- id: 200
name: prod
- id: 300
name: pre
- id: 400
name: dev
# Unknown VLANs will be removed

#10 DO NOT REINVENT THE WHEEL
Re-use DevOps tools, knowledge, experience, best practices...

#11 USE INTERNAL RESOURCES
Chances are your organization has a DevOps team already. Make use
of their knowledge and experience. Don't be a silo

#12 JOIN A COMMUNITY
Mailing lists, IRC and others are great sources of information. Ask and
learn from others, share your own experience, etc.

#13 DON'T FEAR PROGRAMMING
Because you already mastered CLIs and network
protocols that are more complex than most programming languages
and techniques

And knowing some Python and how to interact with an API
doesn't make you a programmer

#0 START SIMPLE
And keep iterating

Summary
Exploit regularity to create patterns,
automate patterns
Simplify configuration using unnumbered
interfaces, OSPF/BGP unnumbered
Validate every step

September 1, 2016 cumulusnetworks.com 59
•Agile Network Deployment
•Guest Speaker: To be Announced
•When: September 29
Next Month’s Webinar

CUMULUS, the Cumulus Logo, CUMULUS NETWORKS, and the Rocket Turtle Logo (the “Marks”) are trademarks and service marks of Cumulus Networks, Inc. in the U.S.
and other countries. You are not permitted to use the Marks without the prior written consent of Cumulus Networks. The registered trademark Linux® is used pursuant to a
sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis. All other marks are used under fair use or license from their
respective owners.
Thank You!
cumulusnetworks.com 60
Bringing the Linux Revolution to Networking
August 30, 2016

Webinar: Network Automation [Tips & Tricks]

More Related Content

What's hot

Viewers also liked

Similar to Webinar: Network Automation [Tips & Tricks]

More from Cumulus Networks

Recently uploaded

Webinar: Network Automation [Tips & Tricks]