The document discusses intermediate Python techniques for automation, focusing on empowering teams and improving productivity through automation. Key concepts include iterators, generators, data loading with JSON/YAML/CSV, and templating with Jinja2, along with best practices in programming and data validation. It emphasizes the importance of decomposing problems and utilizing Pythonic techniques to enhance efficiency and maintainability in coding.

1. intermediate Python techniques
for the automation journey
April 2015
Jeremy Schulman
@nwkautomaniac
Hardly Enough Theory
Barely Enough Code

2. Agenda
● Automation - Why Me?
● Learn key concepts
● Learn cool Pythonic techniques

3. Why Automate?
Don’t be the bottleneck
Empower your team
Increase agility and
productivity
Keep the business running
Automate the mundane
Free up your time
Improve uptime and
reliability
Your time and your
expertise

4. Theory
● Composers, constructors
and consumers
● Decomposition
● What matters most
● Zero, one, or infinity
What You will Learn
Code
● Iterators and generators
● Files - JSON, YAML, CSV
● Validation - JSON schema
● Templating - Jinja2
○ Basics
○ Programming logic
○ Custom extensions

5. Theory
Just a few slides ...

6. Role Relationships
Design things providing instructions to others
Combination of art and science
Creative processes and best-practices
Composers
Build finished goods based on the instructions
Practiced skills - years to master the craft
Specialized domain knowledge and tools
Constructors
Use finished goods
Generally does not require much skill
Sometimes requires use of tools / products
Consumers

7. Decomposition
Everyone has favorites:
● Modeling techniques ...
● Diagramming techniques ...
● Process techniques ...
Breaking a complex problem or system into parts
Easier to conceive, understand, program and maintain
Also known as factoring
Entity Relationship DiagramFlowchart Diagram

8. Apply Decomposition
Consider a switch configuration file ...
● Create shared and unique data-sets
● Define variables into explicit and
derived / calculated items
● Understand cohesion of data and
code
● Reuse the data for other purposes,
like validation and operations
● Data can be used for multi-vendor
Step 1: separate "data" from "code"
Step 2: decompose "data"
Step 3: decompose "code"
Step 4: goto step 2 until manageable
Why do this?

9. What Matters Most?
Does it do what it is supposed to do?
Are there even any requirements?
Is high availability / fault tolerant a consideration?
Functionality
How fast does it run?
Is speed really even an issue?Performance
Does it need to handle the growth of your environment?
Do you need to deal with concurrent processing?Scale
When it breaks, can you fix it (quickly)?
Can other people work on it?
What happens if you leave?
Maintainability

10. Zero, One, or Infinity
No need to do something so just don't do itZero
I must think about everyone else that might
use this or have to deal with it one day.
Empathy.
Infinity
Doing it only for myself and I don't need to
worry about anyone else using it or what
happens when I'm gone
One

11. Code
Iterators and Generators

12. Iterators
Loop over things
● list, dictionary, set, tuple
● files - each line in text file
● strings - each character in string
● class objects supporting the iter
protocol
● generators - see next slide
List iteration code example
Example output

13. Why use Generators?
Lazy Evaluation
Evaluation strategy which delays
the evaluation of an expression
until its value is needed
Benefits
● Reduce memory
● Generate data on-demand
● Generate “infinite” data
● Performance increases
● Use as abstract objects
○ as function args
○ treat like any other iterablehttp://en.wikipedia.org/wiki/Lazy_evaluation

14. Generators
Produce sequence of
things
● behave like iterators, but lazy
● generator functions use yield
● generator expressions are
comprehensions enclosed by
parentheses
Generator expression example
Generator function example

15. Code
Loading Data: JSON, YAML and CSV

16. Loading Data from Files - JSON
Example file Example code
Example output

17. Loading Data from Files - YAML
Example file Example code
Example output

18. Loading Data from Files - CSV
Example file Example code
Example outputDo you see the differences?
1. id is a string, not integer
2. name is not the key
How to fix this?

19. Loading Data from Files - CSV (fixed!)
Example code
Example output

20. Demo: Putting it all together
Task
Load all of the files, of
different types, into a single
dictionary variable
Pythonic techniques
● iterators and generators
● lambda functions
● reduce function
● iterables.chain
● functools.partial

21. Example Code

22. Demo: Filtering while loading
Task
While reading the data files,
filter out records that match
a given criteria
id >= 20
Pythonic techniques
● generator
● compression
● … extract first/only dict item
● … with if statement

23. Example Code
Example output

24. Code
Templating with Jinja2

25. Templating with Jinja2 - Intro
Loaded from text files
Created dynamically by program
Commonly used for configuration
{% if age > 12 %}
...
{% endif %}
Hello {{ name | capitalize }}
Hello {{ name }}
Template content is text
Programmatic controls
Filters and extensions
Variable string interpolation

26. Very Simple Example - Jinja2

27. Programmatic Controls - Jinja2
{% if age > 20 %}
{% else %}
{% endif %}
if / then / else
{% for this in some_iterable %}
{% endfor %}loop over iterators
{% set myvar = some_func() %}create variables
{% include "this file.j2" %}
{% include file_name %}include other files

28. Load template searching directories
Jinja2 "Loader" determines how
to find the template files
Jinja2 "Environment" allows you
to control aspects template
loading and rendering

29. Example: Program Control

30. Custom Filters - Jinja2
Create a sorted key list by using
the inner data dictionary
Bind the function to the Jinja2
environment "filters" dictionary

31. Example Output

32. Including other files - Jinja2

33. Example Output
The state_list variable using
in the template was created in
the python code using the
groupby function, see next
slide ...

34. Example Code: Report
https://docs.python.org/2/library/itertools.html#itertools.groupby

35. Code
Data Validation

36. Data Validation - JSON Schema
Used to validate data
Verify the syntactic contents
Both inputs and outputs
Schema is written in JSON
Other schema mechanism:
● ASN.1 - SNMP
● YANG - NETCONF
● XSD - XML
Example Data
Example Schema

37. More interesting example
Data Schema
● One "record" per file
● Record key is max 16 alpha-chars
● Only the properties:
○ state
○ id
○ job

38. JSON Schema - Docs
http://json-schema.org/latest/json-schema-validation.html#toc

39. Automation Summary
Programming is a tool
not a career change
It's not about you
it's about everyone else
Try being right now
rather than being right

40. More?
● JSON - https://docs.python.org/2/library/json.html
● JSON Schema - http://json-schema.org
● YAML - http://pyyaml.org/wiki/PyYAMLDocumentation
● CSV - https://docs.python.org/2/library/csv.html
● Jinja2 - http://jinja.pocoo.org/
● itertools - https://docs.python.org/2/library/itertools.html
● jq - https://github.com/stedolan/jq
● nwkautomaniac - https://github.com/jeremyschulman/nwkautomaniac

41. Jeremy Schulman
@nwkautomaniac
Thank you!