After years of incremental F5 iRule changes without an overall design, we ended up in a state with poorly written iRules, little to no testability, and a process that required a human to manually make iRule changes for each environment as code was being promoted. Super manual, super error-prone, super lame. So we said enough is enough. We needed to establish standards and come up with a way to refactor the iRules and have a good level of certainty that we didn't break any functionality. So we wrote some standards, and then created a tool to help. In this session I'll be covering how we've changed how we look at F5 Load balancer configurations. Specifically, I'll cover how we transitioned to environment-agnostic iRules which are promotable through the development lifecycle and are fully testable. This effort eventually paved the way for us to create a self-service way for our marketing team to leverage version control and our continuous delivery system to create their own promotional redirects to support active and pending campaigns on our website.

1. 1
TREATING LOAD
BALANCER
CONFIGURATION
LIKE CODE
Presented By: Jesse Mauntel
6/19/2015SAN DIEGO DEVOPS MEETUP 1

2. 2
WHO AM I?
Operations Engineer who’s been in the industry
15+ years and has an extreme dislike for
configuration drift, system snowflakes, manual
administration, and time pirates.
@maunteljw
devopslove.blogspot.com
github.com/jmauntel
6/19/2015SAN DIEGO DEVOPS MEETUP 2

3. 3
AGENDA
So what’s the problem?
iRule Tester
Object cloner & orphan object audit
Self-service website redirects
6/19/2015SAN DIEGO DEVOPS MEETUP 3

4. 4
SO WHAT’S THE PROBLEM?
Work in a world of poorly written iRules
Little to no testability of iRules
Code promotion process that requires a
human to manually make iRule changes
for each environment
6/19/2015SAN DIEGO DEVOPS MEETUP 4

5. 5
CHALLENGES
The current iRule logic includes
environment-specific definitions in the
logic, which makes promotion of whole
iRules through environments impossible
6/19/2015SAN DIEGO DEVOPS MEETUP 5

6. 6
EXAMPLE
acme-qa-irule:
# Force sensitive acmeCommerce traffic to SSL
if { [class match [HTTP::uri] starts_with acmeCommerce-qa-class] } {
HTTP::redirect https://[HTTP::host][HTTP::uri]
# Send requests for Acme Information to the information tier
} elseif { [HTTP::uri] starts_with "/AcmeInformation/"} {
pool acmeInformation-qa-pool
}
6/19/2015SAN DIEGO DEVOPS MEETUP 6

7. 7
CHALLENGES
iRule changes are made by hand in all
environments, which is error-prone
6/19/2015SAN DIEGO DEVOPS MEETUP 7

8. 8
EXAMPLE
acme-dev-irule:
# Force sensitive acmeCommerce traffic to SSL
if { [class match [HTTP::uri] starts_with acmeCommerce-dev-class] } {
HTTP::redirect https://[HTTP::host][HTTP::uri]
acme-qa-irule:
# Force sensitive acmeCommerce traffic to SSL
if { [class match [HTTP::uri] starts_with acmeCommerce-dev-class] } {
HTTP::redirect https://[HTTP::host][HTTP::uri]
6/19/2015SAN DIEGO DEVOPS MEETUP 8

9. 9
CHALLENGES
Creating new environments is a manual,
time-consuming, and tedious process
bleh
6/19/2015SAN DIEGO DEVOPS MEETUP 9

10. 10
EXAMPLE
acme-dev-irule:
# Send all URIs that begin with /Website/ to acmeWeb pool
} elseif { [HTTP::uri] starts_with "/Website/" } {
pool acmeWeb-dev-pool
acme-qa-irule:
# Send all URIs that begin with /Website/ to acmeWeb pool
} elseif { [HTTP::uri] starts_with "/Website/" } {
pool acmeWeb-qa-pool
6/19/2015SAN DIEGO DEVOPS MEETUP 10

11. 11
CHALLENGES
Existing iRules do not have functional
tests, so there is no guarantee that a
change to an iRule won't break other pre-
existing logic
6/19/2015SAN DIEGO DEVOPS MEETUP 11

12. 12
EXAMPLE
acme-dev-irule:
# Send all URIs that begin with /Website/ to acmeWeb pool
} elseif { [HTTP::uri] starts_with "/Website/" } {
pool acmeWeb-dev-pool
> # Send store location details page to content tier
> } elseif { [HTTP::uri] contains "storelocation" } {
> pool acmeContent-dev-pool
# Send REST requests to acmeAPI tier
} elseif { ([HTTP::uri] starts_with "/rest/storelocation/allstores") } {
pool acmeAPI-dev-pool
6/19/2015SAN DIEGO DEVOPS MEETUP 12

13. 13
SOLUTIONS TO CHALLENGES
Revisited
The current iRule logic includes
environment-specific definitions in the
logic, which makes promotion of whole
iRules through environments impossible
6/19/2015SAN DIEGO DEVOPS MEETUP 13

14. 14
SOLUTION EXAMPLE
New iRule standards require environment
detection for variable assignment and
environment-agnostic logic
github.com/jmauntel/irule-standards
6/19/2015SAN DIEGO DEVOPS MEETUP 14

15. 15
SOLUTIONS TO CHALLENGES
Revisited
iRule changes are made by hand in all
environments, which is error-prone
6/19/2015SAN DIEGO DEVOPS MEETUP 15

16. 16
SOLUTION EXAMPLE
Since all new iRule logic is environment-
agnostic, environments no longer use
copies of iRules, but rather the exact same
iRule
Also, because iRule logic is identical in all
environments, automated promotion is now
possible
6/19/2015SAN DIEGO DEVOPS MEETUP 16

17. 17
SOLUTION EXAMPLE (CONT)
ltm virtual acme-dev-vs {
rules {
acmeVars-1.0.0-irule
acme-1.0.0-irule
insertPoolCookie-1.0.0-irule
}
}
ltm virtual acme-qa-vs {
rules {
acmeVars-1.0.0-irule
acme-1.0.0-irule
insertPoolCookie-1.0.0-irule
}
}
6/19/2015SAN DIEGO DEVOPS MEETUP 17

18. 18
SOLUTIONS TO CHALLENGES
Revisited
Creating new environments is a manual,
time-consuming, and tedious process
6/19/2015SAN DIEGO DEVOPS MEETUP 18

19. 19
SOLUTION EXAMPLE
acmeVars-1.0.0-irule:
# Assign environment
if { [IP::local_addr] equals "10.0.0.50" } { set my_env "prd" }
elseif { [IP::local_addr] equals "10.254.1.136" } { set my_env "qa" }
else { [IP::local_addr] equals "10.254.1.137" } { set my_env "dev" }
# Pool definitions, sorted alphabetically
if { $my_env equals "prd" } { set acmeWeb-pool "acmeWeb-prd-pool" }
elseif { $my_env equals "qa" } { set acmeWeb-pool "acmeWeb-qa-pool" }
else { $my_env equals "dev" } { set acmeWeb-pool "acmeWeb-dev-pool" }
6/19/2015SAN DIEGO DEVOPS MEETUP 19

20. 20
SOLUTION EXAMPLE (CONT)
acmeVars-1.0.0-irule – (applied to acme-dev-vs)
# Send all URIs that begin with /Website/ to acmeWeb pool
} elseif { [HTTP::uri] starts_with "/Website/" } {
pool ${acmeWeb-pool}
acmeVars-1.0.0-irule – (applied to acme-qa-vs)
# Send all URIs that begin with /Website/ to acmeWeb pool
} elseif { [HTTP::uri] starts_with "/Website/" } {
pool ${acmeWeb-pool}
6/19/2015SAN DIEGO DEVOPS MEETUP 20

21. 21
SOLUTIONS TO CHALLENGES
Revisited
Existing iRules do not have functional
tests, so there is no guarantee that a
change to an iRule won't break other pre-
existing logic
6/19/2015SAN DIEGO DEVOPS MEETUP 21

22. 22
SOLUTION EXAMPLE
After searching online and not finding an
existing iRule testing tool, I wrote one
irule-tester is written in Bash and leverages Curl
to make web requests, and then validates that
the response matches an expectation
github.com/jmauntel/irule-tester
6/19/2015SAN DIEGO DEVOPS MEETUP 22

23. 23
IRULE TESTER
6/19/2015SAN DIEGO DEVOPS MEETUP 23

24. 24
IRULE TESTER: OVERVIEW
 Written in Bash and uses Curl for requests
 Has simple and extended testing modes
 Supports multiple output formats, including TAP
 All tests are stored in source control
 Changes to any test are validated with Jenkins
6/19/2015SAN DIEGO DEVOPS MEETUP 24

25. 25
IRULE TESTER: JENKINS
INTEGRATION
 Tests are executed after any change in source, or at least daily
 Test failures notify the team via email
 Tests are executed before and after iRule changes in all
environments
6/19/2015SAN DIEGO DEVOPS MEETUP 25

26. 26
IRULE TESTER
DEMO
6/19/2015SAN DIEGO DEVOPS MEETUP 26

27. 27
F5 OBJECT CLONER
6/19/2015SAN DIEGO DEVOPS MEETUP 27

28. 28
F5 OBJECT CLONER
So if I’m versioning iRules and data
groups now, is there an easy way to
clone them?
Copy/paste is error-prone and lame
6/19/2015SAN DIEGO DEVOPS MEETUP 28

29. 29
F5 OBJECT CLONER
I wrote a utility for that
Clones iRules and data-groups on a
single F5 unit or between two different
units
github.com/jmauntel/f5-utils
6/19/2015SAN DIEGO DEVOPS MEETUP 29

30. 30
F5 OBJECT CLONER
Usage: clone-object.sh -o {data-group,rule} -s <SOURCE_OBJECT> -d <DEST_OBJECT> -S
<SOURCE_F5> -D <DEST_F5>
-d destination object name
-D destination F5
-o object type
-s source object name
-S source F5
All arguments are REQUIRED
Example:
# clone-object.sh –o data-group –s UserIPs-1.0.0-class –d UserIPs-1.1.0-class –S
10.0.0.1 –D 10.0.0.1
6/19/2015SAN DIEGO DEVOPS MEETUP 30

31. 31
F5 OBJECT CLONER
DEMO
6/19/2015SAN DIEGO DEVOPS MEETUP 31

32. 32
F5 ORPHAN OBJECT AUDIT
6/19/2015SAN DIEGO DEVOPS MEETUP 32

33. 33
F5 ORPHAN OBJECT AUDIT
As time passes, a collection of F5
objects can build up, cluttering your
F5 config
Why not use a tool to audit for unused
objects and purge them?
6/19/2015SAN DIEGO DEVOPS MEETUP 33

34. 34
F5 ORPHAN OBJECT AUDIT
Locates objects of a given type that only
have a single reference in the bigip.conf
file
Usage: ./f5orphan.pl -f <filename> -i <configuration item>
Valid configuration items:
rule | data-group | profile | snatpool | pool | node | monitor
Example:
# f5orphan.pl –f /config/bigip.conf –i rule
6/19/2015SAN DIEGO DEVOPS MEETUP 34

35. 35
F5 ORPHAN OBJECT AUDIT
DEMO
6/19/2015SAN DIEGO DEVOPS MEETUP 35

36. 36
SELF-SERVICE WEBSITE
REDIRECTS
6/19/2015SAN DIEGO DEVOPS MEETUP 36

37. 37
WEBSITE REDIRECTS
Our WebOps team was constantly asking
us to add vanity redirects to the website in
support of marketing campaigns…
6/19/2015SAN DIEGO DEVOPS MEETUP 37

38. 38
WEBSITE REDIRECTS
…so we made it self-service
6/19/2015SAN DIEGO DEVOPS MEETUP 38

39. 39
REDIRECTS: STEP ONE (MANUAL)
Created a git repo that contains
redirects.config
WebOps team adds redirects to the
redirects.config file and pushes changes
6/19/2015SAN DIEGO DEVOPS MEETUP 39

40. 40
REDIRECTS: STEP ONE (MANUAL)
Example of redirects.config:
# Redirects are defined as Key|Value - The Key is the origin and the Value is the
destination
# Example:
# Somewhere|Somewhere/Else
# Result:
# 24hourfitness.com/Somewhere would redirect to 24hourfitness.com/Somewhere/Else
/blog|/community/blog/
6/19/2015SAN DIEGO DEVOPS MEETUP 40

41. 41
REDIRECTS: STEP TWO
(AUTOMATED)
Bamboo build job:
 Notices a change to the repo and gets latest updates
 Retrieves data-group creation script
 Uses data-group creation script to lint redirects.config
and create environment-specific LTM config files for
later merge
 Creates iRule Tester seed file for redirect validations
 Stages files for later deployment
6/19/2015SAN DIEGO DEVOPS MEETUP 41

42. 42
REDIRECTS: STEP THREE
(AUTOMATED)
Bamboo deployment job (DEV):
 Automatically triggers based on successful build
 Copies LTM config file to target LTM and merges with
running configuration
 Executes iRule Tester script to verify all redirects work
as expected
 Notifies WebOps team via email that the deployment
to DEV was successful
6/19/2015SAN DIEGO DEVOPS MEETUP 42

43. 43
REDIRECTS: STEP FOUR
(MANUAL)
Bamboo deployment job (QA):
 WebOps pushes button to launch deploy of the DEV
release to QA which:
 Copies LTM config file to target LTM and merges
with running configuration
 Executes iRule Tester script to verify all redirects
work as expected
 Notifies WebOps team via email that the
deployment to QA was successful
6/19/2015SAN DIEGO DEVOPS MEETUP 43

44. 44
REDIRECTS: STEP FIVE (MANUAL)
Bamboo deployment job (PRD):
 WebOps notifies Operations team of the release that needs to be
promoted to PRD.
 Operations team keys in the required change control
documentation
 Operations team presses button to launch deploy of the QA
release to PRD which:
 Copies LTM config file to target LTM and merges with running
configuration
 Executes iRule Tester script to verify all redirects work as expected
 Notifies WebOps team via email that the deployment to PRD was
successful
6/19/2015SAN DIEGO DEVOPS MEETUP 44

45. 45
WEBSITE REDIRECTS
DEMO
6/19/2015SAN DIEGO DEVOPS MEETUP 45

46. 46
QUESTIONS?
Contact Info:
Jesse Mauntel
@maunteljw
devopslove.blogspot.com
github.com/jmauntel
6/19/2015SAN DIEGO DEVOPS MEETUP 46