In this presentation Jonathan Hammer from the NetOps team shows our approach to automate our firewall flows/lifecycle management.

1. SRX Automation @ GRPN
Jon Hammer
hammer@groupon.com

2. Background

3. NetDevOps @ GRPN
Netops git commits at Groupon
2014 - 2017

4. The Problem

5. The Problem: Time/Energy

6. The Problem: Environmental Complexity

7. The Problem: Lack of Standards

8. The Problem: Security/Reliability

9. Goals
● First Milestone:
○ Address 80% of firewall request tickets
○ Achieve consistency and standardization in rules
○ Increase reliability
● Second Milestone:
○ Service based
○ Improve speed and reliability

10. Firewall Automation
create_firewall_rule.py

11. Step 1 - Getting the ACL Request
create_firewall_rule.py - step 1

12. Standardizing Inputs: Firewall Request Form

13. Getting the ticket somewhere we can use it
jira = netops_jira.JiraSearch()
jira.query = 'project in (NETOPS) AND status in (OPEN) AND component
in (Firewall)'
data = jira.search()
for request in data:
ticket = netops_jira.FirewallTicketParser(request)
netops_jira.py

14. Jira to Python - netops_jira.py objects
In [11]: ticket.source
Out[11]: {'Group 1': ['host1.grpn', 'host2.grpn',
'host3.grpn']}
In [12]: ticket.destination
Out[12]: {'Group 1': ['otherhost11.ls', 'otherhost12.ls']}
In [13]: ticket.port
Out[13]: {'Group 1': ['2222', '80', '99-100']}

15. Step 2 - Determine which Devices to
Configure
create_firewall_rule.py - step 2

16. Where does a policy go?

17. Global standardization of policy flow
● Decide on a standard for directionality
● Treat VPN Concentrators like Backbone routers
Backbone
Internal
Network
● Result: FW to configure is deterministic
Primary
VPN BB Rtr

18. Step 3 - Determining the Zones
create_firewall_rule.py - step 3

19. Getting the zone of a destination
● Steps:
○ Resolve the host IP
○ Connect to the determined firewall
○ Do a route lookup for the host IP
○ Find the outgoing interface for that
route
○ Check the security zones table for
that interface
hammer@fw> show route 8.8.8.8
inet.0: 258 destinations, 259 routes (258 active, 0 holddown, 0
hidden)
+ = Active Route, - = Last Active, * = Both
0.0.0.0/0 *[Static/5] 36w5d 14:05:18
> to 1.2.3.4 via ge-0/0/7.0
hammer@fw> show security zones
Security zone: untrust
Send reset for non-SYN session TCP packets: Off
Policy configurable: Yes
Screen: untrust-screen
Interfaces bound: 1
Interfaces:
ge-0/0/7.0

20. Determining Zone Step 1: Resolve the host IP
Step 1 - Resolve the host IP
Step 2 - ???
Step 3 - Profit!
● There are hidden complexities
● Why not just >show route host1.grpn ?
● Round Robin DNS
● IP used in request?
● Special cases in request?

21. Determining Zone Step 1a: Parsing the source/dest
● Goal - {name : address}
○ Hostname → Forward Lookup → IP → {hostname : IP}
○ IP → Reverse Lookup → Hostname → {hostname: IP}
○ Subnet → {subnet: subnet}
○ Special cases:
■ E.g., “vpn users” → {‘vpn_users’: predefined_subnet}

22. Determining Zone Step 2: Connecting to the FW
from device_connection import JUNOSConnection
Password = “thepassword”
myconn = JUNOSConnection(username, “fw1.grpn”, password)
myconn.Connect()

23. XML parsing
routes = myconn.get_route_information()
Determining Zone Step 3: Route lookup
Screen scraping / regex
> show route

24. Tables/Views Refresher - XML config
hammer@fw1> show security zones untrust | display xml
<rpc-reply xmlns:junos="http://xml.juniper.net/junos/12.3X48/junos">
<zones-information xmlns="http://xml.juniper.net/junos/12.3X48/junos-zones" junos:style="detail">
<zones-security>
<zones-security-zonename>untrust</zones-security-zonename>
<zones-security-interfaces-bound>2</zones-security-interfaces-bound>
<zones-security-interfaces>
<zones-security-interface-name>ae0.0</zones-security-interface-name>
<zones-security-interface-name>ae1.0</zones-security-interface-name>
</zones-security-interfaces>
</zones-security>
</zones-information>
<cli>
<banner></banner>
</cli>
</rpc-reply>

25. Tables/Views Refresher - RPCs
hammer@fw1> show security zones untrust | display xml rpc
<rpc-reply xmlns:junos="http://xml.juniper.net/junos/12.3X48/junos">
<rpc>
<get-zones-information>
<get-zones-named-information>untrust</get-zones-named-information>
</get-zones-information>
</rpc>
<cli>
<banner></banner>
</cli>
</rpc-reply>

26. Tables/Views Refresher - yaml definitions
SecurityZonesTable:
rpc: get-zones-information
args:
detail: True
args_key:
get-zones-named-information
item: .//zones-security
key: zones-security-zonename
view: SecurityZonesTableView
SecurityZonesTableView:
fields:
interfaces_bound: zones-security-interfaces-bound
interfaces: zones-security-interfaces/zones-security-interface-name

27. Tables/Views: Refresher, part 4 - XML sucks
In [25]:
zones.xpath('zones-security[zones-security-zonename]')
Out[25]:
[<Element zones-security at 0x7f10bfa8cb90>,
<Element zones-security at 0x7f10bfa8ca70>,
<Element zones-security at 0x7f10bfb94b90>,
<Element zones-security at 0x7f10bfb94f80>,
<Element zones-security at 0x7f10bfa950e0>,
<Element zones-security at 0x7f10bfa95440>,
<Element zones-security at 0x7f10bfa95f80>]

28. Determining the zone step 4: Get the zone from the interface
In [8]: zones
Out[8]: SecurityZonesTable:fw1.grpn: 8 items
In [9]: zones[1].name
Out[9]: 'trust'
In [10]: zones[1].interfaces
Out[10]: ['ae2.101', 'ae3.101']
In [82]: route_table
Out[82]: junos_route_table:fw1.grpn: 941 items
In [83]: route.destination
Out[83]: u'10.1.37.0'
In [84]: route.prefix_length
Out[84]: 24
In [85]: route.outgoing_interface
Out[85]: u'ae2.101'

29. Flow for getting the zone
RPC calls for
route, zone table
Lookup outgoing
interface in route
view, and find it
in the zone view
Yes
More
hosts? Close connectionNo
Firewall Connection
Object
Some tickets involve
hundreds of lookups,
extremely resource intensive!

30. Optimizing, AKA be nice to your production devices
Firewall
Connection
Object
Close connection
Pull entire
routing and
zone table
Build offline
queryable FW
object with
routing/zone
tables
Build radix
object, tag every
route with zone
info
Route lookup for
10.20.100.1
More
hosts?E.g.,
zone=web
Yes
Maximum of two calls to the FW
regardless of number of hosts

31. Offline SRX object in action
In [18]: srx_obj = srx.BuildSRX(route_table=route_table, zone_table=zone_table)
In [19]: srx_obj.get_ip_zone('8.8.8.8')
Out[19]: 'untrust'

32. Radix Library
In [7]: route =
fw_obj._rtree.search_best('8.8.8.8')
In [8]: route.data
Out[8]:
{'next_hop': u'10.1.1.1',
'outgoing_interface': u'ae2.0',
'zone_name': 'untrust'}
We tag zone into every route as we build the radix table
https://pypi.python.org/pypi/py-radix

33. Step 4 - Do we even need this rule?
create_firewall_rule.py - step 4

34. Preventing Configuration Bloat

35. Automating policy match detection
Policy Object:
In [13]: sec_pol_table[100].items()
Out[13]:
[('application_term_source_high', '0'),
('to_zone', 'trust'),
('description', None),
('application', 'tcp_8000'),
('destination', ['host1.grpn', 'host2.grpn']),
('from_zone', 'edge'),
('policy_name', 'NETOPS-5864'),
('application_term_destination_low', '8000'),
('application_term_destination_high', '9000'),
('source', ['box1.omg', 'box2.omg']),
('application_term_protocol', 'tcp'),
('destination_prefix', ['10.1.1.189/32', '10.1.1.132/32']),
('source_prefix', ['10.2.2.222/32', '10.2.2.88/32']),
('application_term_source_low', '0')]
Detail view conveniently
flattens all the IPs/names!
Thanks JUNOS!

36. Show the user
ProdOps Commandline User added a comment - 12/Sep/17 6:05 PM
Group 2: Existing policy for Group 2 matches this configured policy: NETOPS-5281

37. Step 5 -
ACL Standardization
create_firewall_rule.py - step 5

38. Standardizing address-book entries: Before
webapp1 10.1.1.1/32;
web-app1.grpn 10.1.1.1/32;
web-ap__grpn 10.1.1.1/32;
dbhost_netops-1234 172.16.1.10/32
emea_vpn_users 10.2.1.0/24;
vpn_users__emea 10.2.1.0/24;
prod_subnet_10.10.1.0 10.10.1.0/24;
address-set vpn_users {..}
address-set all_vpn_users {..}
address-set dns_servers {..}
Multiply by several thousand
address-books + address-sets!

39. Standardizing address-book entries: After
web-app1.grpn 10.1.1.1/32;
If Host (/32): <dns-entry>, <ip>
db1__1.grpn 172.16.1.10/32;
db1__2.grpn 172.16.1.11/32
If RR DNS: <dns-entry> + __<n>, <ip>
10.32.100.0/24 10.32.100.0/24;
If Subnet: <subnet> <subnet>
address-set src_NETOPS-1234 {
web-app1.grpn
}
address-set dst_NETOPS-1234 {
db1__1.grpn
db1__2.grpn
}
Address set: src or dst + ticket
number
Static exception: address-set vpn_users {..}

40. Standardizing Policies: Before
policy dev-hosts--to--monitoring-rw-vip-sjc {
match {
source-address [ devhosts dev12 prodnet_33 devhosts__new ];
destination-address gdr-sjc1-prod-graph-ro-vip__sjc1;
application monitoring_services;
}
then {
permit;
}
}

41. Standardizing Policies: After
policy NETOPS-7432__2 {
match {
source-address src_NETOPS-7432__2;
destination-address dst_NETOPS-7432__2;
application tcp_10906-10910;
}
then {
permit;
}
}
No more trying
to engineer
context into
configurations!

42. Side benefit of Standardization
● Show policy
● Show source address(es) / sets
● Show destination address(es) / sets
● Show application(s)
● show configuration | match <ticket> | display set

43. Addressbook entry re-use
● It doesn’t exist
○ New address-book entry added
● It exists
○ SRX Ignores it
● It exists and is different:
○ Overwrite with new value
○ (Verify by admin)

44. Cleaning Up Legacy Stuff

45. Step 6 -
Representing the ACL in code
create_firewall_rule.py - step 6

46. srx.py
In [13]: address1.ip
Out[13]: '1.1.1.1/32'
In [14]: address1.name
Out[14]: 'myhost1.grpn'
In [15]: address1.zone
Out[15]: 'trust'
In [16]: print address1.config()
security {
zones {
security-zone trust {
address-book {
address myhost1.grpn 1.1.1.1/32;
}
}
}
}

47. Configuration logic stored outside of jinja
In [19]: app = srx.Application('foo')
In [20]: app.port = "9999"
In [21]: app.port = "9999-65535"
In [22]: app.port = "9999-65536"
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
{...}

48. Creating the configuration objects
Parsed Sources /
Destinations
Zones from
device
srx.py
Address objects
Address-set
objects
Application
Objects
Policy Objects
Destination
port(s)
SRX JUNOS
configuration

49. create_firewall_rule.py
create_firewall_rule.py

50. Firewall Automation Flow, part 1
FW Connection
Object
3x RPC:
routing/zone
/policy
tables
JIRA API
Parse ticket
(netops_jira.py)
into object
Iterate over each
group in the
ticket
create_firewall_ru
le.py
Username,
Ticket #
Determine which
FW to configure
based on
standard
Parse
src/dst
Parsable
Assign to oncall
for manual
processing
Not
parsable
Build offline SRX
(close previous
connection)
Ticket # json
Assign to oncall
Leave in retry
queue
Unable to
determine fw
Connection fail

51. Firewall Automation Flow, part 2
Verify rule
doesn’t already
exist
Build ACL objects
(srx.py library)
Get zone info for
src/dst
Save into
per-colo
dictionary
For each colo,
push all rules to
active/backup
FW
Comment diffs to
ticket + resolve
Comment +
Resolve ticket
Doesn’t
exist
Exists
Offline SRX
Object
More groups in
request?
No
Yes
Process other
groups

52. Step 7 -
Ensuring the Firewalls are in Sync

53. Tools for firewall synchronization - sync_policies.py

54. Tools for firewall synchronization - fwsync_check.py

55. Step 7 - Optimization

56. Firewall Automation: The Service
● Created service code that runs periodically and finds “Approved” Firewall tickets
● The code will run create_firewall_rule.py on those tickets
● Engineers simply need to review the request for sanity/security purposes, and click approve.
● No need to run any script or open up a terminal
● Success = Comment + Resolve
● Fail = Re-assign to pagerduty API
● Only see tickets that require approval or troubleshooting
● Effective SLA: 30 minutes

57. Caching, AKA being nice to our Production devices

58. New Feature: Policy Match

59. Conclusion

60. Questions

61. SRX Automation @ GRPN
Jon Hammer
hammer@groupon.com