Matt "Grizz" Griswold and Chris Grundemann are both IX founders, internetworking experts, and automation proponents. With over 4 decades of combined experience they are now turning to sharing what they've learned about automating BGP and interconnection through a set of open source tools, along with support and services for those that need it. This talk will share what they have learned both from personal experience as well as through dozens of recent interviews with IX operators and interconnection engineers over the past several months. Including common challenges, productive methodologies, and best practices. The highlight of the talk will be announcing and describing two open source automation tools built to make interconnection and BGP easier for everyone. One is ixCtl, which is built to automate the most common and problematic tasks involved in running an internet exchange point, particularly configuring and managing secure route servers. The other is PeerCtl, which is built to automate the most common and problematic tasks involved in interconnecting an AS; from bilateral and multilateral peering to PNI and also transit connections. Code for both (along with several other tools) is available on GitHub: https://github.com/fullctl. Speaker: Chris Grundemann Speaker: Matt Griswold

2. Agenda
1. Who are we?
2. What are we talking about?
3. What have we learned?
4. What is the solution?
5. What are we doing to help?
6. What can you do to help?
7. What else should we do?
8. Comments thinly veiled as questions.

3. Chris Grundemann
● 20+ years in networking
● Co-founder of IX-Denver
● OIX Director
● 8 patents in NetTech
● Co-founder and CSO at FullCtl
● 20+ years in software development
● Co-founder of UIX / ChIX
● OIX Director
● PeeringDB Maintainer
● Co-Founder and CTO at FullCtl
Matt “Grizz” Griswold

4. Sure, but why are you here?
● Age of Interconnection
● BGP is hard (for some)
● Lack of Standardization (for all)
● Lack of Automation (for many)
?

5. Age of Interconnection
● Peering managers on cruise ships
● Death of peering?

6. Age of Interconnection
● Cloud
● COVID
● Beyond LAN & WAN
● Digital Equity
● Security
● Reliability

7. But… BGP is hard (for some)
● For many, it’s a FLW, despite being a TLA
● Traditionally avoided by most
○ Enterprise
○ Public Sector
○ SMB (of course?)
● Also hard to find at
○ WISPS
○ FISPS
○ Electric Co-ops
● And now we have RPKI
○ Not getting easier…

8. Plus: Lack of Standardization
● Most networks heterogeneous
○ And Bespoke
● Unique policy requirements
○ TE, etc.
● Communities in use by ISP/IX/etc
○ More than just RTBH
● How-to request interconnection
○ Coordination can be painful
● LOAs
○ Don’t get us started…

9. And: Lack of Automation
● Almost nobody has “enough”
● Many lacking SoT
● Most lacking end-to-end flow
● Hard to automate w/out standard(s)
● Tough to test w/out lab (dev env)

10. On Automation…
A couple excerpts from:
2023 State of Network Automation Survey Results

11. How Automated Are We?
0%
-
Nothing is
automated,
everything
is always
done
manually.
100%
-
Everything is
automated,
nothing is
ever done
manually.

12. So, How Automated Are We?
0% - 20% = 24
30 - 40% = 17
50 - 60% = 20
70 - 80% = 17
90 - 100% = 0
50% and Below: 50
50% and Above: 37
Mean: 42.43%
Median: 40%

13. Automation Skills Gap?

14. Automation Skills Gap?

15. Automation Skills Gap?

16. Expert? Yes Maybe No
Networking 65% 21% 14%
Interconnection 58% 19% 23%
Automation 27% 33% 40%
Automation Skills Gap?

17. Expert? Yes Maybe No
Networking 65% 21% 14%
Interconnection 58% 19% 23%
Automation 27% 33% 40%
73% Uncertain of
Automation Skills
Automation Skills Gap?

18. Bridging the Gap
Automation Fundamentals (Quick Primer)
How-To: Interconnection Automation

19. Summary of that Survey
78 Diverse Responses
40% of Network Automated (Median)
90% “Homebrewing” && 80% Using OSS (<31% $W)
94% Using Python && At Least 46% using Ansible
55% of Orgs have No Dedicated Automation Staff
73% of Respondents Uncertain of Automation Skills

20. Automation Fundamentals (Quick Primer)
● Software Defined Networking (SDN)
● Automation
● Orchestration
● Feedback
● Network Validation / Validation + Automation
● Network Observability
● Network Virtualization
● Network Function Virtualization (NFV)
● Infrastructure as Code (IaC) / Network as Code (NaC?)
● Intent Based Networking (IBN)
● NetDevOps –> Network, Development + Operations

21. How-To: Interconnection Automation
● Standardization
● Source(s) of Truth (SoT)
○ Leverage PeeringDB
● Orchestrate Operations
○ Provide Self-Service

22. Standardization
● Industry wide
○ Best Current Operational Practices (BCOP) - https://github.com/Open-IX/BCOP
○ Speaking of OIX: https://www.oix.org/standards-and-certifications/
○ BGP Communities
○ Cross-Connections
○ LOAs
○ Peering requests (more on this later…)
● Network Wide
○ Configurations
○ Policies
○ Procedures
○ Document >> Standardize >> Automate

23. Source(s) of Truth (SoTs, S’soT, idk…)

24. Orchestrate Operations

25. So… What are we (FullCtl) doing about interconnection?
● ixCtl
● PeerCtl
● PrefixCtl
● …you get the idea

26. So… What are we (FullCtl) doing about interconnection?
● ixCtl
● PeerCtl
● PrefixCtl
● …you get the idea
● Taking the best of all of “us” and embedding that knowledge into simple tools

27. FullCtl Suite
https://github.com/fullctl/
P
r
e
f
i
x
C
t
l
aaaCtl
DeviceCtl
PeerCtl
ixCtl

28. FullCtl Suite
● modern, modular, Unix philosophy services
● 100% API driven
● manage infrastructure
● automate provisioning
● source of truth for wider automation efforts
https://github.com/fullctl/

29. Foundations :: aaaCtl
https://github.com/fullctl/aaactl
Granular
Permissions
Fine grained user and role
permissions down to each
object level
Audit logs
Audit logs for other running
services.
Oauth
|
SAML
Integrates
with
existing
aaa
services.
Azure
AD,
Okta,
PeeringDB,
et
al
Oauth | SAML
Integrates with existing
aaa services.
Azure AD, Okta, PeeringDB,
etc

30. Foundations :: aaaCtl
● Authentication and Authorization gateway
○ Client - integrates with existing directories
○ Provider - can be used as a SoT
● RBAC permissioning
● Audit logging
https://github.com/fullctl/aaactl

31. Foundations :: DeviceCtl
● Source of truth for devices
● Bidirectional sync to other sources of truth
○ Netbox, Nautobot, Proprietary systems, etc
○ PeeringDB facilities, geoloc data
● Caching and verification
https://github.com/fullctl/devicectl

32. Metadata :: PrefixCtl
Used as a SoT for prefixes
Bidirectional sync
Attaches and updates metadata
● IRR
● RPKI status
● IP reputation
● Geoloc data
https://github.com/fullctl/prefixctl *OSS coming soon

33. ixCtl
An automation platform purpose built for internet exchange operators.
● modern, modular application
● 100% API driven
● manage infrastructure
● automate provisioning
● source of truth for wider automation efforts
● connected networks get access to the PeerCtl dashboard
https://github.com/fullctl/ixctl

34. ixCtl turnkey route servers
1. Click “import” to pull baseline data on connected networks from PeeringDB
2. Hit “configure route server” and add basic info to generate secure
configurations
3. 1 time ansible (VM/BM) or helm (k8s) deploy for managed containers, or curl
into your own setup
4. enjoy karmic bliss, knowing that your route servers are automatically
contributing to the good of the internet!
https://github.com/fullctl/ixctl

35. ixCtl - Not “Just” Route Servers
● SoT for devices and ports
● Building graphs
● Manage members / IX-F output for PeeringDB, etc
● Allow member access to update mac address, max prefix, etc
https://github.com/fullctl/ixctl

36. ixCtl - Coming Soon
● Public Portal
● Client Portal
● LOA Generation
● AS112 Configuration
● Advanced Route Server Management
● Automated Switch Configuration
● $your_idea_here
https://github.com/fullctl/ixctl

37. PeerCtl
Interconnection automation and workflow management.
● modern, modular, Unix philosophy services
● 100% API driven
● manage infrastructure
● automate provisioning
● source of truth for wider automation efforts
● Open Source or As A Service
https://github.com/fullctl/peerctl

38. PeerCtl
Peering Toolkit
1. You select an IX. PeerCtl shows you all of the possible peers at that location.
2. You select a peer. PeerCtl provides BGP configurations and email templates
to get connected.
3. You want more. PeerCtl shows you all of the other mutual exchanges where
you can interconnect with this peer.
https://github.com/fullctl/peerctl

39. PeerCtl - but wait, there’s more
● All kinds of “peers”
○ Bilateral, multilateral, PNI, transit, transport
● Source of truth for your Interconnection efforts
○ Sync with third party inventory via DeviceCtl
○ Stay virtual with PDB Ports
● Extensibility
○ PeerCtl BGP configurations are extensible and customizable with templates
○ Templates also allow you to extract peering data for other uses
○ Email templates with override also (and pdb contacts, etc)
● “show BGP summary”
○ View all sessions at once
● Import from network
○ Pull in existing BGP sessions
https://github.com/fullctl/peerctl

40. BGP Summary

41. PeerCtl - Coming Soon
● Public Peering Portal
○ for third parties to use when requesting peering with your network
● “Where To Peer” Improvements
○ integrating traffic analysis to provide intelligent recommendations
● PeeringDB Integration Improvements
○ enable pushing updates to PeeringDB from PeerCtl
○ enable you to add networks to PeerCtl that are not listed in PeeringDB
● UX Improvements
○ enhanced alerting, along with interface design improvements
https://github.com/fullctl/peerctl

42. AutoPeer
Peering automation API
Zero Click Peering™

43. Joint Effort
● Meta
● FullCtl
● Cloudflare
● AWS
● Google

44. Ghost of Peering Past
Coordinated over email
In person meetings required at first (🍻)
Multiple emails exchanged per request

45. Peering, Now

46. Peering, Future

47. Peering API Goals
● Machine-to-Machine Configuration
● GOAL:
○ Reduce human time spent configuring
peering
○ Improve network latency
● REQUIREMENT:
○ (Security) PeeringDB OIDC support

48. HTTP POST

49. HTTP Reply

50. Peering API, Future
● Proof of concept
● Keep it simple
● Establish definitions

51. Peering Request via Email

52. Peering Request via API

53. Peering Request Queue

54. Check it out!
Announcements Mailing List -
https://lists.20c.com/mailman3/lists/autopeer-announce.lists.20c.com
Developers Mailing List -
https://lists.20c.com/mailman3/lists/autopeer-dev.lists.20c.com/
Waiting on legal
https://github.com/grizz/autopeer

55. Wanna learn more?
Tune in this Friday (21 July) Morning:
https://netboxlabs.com/events/fullctl-netboxlabs-webinar/

56. Ready to Play?
Free account: https://account.fullctl.io (sign in with PeeringDB)
Contact us about a demo:
hello@fullctl.com

57. Not Alone
● PeeringDB
● NetBox (& Nautobot)
● Kentik [tooling/telemetry]
● BGPQ4
● Global Traceroute
● IXP Manager
● Peering Manager
● Rancid
https://openpipes.net

58. What can YOU do?
● Communicate
● Standardate
● Automate

59. What else can we do?
Time for comments thinly veiled as questions...
FullCtl.com