Speaker: Jason Davis, Cisco Systems Abstract: Since 1987 CiscoLive has been the premiere event for networking education, collaboration and fun. Las Vegas 2023 brought 19,600 attendees to the Mandalay Bay Conference Center and adjacent properties. To support that we have a few days to install and provision 2300 wireless access points, over 650 switches and a mobile data center. This session is a behind the scenes look at the automation and network monitoring done to support such a large event in such a short time-period. You'll hear about open source solutions like InfluxDB and Grafana. You'll also hear about the importance of standardized management interfaces like gRPC and NETCONF/YANG.

1. Jason Davis
Distinguished Engineer (Cisco DevNet)
November 14, 2023
Network Automation Forum - AutoCon0
Automating a World-Class Technology
Conference;
Behind the Scenes of CiscoLive

2. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
2
Since 1989 CiscoLive (Networkers)
The network industries’ premiere event
for education, inspiration and fun

3. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
3
Held in multiple theatres each year;
US and Europe events are large
requiring our own event IT support staff
Fira de Barcelona
Messe Berlin
San Diego Convention Center
Las Vegas

4. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
4
By The Numbers
19,600 attendees
with ~47K mobile devices
Over 2,300 wireless access
points
650 network switches
mobile containerized data center
Servers
Storage
Firewalls
Monitoring/
Provisioning
Apps
Triple 100 Gigabit/second
Internet links

5. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
5
Build-Out: Starting from Scratch

6. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
6

7. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
7
“WHY HAVEN’T WE SEEN FULL
ADOPTION OF NETWORK
AUTOMATION, YET?”
The Grundemann Conundrum, mann…

8. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
8
Observability - as a Car analogy…

9. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
9
It’s all about the
Apey Eyes
APIs…
get it!?
Design
considering
varied data
sources
gRPC
NETCONF
RESTCONF
REST
SNMP
gNMI
CLI

10. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
10
“WHY HAVEN’T WE SEEN FULL
ADOPTION OF NETWORK
AUTOMATION, YET?”
The lack of awareness and understanding about available
telemetry, instrumentation
The diversity and varying implementation of management
protocols

11. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
11
•Leverage Commercial Management
products
DNAC, Prime Network Registrar,
Cisco Telemetry Broker,
Umbrella, Meraki Dashboard,
WLC, ThousandEyes,
etc.
Monitoring Strategy 1 of 3

12. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
12
•Use Open-source solutions to supplement
needs specific to our use-case
Monitoring Strategy 2 of 3

13. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
13
•Do purposeful development to fit our
unique requirements
Monitoring Strategy 3 of 3

14. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
14

15. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
15
320 Gig
connection
with Internet
drives what
volume?

16. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
16
Design for
an
extended
audience…

17. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
17
Think of
alternative
visualizations

18. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
18
What is the
Automation
ROI?
Measure it!
What are your ROI Metrics?
Do You Have an Automation ROI Tracker?

19. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
19
We found
a classic
memory
leak!

20. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
20

21. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
21

22. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
22
Automation of Observability/Monitoring
• First pass development of SSH2Influx was linear
Capturing 3 CLI ‘show commands’ from 200 network switches
took almost 15 min to complete a cycle
I/O
Network, Disk
Time
Processing

23. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
23
Automation of Observability/Monitoring
• Rewrite using cooperative multitasking asyncSSH and ThreadPoolExecutor
methods
8 threads got us down to 90 seconds
I/O
Network, Disk
Processing
Time

24. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
24
#CiscoLive
Channel Utilization Heatmap
Breakfast
This is the AP in the
dedicated “tablet
charging room”
First sessions of
the day
Attendees moving
between session
rooms
Channel
Utilization
Receive
Utilization
Transmit
Utilization
Client
Count
Time
Now
2h ago

25. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
25
Automation Requires Orchestration
ASR1009
(2)
Cisco Prime
Network Registrar
(x6)
DNACenter
Orchestrator
REST
API
REST API
SSH, REST
SNMP
Internet
3x100 GB
fping, NETCONF RPC
Service Catalog
TFTP, gRPC ST
gitbucket
R
E
S
T
A
P
I
S
C
P
SSH, RESTCONF, NETCONF RPC
Webex
R
E
S
T
A
P
I
InfluxDB
Grafana
SSH & REST API
Smart
Sheet
Catalyst
9800
SSH & RESTCONF API
Meraki
R
E
S
T
A
P
I
TFTP, gRPC ST
REST API
REST API

26. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
26
n8n.io
Low to no code
Graphical workflow development

27. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
27
Reuse our work!
https://github.com/jasoncdavis/CiscoLiveNOC23US
CiscoLive NOC data collection
scripts and Grafana dashboard
definitions

28. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
28
Reuse our work!
https://github.com/jasoncdavis/SSH2Influx
SSH2Influx
Connect to any device that
supports SSH,
extract command outputs,
pattern match and tag to
InfluxDB

29. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
29
Reuse our work!
https://github.com/jasoncdavis/devnetdashboards-
convergedavailabilitymonitor
DevNet Dashboards Converged
Availability Monitor

30. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
30
Automation Learnings
• Accurate inventory is key - all other works flow from it
• Standardize on your naming conventions - devices, scripts, inventory
• Common vocabulary in a diverse team improves efficiency
• Zero defects is NOT the standard - expect failures; learn and improve

31. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
31
Tips for Success
• Start with your Intent
• Identify your audience(s) - What visualization
will resonate with them?
• Identify capabilities and shortcomings in tools
at hand (commercial and open source)
• Research Instrumentation/Telemetry Options
• Use gRPC Streaming Telemetry where possible
• Fall-back to other forms (NETCONF RPC, SNMP)
where necessary
• Use InfluxData’s InfluxDB when dealing with
time-series data; don’t try to make a
relational database be time-series
• Use Grafana’s resources for learning and
SHARE your dashboards & experiences with
others
• Investigate Cross-domain, Technology-
Agnostic Orchestration options
[low-to-no code]
SecureX Orchestrator, n8n.io
• DevOps for the Win!
https://developer.cisco.com

32. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
32
https://developer.cisco.com
Resources for…
Enterprise
AppDevs
DevOps/SRE
Infra
Engineers
API Guides
Learning Labs
Reservable Resources
Code Exchange
Developer Forums
Training & Certification
Tools

33. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
33
Raspberry Pi NTP Servers
GPS, PPS and
PoE Enabled
Stratum-1
2 usec jitter
$150 parts
Version 2
No new ‘friends’
made at TSA
J

35. © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
35
Thank You!