1. Telemetry in Large Data Center
Networks
Sriram Natarajan
Google Networking
on behalf of Google Technical Infrastructure

2. ❯ Rich and deep history of building world-class infrastructure
❯ Common infrastructure across Google apps and Google Cloud Platform
❯ This infrastructure is available to you via GCP
Google Technical Infrastructure
2

3. Over the past 15+ years Google has built one
of the fastest, most capable
network infrastructures on the planet.

4. 2002 2004 2006 2008 2010 2012 2013 2014
Colossus
Spanner
Dremel
Dataflow
Kubernetes
2003 2005 2007 2009 2011
Efficient Server Power
GFS
MapReduce
On-Board UPS
Warehouse Scale
Datacenter
Carbon Neutral
Bigtable
Containerized Datacenter
First Public
PUE Reporting
Containers Software Defined
Networking
Flume
Megastore

5. B4
2006
2008
2010
2012
2014
Google
Global
Cache
BwE
Jupiter
gRPC
FreeDome
Watchtower
QUIC
Andromeda
Google Innovations in Networking
2016

6. Google Network Infrastructure
•GGC: edge presence
•B4: WAN interconnect
•Jupiter: building scale data center network
•Freedome: campus-level interconnect
•Andromeda: virtualize the physical network

7. ❯ Tiny round trip times
❯ Massive multi-path
❯ More plentiful and more uniform bandwidth
❯ Little buffering
❯ Latency, tail latency as important as bandwidth
❯ Homogeneity, protocol modification much easier
❯ Aggregate bandwidth of entire Internet in one building
What’s different about Data Center Networks

8. Five Generations of Data Center Fabrics
Year
Datacenter
Generation
Fabric
Speed
Host
Speed
Aggr
BW
2004 Four-Post CRs 10G 1G 2T
2005 Firehose 1.0 10G 1G 10T
2006 Firehose 1.1 10G 1G 10T
2008 Watchtower 10G 1G 82T
2009 Saturn 10G 10G 207T
2012 Jupiter 10/40G 10G/40G 1.3P
Aggregatetraffic
Traffic generated by servers in our data centers
Jul
‘08
Jun
‘09
May
‘10
Apr
‘11
Mar
‘12
Feb
‘13
Nov
‘14
50x
1x
Dec
‘13
8

9. So, why do we need Telemetry & Analytics in DC Fabrics?
9
5+
10+
20+
# of fabric
architectures in
production
kinds of switches
as part of fabrics in
production
# consumers per
production fabric
}Need Sensors, Software and Systems that help
• perform network design and modeling
• perform topology, configuration and routing verification
• perform smart analytics for root cause isolation

10. Data Center Fabrics are Complex
10
n stage
host stacks:
virtualization
transport
application
switch stack
1
2
3
4
State is distributed across various
elements inside and out of the fabric
Complex interaction between the states
Multiple uncoordinated writers of
state under various control loops
Large: Impossible to humanly observe
state and react to ambiguity or faults
controller software
Challenges are similar for SDN-centric or traditional protocol-based networks

11. 11
Life of a typical Data Center Fabric
OPERATE (Apps and SLAs)
• Define application SLAs
• Measure SLAs and traffic characteristics
• Feed stats to TE, enforcers, PCR schedulers
OPERATE (Apps & SLA)
CONNECT (Routing)
• Design connectivity policies
• Create the intended configuration
• Push configuration to devices
CONNECT (routing, reachability)
BUILD (Initial Topology)
• Design the network topology
• Model the intended topology
• Populate (deploy, wire-up) DC floor
BUILD (init topology)

12. 12
Safety, Correctness and Visibility in a DC Fabric
• Define application SLAs
• Measure SLAs and traffic characteristics
• Feed stats to TE, enforcers, PCR schedulers
OPERATE (Apps & SLA)
• Create connectivity policies & config
• Push configuration to devices
• Verify routing consistency
CONNECT (routing, reachability)
BUILD (Initial Topology)
• Design & model the network topology
• Populate (deploy, wire-up) DC floor
• Verify deployed topology against intent
BUILD (init topology)

13. 13
Compounded by Scale
10,000+
switches per fabric**
3.5 Billion
searches per day
30,000 burst
updates within 1 min**
300 hours
of video uploaded every minute
10 Million +
routing rules per fabric**
0.25Million+
links per fabric**
SLA
& app visibility
BUILD
& topology
CONNECTIVITY
& routing
**Typical numbers seen in large data center fabrics

14. 14
Lots of sensors and data sources
fabric data plane application plane
TCP, UDP / IP stats
virtual switch state
RPC sensors
application sensors
host enforcers
stats | counters
queue depths
switch probes
host probes
routing state
topology models
configuration models
Challenge: To use these effectively to simplify operational complexities in Data Center Fabrics
control & management plane
forwarding state

15. Systems to Enable Safety, Correctness & Visibility
15
Topology Verification
To continually verify that what is
deployed is what was intended
01.
Route Consistency
To verify routing state
consistency between
controllers and data plane
02.
Traffic Characteristics
**
To verify host-granular
reachability and measure traffic
characteristics
03.
SLA
& app visibility
BUILD
& topology
CONNECTIVITY
& routing
**The analytics system described here focuses primarily on the host-level reachability
and packet-loss characterization of app2app communication

16. 01. Topology Verification at Scale
16
How do we verify that what has been
deployed and wired up matches
intended topology
in a 10,000+ node / 250,000+ links fabric

17. n stage
17
1
2
3
4
Generate probe traffic from Hosts
This is not switched like production traffic
Don’t rely on just destination-based routing
Source Route: Ensures targeted full coverage
Analytics App: Takes all this data generated and
localize connectivity problems
Read in the intended model of the fabric
Generate a topology to verify against
01. Topology Verification at Scale
Simultaneous Detection of Topological faults within a minute of occurrence

18. 18
How do we verify consistency between
configured policy, routing state and
forwarding state
in a fabric with 10M+ rules and detect & isolate
loops & black holes quickly
02. Routing Consistency Verification at Scale

19. 19
1
2
3
4
Map: Create a network slice by destination subnet by
picking rules relevant to that subnet against a shard
of the full set
Reduce: Construct a directed forwarding graph and
verify properties such as loop freedom and reachability
At every subsequent routing update, only
analyze incremental updates
Generate snapshot of the routing state by
recording route change events
SDN Controller
Route Snapshot
}
Rule Shard 1 Rule Shard n…
M M M…
R R R
M
R …
Report Subnet 1 Report Subnet m…
02. Libra: Routing Consistency Verification at Scale
Detection of Loops & Holes within 1ms of occurrence in a 10K Node Network

20. 20
How do we measure host-level
reachability and app-level traffic
characteristics comprehensively
across all host-pairs and all traffic classes with
varying SLA & TE needs
03. App-level SLA measurements at Scale

21. 21
1
2
3
4
Exercise src2dest and dest2src paths through
entire host-fabric-host SW stack
Structured rotation of probes across all hosts &
traffic queues for full coverage
Correlate probe loss & latency in fwd & rev directions
across a number of probe sets to localize issues
Randomly pick a subset of hosts and generate
probes to and from the hosts.
03. App-level SLA measurements at Scale

22. 22
That IS a lot of data...
Now that your network infrastructure is richly
instrumented...how do you extract this information?
We use a RPC framework optimized for encrypted,
streaming, and multiplexed connections.
...and so can you.
http://grpc.io

23. 23
Network equipment
● We’ve discussed a lot of external software telemetry sources...what about data from
network devices themselves?
● Today, SNMP is often the de facto network telemetry protocol. Time to upgrade!
○ legacy implementations -- designed for limited processing and bandwidth
○ expensive discoverability -- re-walk MIBs to discover new elements
○ no capability advertisement -- test OIDs to determine support
○ rigid structure -- limited extensibility to add new data
○ proprietary data -- require vendor-specific mappings and multiple requests to
reassemble data
○ protocol stagnation -- no absorption of current data modeling and transmission
techniques

24. 24
Next generation network telemetry:
● network elements stream data to collectors (push model)
● data populated based on vendor-neutral models whenever possible
● utilize a publish/subscribe API to select desired data
● scale for next 10 years of density growth with high data freshness
○ other protocols distribute load to hardware, so should telemetry
● utilize modern transport mechanisms with active development communities
○ e.g., gRPC (HTTP/2), Thrift, protobuf over UDP
www.openconfig.net

25. 25
Building Safe, Correct and Transparent DC Fabrics
Detect host reachability issues and SLA violations
within minutes of these events
OPERATE (Apps & SLA)
Verify routing consistency and detect black
holes, loops within a minute of a routing or
configuration change
CONNECT (routing, reachability)
Explore Topology Design Space & Synthesize
Verify deployed topology against intent within
minutes of turn-up
BUILD (init topology)

26. ❯ Think outside the BOX
❯ Sensors and DATA ANALYTICS are key for building data center networks
❯ HUMANS are (almost) useless
Key Take-Aways
26

27. Data Center Networking @ Google
• An SDN centric DevOps organization
• Responsible for the architecture, design, and build of Google’s data center networks
• Responsible for development of software & systems for modeling, automation, telemetry, analytics
Network Engineers Network Systems Engineers Technical Program ManagersSREArchitects
Join Data Center Networking @ Google

28. Sriram Natarajan
snr@google.com
References THANK YOU
1. “Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s
Datacenter Network,” SIGCOMM 2015.
2. “Network Detective: Finding network blackholes”, Israel Networking Day 2014.
3. “Libra: Divide and Conquer to Verify Forwarding Tables in Huge Networks”, NSDI 2014.
4. “B4: Experience With a Globally-Deployed Software Defined WAN,” SIGCOMM 2013.
5. “Bandwidth Enforcer: Flexible, Hierarchical Bandwidth Allocation for WAN Distributed
Computing,” SIGCOMM 2015.