Next Gen Monitoring with INT

Copyright © Arista 2018. All rights reserved.
Next Gen Monitoring with INT
(In Band Network Telemetry)
Ismail Ali (ismail@arista.com)
MyNOG 2019

Agenda
• Network Monitoring Evolution
• Motivation for New INT Model
• What is INT and How it Works
• INT Use Cases
2

Network Monitoring
• Why we need to monitor a network?
• What we need to monitor for a network?
• When and How we monitor a network?
• What and when we can get payback for network monitoring?
3

Today’s Network Monitoring
4
Expensive and inefficient No fine granularity visibility NO visibility = no control

Network Monitoring Evolution -- Telemetry
• Telemetry: is an automated communications process by which measurements
and other data are collected at remote or inaccessible points and transmitted to
receiving equipment for monitoring
• Need more finer granularity network monitoring data
• Mostly vendor-specific chipset/system based
• Involved some open projects/standards
5

Network Monitoring Evolution -- INT
• INT: Inband Network Telemetry (INT) is a framework that is designed to
monitor, collect, and report (Telemetry) flows and network states (Network),
by the data plane, without requiring intervention or work by the control plane
(Inband),
• Through data plane, metadata based, more vendor system depending
• Network-wide, instead of original platform centralized
6

Motivations for Next-Gen Monitoring
• Realtime network monitoring
- Line rate monitoring
• Canonical data mode
- OpenConf/YANG
- Easy to multi-vendor deployment
• Assurance for services/quality
- SLA and proof
- Per packet metadata
• New services deployment
- NFV
- micro-service
• Next-Gen Operation
- Fast fault detecting
- Real time path tracing
- Fast responding
7

INT First Spec Draft in 2015
8
https://p4.org/assets/INT-current-spec.pdf

And the latest is…
9
https://github.com/p4lang/p4-applications/blob/master/docs/INT_v1_0.pdf

INT : In-band Network Telemetry
• Mechanism for collecting network state in the data plane
- As close to real-time as possible
- At current and future line rates
- With a framework that can adapt over time
• Examples of network state
- Switch ID, Ingress Port ID, Egress Port ID
- Egress Link Utilization
- Hop Latency
- Egress Queue Occupancy
- Egress Queue Congestion Status
- ….
10
Naga Katta, Mukesh Hira, Changhoon Kim, Anirudh Sivaraman, and Jennifer Rexford. HULA: Scalable Load Balancing Using
Programmable Data Planes. In SOSR 2016.

How INT works
• Network element inserting it’s state (referred to as INT Metadata) inline into
packets in the data path encapsulated within an INT Header.
• Each network element in the packet path that supports INT, inserts its state
onto the packet.
• At the tail end these are stripped off and sent to collectors where the
metadata can be analyzed to provide much deeper information regarding
the network element states at the time of packet transit.
11

INT Typical Deployment
12

INT Packet
• INT information/data is carried inline in data plane frames
- Possible for every packet in the network
• Two components for the INT information
- INT Header: A packet header that carries INT information
≫ Identifies the INT frame and also carries information for transit nodes. One of the primary fields in the header is the
‘INT-vector’ which is typically a bit-map of data-types that each transit node collects and inserts in the frame. Note
that draft-kumar, for example, uses template ids to map to a particular set of metadata as opposed to explicitly
specifying the metadata set via a bit vector.
≫ Note that the draft-kumar and draft-kumar-v2 use the term ‘IFA’ (Inband Flow Analyzer) to refer to ‘INT’. For example
the INT Header is referred to as the IFA Header in the drafts.
- INT Metadata: Information that an INT Source inserts into the INT Header
≫ can be viewed as the collection of data-plane state that is stamped by a node in the frame. Typically this can be
viewed as a variable-length array of node metadata, where each element of the array represents the metadata for a
particular node. The Metadata may follow immediately after the header with each transit node inserting its metadata
at the head of the array immediately after the header.
≫ Note that certain implementations mention that INT Metadata (or a part of it) may also be carried in the tail of the
frame, but this is typically done to get around some hardware limitations.
13

INT Header
• Two types of INT header
- Type 1: hop-by-hop type
≫ Intermediate devices must process this type of INT header
- Type 2: destination type
≫ Intermediate devices must ignore this type of head and must be only consumed by INT Sink
- Yet another type:
≫ When both INT header types are present, the hop-by-hop type must be precede the destination type
header
14

INT Header Over ANY Encapsulation
• Basically, and INT Header can be inserted as an option or payload of any
encapsulation type
- INT over VXLAN (as VXLAN payload, per GPE extension)
- INT over Geneve (as Geneve option)
- INT over NSH (as NSH payload)
- INT over TCP (as payload)
- INT over UDP (as payload)
- INT over GRE (as a shim between GRE header and encapsulated payload)
• All devices along the way need to agree with it
15

INT Hop-by-Hop Metadata Header Format
16

INT Header: Potential Locations for Different Encapsulation
17

An INT Header and Metadata Example for a Simple Topology
18
• Host1 sends a TCP packet to host2.
• The ToR switch of host1 (Switch1) acts
as the INT source.
• Switch1 adds INT headers and its own
metadata in the packet.
• Switch2 prepends its metadata.
• Finally, the ToR switch of host2
(Switch3) acts as the INT sink and
removes INT headers before forwarding
the packet to host2.

Metadata List for Present and Future (to be added)
• Switch Level
- Switch id
- Control plane state version number
• Ingress
- Ingress port identifier
- Ingress timestamp
- Ingress port RX pkt count
- Ingress port RX byte count
- Ingress port RX drop count
- Ingress port RX utilization
19
• Egress
- Egress port identifier
- Egress timestamp
- Egress port TX pkt count
- Egress port TX byte count
- Egress port TX drop count
- Egress port TX utilization
• Buffer Information
- Queue id
- Instantaneous queue length
- Average queue length
- Queue drop count
• Miscellaneous
- Checksum Complement

INT Flow Event – Watchlist & Event Detection
20
INT Endpoint
(source)
Flow watchlist
payload
header
Switch1
INT Metadata
payload
header
Switch2
INT Metadata
payload
header
Switch1
INT Metadata
INT Endpoint
(sink)
Event detection
payload
header
Switch2
INT Metadata
Switch3
Local report
report
header
Switch1
INT Metadata
INT Endpoint
(source)
Flow watchlist
Host1 Host2Switch1 Switch2 Switch3
Monitor
Collector

Some INT Report Types
• Local flow reports — Generated from flow events. Sent from the source or
sink for host-to-host data flows matching the watchlist
• Drop reports — Generated from drop events. Sent for certain supported
drops. Every INT-enabled switch sends these reports to the monitor-
collector
• Queue Congestion reports — Generated from queue-related events. Sent for
packets exceeding the queue depth or latency. Every INT-enabled switch
sends these reports to the monitor-collector
• INT reports — Sent by the sink. When INT-encapsulated data packets are
received on the sink fabric port, two reports are generated by the sink:
- Local report for traffic arriving on fabric port
- INT report for data received from the source
21

INT Report Example – Drop Report
22
payload
header
payload
header
INT Endpoint
Watchlist: Event detection
report
header
Switch1
Drop information
Host1 Host2Switch1 Switch2 Switch3
Monitor
Collector
report
header
Switch2
Drop information
report
header
Switch3
Drop information
payload
header
payload
header
INT Endpoint
INT Endpoint

Copyright © Arista 2018. All rights reserved.23
A simple INT use case:
Measuring and reporting end-to-end latency between virtual switches

How packet level telemetry helps
• Inflated latencies and congestion analysis
• Network topology and packet traversals
• Timeliness and flexibility for exceptions
• Doorway to machine learning
24

Questions INT tried to Address
25
http://www.opencompute.org/assets/Uploads/INT-In-Band-Network-Telemetry-A-Powerful-
Analytics-Framework-for-your-Data-Center-OCP-Final3.pdf

Network Path and Forwarding Rule
26

Network Latency
27

Congestion Cause
28

How INT Works
29
Remote
monitoring
engine

Extending INT telemetry beyond physical switches
30
Trident 3
Jericho 2

Three INT Models
• Out of Band Probes
- Similar to ping/traceroute in the sense that these are admin/initiated
≫ except that unlike ping/traceroute, these are handled in data plane.
- The transit nodes add INT metadata in data plane.
- Typically these probes are initiated and terminated on the Host side and the network switches/nodes perform the ‘transit’ function
• Inline INT Model
- INT information is carried inband/inline in existing data flows.
- By including the INT information (header + metadata) at the Initiating/Encapsulating node and by removing the INT headers and
metadata at the Terminating/Decapsulating node.
- Each transit node adds its own INT metadata
≫ Initiating and Terminating nodes may also perform the ‘transit’ role
• INT over mirror
- The INT Initiating node applies a selection process to select candidate frames for applying INT based on interface, ACL etc.
- The candidate frames are then subject to sampling at a configured rate to arrive at the final decision to apply further INT processing.
- The INT Initiating/Encapsulating node then mirrors or generates a copy of the selected frames.
- The original packet goes through unchanged, whereas the copy is augmented with INT data.
- The Initiating node inserts an INT header in the copy and each transit hop adds its metadata carrying INT information.
- The terminating node collects the INT information, drops the copy and forwards the summarized data to collector(s).
31

INT Deployment model
• INT Initiation at TOR
- The TOR switches perform the INT
Initiation and Termination functions.
- The servers are INT Agnostic and the
Spine switches perform INT transit
• INT Initiation at Server/NIC
- INT Initiation/Termination happens at
the NIC
- TOR and Spine switches are purely
transit from INT perspective.
32

References
• [draft-kumar-ifa-00]: Mechanism to sample and mirror data plane packets and
carry INT info in the mirrored packets which would be dropped at the terminating
node. This mechanism is referred to as IFA 1.0 in the draft. This is superseded
by draft-kumar-v2.
• [draft-kumar-ippm-ifa-01]: Update to draft-kumar that specifies usage of a new
‘experimental IP protocol type’ to identify INT packets. This is the current version
proposed in IETF. It is referred to as IFA 2.0 in the draft.
• [draft-ietf-ippm-ioam-data-03]: Mechanism to carry telemetry info in-situ i.e
along with data plane packets by inserting INT headers and metadata into
packets.
• [https://p4.org/assets/INT-current-spec.pdf]: specified mechanism to carry INT
info in normal data plane packets or special probe packets
• [draft-lapukhov-dataplane-probe-01]: Active data plane probes. These don't
carry application traffic i.e separate INT packets - sort of like ping, traceroute
except they would be handled in the data plane in the transiting nodes.
33

Next Gen Monitoring with INT

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Next Gen Monitoring with INT

Similar to Next Gen Monitoring with INT (20)

More from MyNOG

More from MyNOG (20)

Recently uploaded

Recently uploaded (20)

Next Gen Monitoring with INT