• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
T Series Core Router Architecture Review (Whitepaper)
 

T Series Core Router Architecture Review (Whitepaper)

on

  • 1,132 views

Juniper Networks® T Series Core Routers have been in production since 2002, with the introduction of the Juniper Networks T640 Core Router. Since that time, T Series routers have evolved to maintain ...

Juniper Networks® T Series Core Routers have been in production since 2002, with the introduction of the Juniper Networks T640 Core Router. Since that time, T Series routers have evolved to maintain an unequivocal industry lead in capacity (slot, chassis, and system) and operational efficiencies in power and usability. Maintaining this standard has in part been possible due to design decisions made with the very first T Series system. The T Series demonstrates how Juniper has evolved its router architecture to achieve substantial technology breakthroughs in packet forwarding performance, bandwidth density, IP service delivery, and system reliability. At the same time, the integrity of the original design has made these breakthroughs possible. Not only do T Series platforms deliver industry-leading scalability, they do so while maintaining feature and software continuity across all routing platforms. Whether deploying a single-chassis or multichassis system, service providers can be assured that the T Series satisfies all networking requirements.

Statistics

Views

Total Views
1,132
Views on SlideShare
1,132
Embed Views
0

Actions

Likes
0
Downloads
49
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    T Series Core Router Architecture Review (Whitepaper) T Series Core Router Architecture Review (Whitepaper) Document Transcript

    • White PaperT Series Core RouterArchitecture OverviewHigh-End Architecture for Packet Forwardingand SwitchingCopyright © 2012, Juniper Networks, Inc. 1
    • White Paper - T Series Core Router Architecture Overview Table of Contents Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Nine Years of Industry-Leading Core Routing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Forwarding Plane Design Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Architectural Framework and Components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 PIC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 FPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Switch Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Routing Engine. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 T Series Forwarding Path Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Data Flow Through the 3D FPC on the T4000 Router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Data Flow Through the Enhanced Scaling (ES) FPCs on T Series Routers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 T Series Switch Fabric Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Switch Fabric Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Multichassis Switch Fabric (TX Matrix and TX Matrix Plus) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 CLOS Topology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Juniper Networks Implementation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Multichassis System Fabric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 About Juniper Networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 List of Figures Figure 1. T Series scaling by slot and chassis (including the JCS1200). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Figure 2. Scalable performance in multiple dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Figure 3. T Series routing platform architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Figure 4. T Series router components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Figure 5. Data flow through the T4000 router. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Figure 6. PFE and switch fabric using the T Series chipset (“to fabric” direction). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Figure 7. Packet flow from fabric to egress PFE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Figure 5. Five switch fabric planes for the T Series. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Figure 6. Multicast and unicast each utilizing fabric queues in T Series routers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Figure 7. Three-stage CLOS topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Figure 8. T Series multichassis switch fabric planes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Figure 9. Multichassis system high-level overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Figure 10. Switch-card chassis to line-card chassis interconnections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 Copyright © 2012, Juniper Networks, Inc.
    • White Paper - T Series Core Router Architecture Overview Executive Summary Juniper Networks® T Series Core Routers have been in production since 2002, with the introduction of the Juniper Networks T640 Core Router. Since that time, T Series routers have evolved to maintain an unequivocal industry lead in capacity (slot, chassis, and system) and operational efficiencies in power and usability. Maintaining this standard has in part been possible due to design decisions made with the very first T Series system. Nine Years of Industry-Leading Core Routing In April 2002, Juniper Networks began shipping the first T Series routing platform: the T640 Core Router. The T640 is a carrier-class, multichassis-capable core routing platform that supports high-density 10 Gbps (OC-192c/STM-64 and 10 Gbps Gigabit Ethernet) to 40 Gbps (OC-768c/STM-256 and 40 Gbps Gigabit Ethernet) interfaces. In July 2002, Juniper Networks began shipping the second T Series platform: the Juniper Networks T320 Core Router. The T320 is a carrier-class, single-chassis core router that has a smaller form factor than the T640 and a lower entry price point. In 2004, Juniper offered the first multichassis core routing system with the Juniper Networks TX Matrix, supporting 3.2 Tbps in a four-chassis system. Then in 2007, Juniper announced and released the industry’s first 100 Gbps/slot system in the Juniper Networks T1600 Core Router, a multichassis-capable routing platform that was designed—taking advantage of the original switch plane architecture of the T640 —to be upgradeable in service from the T640. In 2009, Juniper introduced the Juniper Networks TX Matrix Plus, a central switching and routing element that connects up to 4 T1600 routing chassis into a single routing entity: a 6.4 Tbps system. In 2011, Juniper introduced the 350 Gbps/slot-capable system, Juniper Networks T4000 Core Router, which has significantly extended the T Series system capacity, and allowed customers’ investment in the T Series to be protected by upgrading existing T640 and T1600 systems to the T4000. All T Series platforms use Juniper Networks Junos® operating system and T Series ASICs to provide the ease of use, performance, reliability, and feature richness that service providers have come to expect from all Juniper Networks products. The T Series Core Routers—T320, T640, T1600, T4000, TX Matrix, and TX Matrix Plus—provide the ingredients for high-end and core networks of the future, especially when controlled by the Juniper Networks JCS1200 Control System. Figure 1 illustrates the industry-leading scaling characteristics of the T Series on the forwarding and control planes. JCS1200 Single Chassis T640 TX Matrix Single Chassis T1600 TX Matrix Plus Single Chassis T4000 Multichassis Ports Ports Ports Ports Multichassis Ports 320 10 Gbps 40 10 Gbps 160 10 Gbps 80 10 Gbps 208 10 Gbps 32 64 40 Gbps 8 40 Gbps 32 40 Gbps 16 40 Gbps 16 40 Gbps 32 100 Gbps 8 100 Gbps 16 100 Gbps 40 Gbps/slot 2002 2004 100 Gbps/slot 2007 2009 350 Gbps/slot 2011 Figure 1. T Series scaling by slot and chassis (including the JCS1200) This paper provides a technical introduction to the architecture of T Series Core Routers, both single-chassis and multichassis systems. It describes the design objectives, system architecture, packet forwarding architecture, single- chassis switch fabric architecture, and multichassis switch fabric architecture. For an explanation of the core virtualization capabilities available with the integration of the JCS1200, see the References section in Appendix A, particularly the paper entitled Virtualization in the Core of the Network at www. juniper.net/us/en/local/pdf/whitepapers/2000299-en.pdf.Copyright © 2012, Juniper Networks, Inc. 3
    • White Paper - T Series Core Router Architecture Overview Forwarding Plane Design Objectives Juniper’s architectural philosophy is based on the premise that multiservice networks can be viewed as three- dimensional systems containing forwarding, control, and service dimensions: • Forwarding (“how you move the bits”) • Control (“how you direct the bits”) • Software/service (“how you monetize the bits”) For a network to be a converged, scalable, packet core infrastructure, it must scale in all these dimensions (Figure 2): Control: JCS1200 P Forwarding: T1600 D S P : ce vi er S Figure 2. Scalable performance in multiple dimensions In the forwarding plane, traffic growth is the key driver of the core router market. As the global economy becomes increasingly networked and dependent upon the communications infrastructure, traffic rates continue to balloon— growing 70 to 80 percent a year by most estimates—and high-density core routing remains critical. Furthermore, the importance of the control plane cannot be overlooked. A router’s control plane must scale to accommodate ever-growing routing and forwarding tables, service tunnels, virtual networks, and other information related to network configuration and management. Finally, the importance of the service plane is brought to bear when considering the requirements of an increasingly disparate and global marketplace. These changing market dynamics create pressure for greater network innovation and a much deeper integration between applications and the network.1. T Series routing platforms are designed with this philosophy in mind at all times—the main focus of this paper is on the forwarding plane of the router. The T Series routing platforms were developed to support eight key design objectives: • Packet forwarding performance • Bandwidth density • IP service delivery • Multichassis capability • High availability (HA) • Single software image • Security • Power efficiency Juniper Networks leads the industry in all of these categories, as the white papers in the References section illustrate.4 Copyright © 2012, Juniper Networks, Inc.
    • White Paper - T Series Core Router Architecture Overview Architectural Framework and Components Figure 3 illustrates the high-level architecture of a T Series routing platform. A T Series platform uses a distributed architecture with packet buffering at the ingress Packet Forwarding Engine (PFE) before the switch fabric, as well as packet buffering at the egress PFE before the output port. As a packet enters a T Series platform from the network, the ingress PFE segments the packet into cells, the cells are written to ingress memory, a route lookup is performed, and the cells representing the packet are read from ingress memory and sent across the switch fabric to the egress PFE. When the cells arrive at the egress PFE, they are written to the second memory; an egress lookup is performed; the cells representing the packet are read out of the second memory, reassembled into a packet, and transmitted on the output interface to the network. Packet Processing/ Packet Processing/ Buffer Buffer Packet Processing/ Switch Packet Processing/ Buffer Buffer Packet Processing/ Packet Processing/ Buffer Buffer Figure 3. T Series routing platform architecture Drilling more deeply into the components, a T Series system consists of four major components: PICs, Flexible PIC Concentrators (FPCs), the switch fabric, and one or more Routing Engines (Figure 4). Routing Engine Routing Engine Switching FPC FPC FPC FPC Planes PIC PIC PIC PIC Figure 4. T Series router components These components are discussed in more detail in the following sections. PIC The PICs connect a T Series platform to the network and perform both physical and link-layer packet processing. They perform all of the functions that are required for the routing platform to receive packets from the network and transmit packets to the network. Many PICs, such as the Juniper IQ PICs, also perform packet processing. FPC Each FPC can contain one or more PFEs. For instance, a T4000 240 Gbps slot supports two 120 Gbps PFEs. Logically, each PFE can be thought of as a highly integrated packet-processing engine using custom ASICs developed by Juniper Networks. These ASICs enable the router to achieve data forwarding rates that match fiber optic capacity. Such high forwarding rates are achieved by distributing packet-processing tasks across this set of highly integrated ASICs. When a packet arrives from the network, the ingress PFE extracts the packet header, performs a routing table lookup and any packet filtering operations, and determines the egress PFE connected to the egress PIC. The ingress PFE forwards the packet across the switch fabric to the egress PFE. The egress PFE performs a second routing table lookup to determine the output PIC, and it manages any egress class-of-service (CoS) and quality-of-service (QoS) specifications. Finally, the packet is forwarded to the network.Copyright © 2012, Juniper Networks, Inc. 5
    • White Paper - T Series Core Router Architecture Overview Switch Fabric The switch fabric provides connectivity between the PFEs. In a single-chassis system, the switch fabric provides connectivity among all of the PFEs residing in the same chassis. In a multichassis system, the switch fabric provides connectivity among all of the PFEs in the different chassis of the routing node cluster. In a single-chassis or a multichassis system, each PFE is considered to be logically contiguous to every other PFE connected to the switch fabric. Routing Engine The Routing Engine executes the Junos OS and creates the routing tables that are downloaded into the lookup ASICs of each PFE. An internal Ethernet connects the Routing Engine to the other subsystems of a T Series platform. Each subsystem includes one or more embedded microprocessors for controlling and monitoring the custom ASICs, and these microprocessors are also connected to the internal Ethernet. T Series Forwarding Path Architecture The function of the PFE can be understood by following the flow of a packet through the router: first into a PIC, then through the switching fabric, and finally out another PIC for transmission on a network link. Generally, the data flows through the PFE as follows: • Packets enter the router through incoming PIC interfaces, which contain controllers that perform media-specific processing (and optionally intelligent packet processing). • PICs pass the packets to the FPCs, where they are divided into cells and are distributed to the router’s buffer memory. • The PFE performs route lookups, forwards the cells over the switch fabric to the destination PFE, reads the cells from buffer memory, reassembles the cells into packets, and sends them to the destination port on the outgoing PIC. • The PIC performs encapsulation and other media-specific processing, and it sends the packets out into the network. The use of highly integrated ASICs is critical to the delivery of industry-leading forwarding performance and packet processing. The T Series chipset is specifically designed and developed to comprise the state of the art in carrier-class routers. Data Flow Through the 3D FPC on the T4000 Router 3D refers to the newer Type 5 FPC introduced along with the T4000 system. To ensure the efficient movement of data through the 3D FPC on the T4000 router, the router is designed so that ASICs on the hardware components handle the forwarding of data. Data flows through the 3D FPC on the T4000 router in the following sequence (see Figure 5): Lookup and Packet- Midplane Processing ASIC P LAN/WAN Switch Packets Switch I Interface, Interface Fabric In Buffering ASIC ASIC C P LAN/WAN Switch Packets Switch I Interface, Interface Fabric Out Buffering ASIC ASIC C Lookup and Packet- Processing ASIC Figure 5. Data flow through the T4000 router6 Copyright © 2012, Juniper Networks, Inc.
    • White Paper - T Series Core Router Architecture Overview 1. Packets arrive at an incoming PIC interface. 2. The PIC passes the packets to the FPC, where the interface ASIC does pre-classification and sends the packet header to the Packet-Processing ASIC that performs Layer 2 and Layer 3 lookup. The Packet-Processing ASIC is also capable of doing Layer 4 through Layer 7 packet processing. 3. The Interface ASIC receives the modified header from the Packet-Processing ASIC and updates the packet and divides it into 64-byte cells. 4. The Interface ASIC sends these 64-byte cells to the Switch Fabric via the Switch Interface ASIC-facing Switch Fabric, unless the destination is on the same Packet Forwarding Engine. In this case, the Interface ASIC sends packets to the outgoing port without passing them through the Switch Fabric. 5. The Interface ASIC sends bandwidth requests through the Switch Fabric to the destination port. 6. The destination Interface ASIC sends bandwidth grants through the Switch Fabric to the originating Interface ASIC. 7. Upon receipt of each bandwidth grant, the originating Interface ASIC sends a cell through the Switch Fabric to the destination Packet Forwarding Engine. 8. The destination Interface ASIC receives cells from the Switch Fabric, reorders the data received as cells and reassembles into packets, and passes the header to the Packet-Processing ASIC. 9. The Packet -Processing ASIC performs the route lookup, adds Layer 2 encapsulation, and sends back the header to the Interface ASIC. 10. The Interface ASIC appends the modified header received from the Packet-Processing ASIC and sends the packets to the outgoing PIC interface. 11. The outgoing PIC sends the packets out into the network. Data Flow Through the Enhanced Scaling (ES) FPCs on T Series Routers This chipset includes the following ASICS: • L2/L3 Packet-Processing ASIC (L2/3) • Switch Interface ASIC • T-Series Lookup Processor ASIC • Queuing and Memory Interface ASIC (Q&M) The T Series ASICs leverage the conceptual framework, many of the building blocks, and Juniper Networks extensive operational experience—gained from the Juniper Networks M Series Multiservice Edge Routers chipset. Figure 6 demonstrates data flow through a T Series routing node by illustrating how the T Series chipset is arranged to implement a single instance of a PFE. PIC FPC 5 FPC SIB l2/L3 T Series Layer 1 WAN ASIC Packet Lookup Processing 3 Processor 6 Fabric Switch Switch 1 2 Interface 4 Interface ASIC ASIC l2/L3 Queuing Layer 1 WAN ASIC Packet and Memory Processing Interface ASIC Figure 6. PFE and switch fabric using the T Series chipset (“to fabric” direction)Copyright © 2012, Juniper Networks, Inc. 7
    • White Paper - T Series Core Router Architecture Overview In this example, data flows in the following sequence: 1. Packets enter through an incoming PIC, which contains the Layer 1 interface chips, and are passed to the PFE on the originating FPC. The PIC is connected to the PFE on the FPC via a high speed link (HSL). 2. The Layer 2/Layer 3 packet-processing ASIC parses the packets and divides them into cells. In addition, a behavior aggregate (BA) classifier determines the forwarding treatment for each packet. 3. The network-facing Switch Interface ASIC places the lookup key in a notification and passes it to the T Series Lookup Processor. 4. The Switch Interface ASIC also passes the data cells to the Queuing and Memory Interface ASICs for buffering on the FPC. 5. The T Series Lookup Processor performs the route lookup and forwards the notification to the Queuing and Memory Interface ASIC. In addition—if configured—filtering, policing, sampling, and multifield classification are performed at this time. 6. The Queuing and Memory Interface ASIC sends the notification to the switch-fabric-facing Switch Interface ASIC, which issues read requests to the Queuing and Memory Interface ASIC to begin reading data cells out of memory. The destination FPC is shown in the following figure. SIB FPC 10 FPC 11 T Series 8 l2/L3 Layer 1 Lookup Packet WAN Fabric Processor Processing ASIC Switch Switch Interface 7 9 Interface ASIC ASIC Queuing l2/L3 Layer 1 and Memory Packet ASIC WAN Interface ASIC Processing Figure 7. Packet flow from fabric to egress PFE 7. Cells representing packets are received from the Switch Interface ASIC connected to the fabric. The cells are handed off to the Queuing and Memory Interface ASIC for buffering. Lookup keys and data pointers are sent to the T Series Lookup Processor. 8. The T Series Lookup Processor performs the route lookup and forwards the notification to the Queuing and Memory Interface ASIC, which forwards it to the network-facing Switch Interface ASIC. 9. The Switch Interface ASIC sends requests to the Queuing and Memory Interface ASIC to read the data cells out of memory, and it passes the cells to the Layer 2/Layer 3 packet-processing ASIC. 10. The Layer 2/Layer 3 packet-processing ASIC reassembles the cells into packets, performs the necessary Layer 2 encapsulation, and sends the packets to the outgoing PIC. Queuing policy and rewrites occur at this time on the egress router. 11. The PIC passes the packets into the network. The T Series chipset provides hardware-based forwarding performance and service delivery for IPv4 (unicast and multicast), IPv6 (unicast and multicast), and MPLS, while also having the flexibility to support other protocols in the future. The chipset was designed to provide the functionality needed for the development of single-chassis and multichassis systems.8 Copyright © 2012, Juniper Networks, Inc.
    • White Paper - T Series Core Router Architecture Overview T Series Switch Fabric Architecture The N+1 redundancy of the T Series switch fabric architecture is what has led to the in-service upgrade capability and the flexibility of scaling the T Series in the multichassis dimension since its inception. For example, just by swapping the power entry modules and the switch interface boards (using the N+1 redundancy), service providers can upgrade a T640 Core Router to a T1600 Core Router without disrupting service or changing customer-facing interfaces. Furthermore, connecting a T1600 platform to a TX Matrix Plus system is also a smooth in-service process, involving upgrading the switch fabric boards and the control boards on the T1600 and then connecting the T1600 to the central switching element—the TX Matrix Plus. The five planes of the T Series switch fabric (Figure 5) are implemented using four operationally independent, parallel switch planes (Labeled A through D) that are simultaneously active and an identical fifth plane (Labeled E) that acts as a hot spare to provide N+1 redundancy. Plane E (Backup) Plane D Plane C Network PIC PIC Network Ingress Plane B Egress SI (Fabric) SI (Fabric) PFE PFE Network PIC PIC Network Plane A Fabric ASIC Figure 5. Five switch fabric planes for the T Series The Fabric ASIC provides non-blocking connectivity among the PFEs that populate a single-chassis system. On a single-chassis T Series system, each chassis contains a maximum of 8 FPCs, with each FPC supporting 2 PFEs (up to 50 Gbps each), for a total of 16 PFEs that communicate across the switch fabric. The required input and output aggregate bandwidth of the switch fabric exceeds the I/O capabilities of a single Fabric ASIC by a large margin. In a multichassis-capable T Series router, each PFE is connected to four active switch planes, and each switch plane carries a portion of the required bandwidth. To guarantee that cells are evenly load-balanced across the active switch planes, each PFE distributes cells equally across the switch planes on a cell-by-cell basis rather than a packet-by-packet basis. For further design considerations of the T Series switch fabric, see Appendix B: Switch Fabric Properties. Switch Fabric Operation The switch fabric implements a “fairness protocol” that, when congestion is occurring, ensures fairness across source PFEs. The process of transmitting a data cell across the switch fabric involves a request and grant protocol. The source PFE transmits a request across the switch fabric to the destination PFE. Each request for a given destination is transmitted across a different switch plane in a round-robin order to distribute the load equally. When the request is received by the destination PFE, the destination PFE transmits a grant to the source PFE across the same switch plane on which the corresponding request was received. When the grant is received by the source PFE, the source PFE transmits the data cell to the destination PFE across the same switch plane on which the corresponding grant was received. This approach provides both a flow control mechanism for transmitting cells into the fabric and a mechanism to detect broken paths across the switch fabric. Appendix B covers Switch Fabric Design Properties. In particular, there are two key features of the T Series switch fabric that deserve special attention in terms of their ability to handle modern multiservice applications such as video delivery—these are QoS and multicast. As shown in Figure 6, any crossbar endpoint (PFE) can theoretically become congested in the egress direction: multiple ingress PFEs can send traffic to one egress PFE, thus creating potential for congestion. This condition can be handled in any number of ways—backpressure, rate limiting, fabric speedup, or a combination—the key is that it must not result in the loss of priority traffic.Copyright © 2012, Juniper Networks, Inc. 9
    • White Paper - T Series Core Router Architecture Overview Multicast traffic always must coexist with unicast traffic, which is why a T Series switch fabric treats both traffic types equally: unicast and multicast traffic must conform to their QoS profiles mapped into fabric with two priority levels (note PFE 0). Routing Engine Regular Fabric Queues Multicast (Per Destination PFE) Input Unicast Input PFE PFE WAN 0 4 WAN PFE PFE WAN 1 5 SWITCH FABRIC WAN PFE PFE WAN 2 6 WAN PFE PFE WAN 3 N Figure 6. Multicast and unicast each utilizing fabric queues in T Series routers Because the T Series Core Routers use a patented tree replication algorithm, they are able to distribute multicast replication across available PFEs. Thus, in the very rare case of a congested egress PFE, T Series routers can easily apply backpressure notification from egress to ingress to limit the traffic and avoid drops. As competing designs replicate in the fabric itself, and allow multicast traffic to bypass the fabric queues, the potential for intelligent drops with even minimal congestion is very demonstrable. Multichassis Switch Fabric (TX Matrix and TX Matrix Plus) An increasingly popular approach to augmenting forwarding performance, boosting bandwidth density, and extending the deployable lifetime of core routers is to use a multichassis system. Multichassis systems are designed with an expandable switch fabric that allows service providers to grow systems in increments required by budget and traffic loads. While most service providers typically follow the technology curve and upgrade as soon as the next generation of routers comes along (mainly because of improved efficiencies such as higher capacity, better footprint, and lower power), Juniper’s multichassis solution allows providers to grow node capacity either to bridge between generations, or to build multi-terabit nodes. Another common reason for a service provider to use a multichassis system is to prevent the proliferation of too many interconnects. More network elements mean more interconnects that do no support revenue-generating traffic but merely ensure a full mesh. Interconnects are not a good way to scale—in addition to the obvious cost of the ports, there is also a cost in terms of driving additional chassis as well to handle overhead connections. Additionally, multichassis systems can also be combined with an independent control plane such as the JCS1200 to create a virtualized core.3 The JCS1200 allows the creation of hardware-virtualized routers that can assume individual networking functions, such as core or aggregation; service functions (such as private VPNs); mobile packet backbones; or peering functions. CLOS Topology Figure 7 illustrates the topology of a typical three-stage CLOS fabric4. The blue squares represent individual single- stage crossbar switches. The topology consists of multiple rows of single-stage crossbar switches arranged into three columns. The benefit of this topology is that it facilitates the construction of large, scalable, and non-blocking switch fabrics using smaller switch fabrics as the fundamental building block.10 Copyright © 2012, Juniper Networks, Inc.
    • White Paper - T Series Core Router Architecture Overview STAGE 1 STAGE 2 STAGE 3 Input Output Input Output Input Output 1 • • • • • • SWITCH 1 • • P1 • • • • P2 • • • • P3 • • 16 1 • • • • • • SWITCH 2 • • P1 • • • • P2 • • • • P3 • • 16 • • • • • • • • • 1 • • • • • • SWITCH 16 • • P1 • • • • P2 • • • • P3 • • 16 Figure 7. Three-stage CLOS topology The topology of a CLOS network has each output port of the first stage connected to one of the crossbars in the second stage. Observe that a two-stage switch fabric would provide any-to-any connectivity (any ingress port to the fabric can communicate with any egress port from the fabric), but the path through the switch is blocking. The addition of the third stage creates a non-blocking topology by creating a significant number of redundant paths. It is the presence of the redundant paths that provides the non-blocking behavior of a CLOS fabric. Juniper Networks Implementation The Juniper Networks implementation of a CLOS fabric for a multichassis routing node is specifically designed to support the following attributes: • Non-blocking • Fair bandwidth allocation • Maintains packet order • Low latency for high-priority traffic • Distributed control • Redundancy and graceful degradation The existence of multiple parallel paths through the switching fabric to any egress port gives the switch fabric its rearrangeably non-blocking behavior. Dividing packets into cells and then distributing them across the CLOS switch fabric achieves the same effect as moving (rearranging) connections in a circuit-switched network because all of the paths are used simultaneously. Thus, a CLOS fabric proves to be non-blocking for packet or cell traffic. This design allows PFEs to continue sending new traffic into the fabric and, as long as there are no inherent conflicts—such as an overcommitted output port—the switch remains non-blocking. Finally, an important property of a CLOS fabric is that each crossbar switch is totally independent of the other crossbar switches in the fabric. Hence, this fabric does not require a centralized scheduler to coordinate the actions of the individual crossbars. Each crossbar acts independently of the others, which results in a switch fabric that is highly resilient to failures and scales to support the construction of extremely large fabrics. Multichassis System Fabric The T Series router multichassis fabric is constructed using a crossbar designed by Juniper Networks called the Fabric ASIC, which is the same Fabric ASIC used in a single-chassis T Series platform. Depending on its placement in the CLOS topology, each Fabric ASIC is a general building block that can perform stage 1, stage 2, or stage 3 functionality within the switch fabric. The CLOS fabric provides the interconnection among the PFEs in a T Series multichassis routing node (Figure 7). The switch fabric for a multichassis system is implemented using four operationally independent, but identical, switch planes (labeled A through D) that are simultaneously active and an identical fifth plane (labeled E) that acts as a hot spare to provide redundancy. Each plane contains a three-stage CLOS fabric built using Juniper Networks Fabric ASICs.Copyright © 2012, Juniper Networks, Inc. 11
    • White Paper - T Series Core Router Architecture Overview Plane E CHASSIS 1 Plane D CHASSIS 2 Plane C Network PIC PIC Network Ingress Plane B Egress SI (Fabric) SI (Fabric) PFE PFE Network PIC PIC Network Plane A F1 F2 F3 F1 F2 F3 • • • • • • • • • F1 F2 F3 Figure 8. T Series multichassis switch fabric planes The required input and output bandwidth of the switch fabric exceeds the I/O capabilities of a single plane. As with the single-chassis system, each PFE is connected to four active switch planes, with each switch plane providing a portion of the required bandwidth. To guarantee that cells are evenly load-balanced across the active switch planes, each PFE distributes cells equally across the four switch planes on a cell-by-cell basis rather than a packet-by-packet basis. Routing Engine Routing Engine TX Matrix (Plus) Primary Secondary Matrix Control Paths Matrix Data Paths T Series LCC T Series LCC Routing Engine: Redundant Routing Engine: Redundant Local Routing Engine Routing Engine Local Routing Engine Routing Engine FPC FPC FPC FPC FPC FPC PIC PIC PIC PIC PIC PIC Figure 9. Multichassis system high-level overview Line-card chassis are connected to switch chassis with fiber-optic cable. This connectivity uses the latest VCSEL technology to provide extremely high throughput, low power consumption, and low bit-error rates. An abstract view of the interconnections for a TX Matrix Plus multichassis system is shown in Figure 10.12 Copyright © 2012, Juniper Networks, Inc.
    • White Paper - T Series Core Router Architecture Overview TX MATRIX Switch-Card Chassis RE/CB 0 RE/CB 1 RE/CB 0 RE/CB 1 Standby Main Standby Main Optical Data Plane T1600 Plane 0 - Standby T1600 Plane 1 F2SIB LCC 00 F13 SIB LCC 01 LCC 02 F13 SIB LCC 03 T1600 Plane 2 Blank T1600 Plane 3 F2SIB LCC 00 F13 SIB LCC 01 T1600 Plane 4 LCC 02 F13 SIB LCC 03 T1600 Blank Line-Card F2SIB LCC 00 F13 SIB LCC 10 Chassis LCC 02 F13 SIB LCC 03 F2SIB LCC 00 F13 SIB LCC 01 LCC 02 F13 SIB LCC 03 Blank F2SIB LCC 00 F13 SIB LCC 01 LCC 02 F13 SIB LCC 03 Blank Blank Blank Figure 10. Switch-card chassis to line-card chassis interconnections In all T Series systems, there is a fifth data plane on hot standby in the event any of the other four data planes fail—the same is true for a TX Matrix Plus system. Redundancy in the center stage is provided, as the system functions with any two of the center stage chassis. Each center stage chassis has no more than two data planes, so even if one chassis fails, forwarding still proceeds at full throughput for large packets. Five chassis can be supported by the architecture. Conclusion The T Series demonstrates how Juniper has evolved its router architecture to achieve substantial technology breakthroughs in packet forwarding performance, bandwidth density, IP service delivery, and system reliability. At the same time, the integrity of the original design has made these breakthroughs possible. Not only do T Series platforms deliver industry-leading scalability, they do so while maintaining feature and software continuity across all routing platforms. Whether deploying a single-chassis or multichassis system, service providers can be assured that the T Series satisfies all networking requirements. About Juniper Networks Juniper Networks is in the business of network innovation. From devices to data centers, from consumers to cloud providers, Juniper Networks delivers the software, silicon and systems that transform the experience and economics of networking. The company serves customers and partners worldwide. Additional information can be found at www.juniper.net.Copyright © 2012, Juniper Networks, Inc. 13
    • White Paper - T Series Core Router Architecture Overview Appendix A: References Applications for an Independent Control Plane: www.juniper.net/us/en/local/pdf/app-notes/3500134-en.pdf Control Plane Scaling and Router Virtualization: www.juniper.net/us/en/local/pdf/whitepapers/2000261-en.pdf Efficient Scaling for Multiservice Networks: www.juniper.net/us/en/local/pdf/whitepapers/2000207-en.pdf Energy Efficiency for Network Equipment: www.juniper.net/us/en/local/pdf/whitepapers/2000284-en.pdf Network Operating System Evolution: www.juniper.net/us/en/local/pdf/whitepapers/2000264-en.pdf Virtualization in the Core of the Network: www.juniper.net/us/en/local/pdf/whitepapers/2000299-en.pdf14 Copyright © 2012, Juniper Networks, Inc.
    • White Paper - T Series Core Router Architecture Overview Appendix B: Switch Fabric Properties The T Series switch fabric for both single-chassis and a multichassis system is specifically designed to provide the following attributes. • Non-blocking • Fair bandwidth allocation • Maintains packet order • Low latency for high-priority traffic • Distributed control • Redundancy and graceful degradation Non-blocking A switch fabric is considered non-blocking if two traffic flows directed to two different output ports never conflict. In other words, the internal connections within the switch allow any ingress PFE to send its fair share of bandwidth to any egress PFE simultaneously. 4X4 CROSSBAR SWITCH Port 1 Ingress Port 1 Egress Port 2 Ingress Port 2 Egress Port 3 Ingress Port 3 Egress Port 4 Ingress Port 4 Egress Figure 11. A 4 x 4 non-blocking crossbar switch Figure 11 illustrates the internal topology for a non-blocking, single-stage, 4-port crossbar switch. The challenge when building a crossbar is that it requires n² communication paths internal to the switch. In this example, the 4-port crossbar requires a communication path connecting each input port to each output port, for a total of 16 communication paths. As the number of ports supported by the crossbar increases, the n² communication path requirement becomes an implementation challenge. Fair Bandwidth Allocation In a production network, it is impossible to control the pattern of ingress traffic so that an egress port of the crossbar switch is never overcommitted. An egress port becomes overcommitted when there is more input traffic destined for the egress port than the egress port can forward. In Figure 12, the aggregate amount of traffic that ingress ports 1, 2, and 3 forward to egress port 4 is greater than the capacity of egress port 4. Fair bandwidth allocation is concerned with the techniques that a switch uses to share the bandwidth among competing ingress flows to an overcommitted egress port. 4X4 CROSSBAR SWITCH Port 1 Ingress Port 1 Egress Port 2 Ingress Port 2 Egress Port 3 Ingress Port 3 Egress Port 4 Ingress Port 4 Egress Figure 12. An overcommitted egress switch port The T Series router switch fabric provides fairness by ensuring that all ingress PFEs receive an equal amount of bandwidth across the switch fabric when transmitting cells to an oversubscribed egress PFE. Providing this type of fairness across all streams is hard to support because it is difficult to keep track of all users of the bandwidth to the egress PFE. The challenge is that if there are n ports on the switch, then there are n² streams of traffic through theCopyright © 2012, Juniper Networks, Inc. 15
    • White Paper - T Series Core Router Architecture Overview switch. Since most switch architectures are not capable of keeping track of n² individual streams, they are forced to aggregate traffic streams, thus making it impossible to be completely fair to each individual stream. The T Series switch fabric can monitor the n² streams so that each stream receives its fair share of the available fabric bandwidth to an oversubscribed egress PFE. Maintains Packet Order The potential for misordering cells as they are transmitted across parallel switch planes to the egress PFE is eliminated by the use of sequence numbers and a reorder buffer. In this design, the Switch Interface ASIC on the ingress PFE places a sequence number into the cell header of each cell that it forwards into the fabric. On the egress PFE, the Switch Interface ASIC buffers all cells that have sequence numbers greater than the next sequence number it waits to receive. If a cell arrives out of order, the Switch Interface ASIC buffers the cells until the correct in-order cell arrives and the reorder buffer is flushed. The reorder buffer and sequence number space are large enough to ensure that packets (and the cells in a given packet) are not reordered as they traverse the switch fabric. Low Latency for High-Priority Traffic Some types of traffic, such as voice or video, have both latency and bandwidth requirements. The T Series switch fabric is designed so that blocking in the fabric is extremely rare because the size of the ports from the ingress PFE into the fabric is considerably larger than the network ports into the ingress PFE. In the rare case that congestion does occur, each ingress PFE is allocated priority queues into the switch fabric. As previously discussed, the switch fabric fairly allocates bandwidth among all of the ingress PFEs that are competing to transmit cells to an overcommitted egress PFE. • The use of priority queues from the ingress PFE into the switch fabric provides two important benefits: • The latency for high-priority traffic is always low because the fabric never becomes congested. • The CoS intelligence required to perform admission control into the fabric is implemented in the PFEs. This design allows the switch fabric to remain relatively simple because CoS is not implemented inside the fabric, but at the edges of the switch fabric. It also scales better than a centralized design because it is distributed and grows with the number of FPCs. Distributed Control A T Series routing platform does not have a centralized controller that is connected to all of the components in the switch fabric. Hence, within the fabric, if any component fails, the other components around the failed component continue to operate. Additionally, a centralized control channel does not need to be operational for the switch fabric to function. Redundancy and Graceful Degradation Each Switch Interface ASIC monitors a request-grant mechanism. If the ingress PFE depends on a grant for an outstanding request but the grant does not return after a reasonable amount of time, or a data cell is lost, then the ingress PFE takes into account that the destination PFE is unreachable on the plane that was used to send the request. If a switch plane fails, only the cells that are currently in transit across the switch plane are lost because buffering does not occur within a switch plane and the request-grant mechanism ensures that cells are never transmitted across a failed switch plane. The request-grant mechanism allows a failed component within a switch plane to be removed from service by diverting traffic around the faulty component, or all traffic using the faulty plane can be switched to a redundant plane. If there are a significant number of faults on a given switch plane or the plane must be swapped out for maintenance, the chassis manager coordinates moving traffic to the redundant switch plane. Each step in the migration of traffic to the redundant plane involves moving only a small fraction of overall traffic. This design allows the system to remain operational with no significant loss in fabric performance.Corporate and Sales Headquarters APAC Headquarters EMEA Headquarters To purchase Juniper Networks solutions,Juniper Networks, Inc. Juniper Networks (Hong Kong) Juniper Networks Ireland please contact your Juniper Networks1194 North Mathilda Avenue 26/F, Cityplaza One Airside Business Park representative at 1-866-298-6428 orSunnyvale, CA 94089 USA 1111 King’s Road Swords, County Dublin, Ireland authorized reseller.Phone: 888.JUNIPER (888.586.4737) Taikoo Shing, Hong Kong Phone: 35.31.8903.600or 408.745.2000 Phone: 852.2332.3636 EMEA Sales: 00800.4586.4737Fax: 408.745.2100 Fax: 852.2574.7803 Fax: 35.31.8903.601www.juniper.netCopyright 2012 Juniper Networks, Inc. All rights reserved. Juniper Networks, the Juniper Networks logo, Junos,NetScreen, and ScreenOS are registered trademarks of Juniper Networks, Inc. in the United States and othercountries. All other trademarks, service marks, registered marks, or registered service marks are the property oftheir respective owners. Juniper Networks assumes no responsibility for any inaccuracies in this document. JuniperNetworks reserves the right to change, modify, transfer, or otherwise revise this publication without notice.2000302-003-EN Mar 2012 Printed on recycled paper16 Copyright © 2012, Juniper Networks, Inc.