Interoperability Update: Dynamic Ethernet Services Via Intelligent Optical Networks James D. Jones, Alcatel; Lyndon Ong, Ciena; Monica Lazer, AT&TAbstract This article describes the 2005 Worldwide Interoperability Demonstration heldby the Optical Internetworking Forum (OIF) and showcased during SUPERCOMM 2005. Theevent highlighted Ethernet services transported over intelligent optical networks, using equipmentfrom 13 of the industry’s leading vendors located in 7 carrier lab facilities around the world. Thedemonstration utilized a distributed optical control plane based on OIF ImplementationAgreements to control a multi-layer network providing Ethernet over SONET/SDH adaptationand transport. The article describes the global test network, services, architecture and overall testapproach. It also describes innovations made to the optical control plane to handle multi-layersignaling and lists further refinements needed to make these services operational.1 IntroductionThe Optical Internetworking Forum (OIF) conducted its second World InteroperabilityDemonstration, in conjunction with SUPERCOMM held in Chicago on June 7 - 9, 2005. Membercompanies demonstrated dynamic Ethernet services enabled over a global optical network,building on the results of a similar global demonstration in 2004.At SUPERCOMM 2004, OIF demonstrated dynamic end-to-end SONET/SDH connectionmanagement between client devices and transport network elements from many vendors in amulti-domain, transport network spanning multiple carrier laboratories. The 2004 event alsodemonstrated Ethernet adaptation over SONET/SDH using GFP (Generic Framing Procedure),VCAT (Virtual Concatenation) and LCAS (Link Capacity Adjustment Scheme) as a separateobjective. At SUPERCOMM 2005, the OIF took a major step by integrating these two featuresand creating a multi-layer control plane control to trigger end-end Ethernet connections over aSONET/SDH network. The result was a global network enabling clients to directly requestEthernet services over carriers’ SONET/SDH networks. While the demonstration focused onEthernet Private Line service enabled by the distributed control plane, it also evaluated EthernetVirtual Services (Virtual Private Line, Virtual Private LAN and Internet Trunking) over theoptical transport network.This demonstration was motivated by continued growth in demand for Ethernet services in publicnetworks, and the carriers’ imperative to maximize utilization of their existing SONET/SDHtransport infrastructure. To do this, interoperability is required at many levels (i.e., transport,control and management planes) to allow flexibility as the network evolves to support present andfuture Ethernet services. At the same time, carriers have heterogeneous core optical transportnetworks comprised of a range of bearer technologies, infrastructure granularity options, andsurvivability mechanisms. Products and systems tested for interoperability included routers,
multi-service provisioning platforms (MSPPs), SONET/SDH cross-connects, optical switches,optical add-drop multiplexers (OADMs) and reconfigurable OADMs (ROADMs). OIFImplementation Agreements and interoperability trials address the challenge by requiring controlplane solutions be developed in the context of such heterogeneous environments, and be able toco-exist with the existing network.The demonstration was executed on a global stage with seven Carriers across three continentsinter-networking through an intelligent control plane with equipment from thirteen vendorparticipants. A network of over 70 nodes was built up in progressive stages, beginning with locallab testing, followed by intra-continental regional testing and culminating in a live, global real-time network test. During the global demonstration, over 20 optical connections weresimultaneously active among the test sites. This included a live video feed between two carrierlabs, which was then transported to the SUPERCOMM show site. This video connection wasdynamically setup and torn down remotely from the show floor via the optical control plane.2 Creating a World Wide DemonstrationThe OIF World Interoperability Demonstration at SUPERCOMM 2005 was built as a globalnetwork, in that all equipment was located at carriers’ research laboratories across the world:Asia, Europe, and North America. Equipment from multiple vendors was interconnected withinthe laboratories. Virtual connections were established among carriers, with the exception ofseveral instances, as discussed in more detail below.Participants included:• Seven Carrier Lab Locations: • Europe: Deutsche Telekom, France Telecom, Telecom Italia • Asia: China Telecom, NTT • North America: AT&T, Verizon • Thirteen Vendors: Alcatel Lucent Technologies Avici Systems Mahi Networks CIENA Corporation Marconi Corporation Cisco Systems Nortel Networks Fujitsu Sycamore Networks Huawei Tellabs Lambda Optical SystemsThe overall equipment topology for this event is shown in Figure 1. This diagram shows all theequipment involved in the OIF World Interoperability Demonstration, its function in each of thecarrier labs and the transport plane. As mentioned above, in addition to the virtual linksinterconnecting carrier labs facilities, there were several real links that were used to carry videoapplication to showcase the demonstration: one OC3 link connected the AT&T and Verizonfacilities, and one DS3 link connected the AT&T facilities to the SUPERCOMM demonstrationbooth. Video streams were used to illustrate the status of calls during the demonstration.
LEGEND Verizon Asia to Europe Labs Sycamore/EoS Tellabs Alcatel America to Europe or America to Asia Ciena/EoS Intra-continental Nortel Tellabs Client TNE Alcatel Tellabs Mahi NTT Labs Avici Tellabs Alcatel Fujitsu NTT Fujitsu/EoS Alcatel NTT Sycamore Ciena NTT Lucent/EoS Cisco/EoS Avici OC-3 Deutsche Telekom Fujitsu AT&T Labs Ciena NTT Ciena Lucent/EoS Sycamore Avici Avici Marconi Cisco Cisco/EoS Ciena/EoS Navtel Huawei Alcatel Alcatel/EoS China Telecom Ciena Huawei Avici Cisco Marconi Sycamore Marconi Avici Lambda Cisco Optical Telecom Avici Avici Italia France TelecomFigure 1 Overall Topology of OIF World Interoperability DemonstrationIn addition, a Signaling Control Network (SCN) was set-up in an architecture simulatingoperational networks. In each carrier lab, an internal SCN connected all the equipment. Theindividual carrier SCNs were interconnected simulating carrier-to-carrier interconnections.However, since this is still a demonstration network, not an operational network, IPsec tunnels
were used over the public Internet (as opposed to dedicated signaling networks). This SCN wasused for signaling and routing information exchanges. For display purposes, a custom softwareapplication intercepted signaling messages sent over the SCN to build a live view of the activecalls. This application analyzed data fields within the signaling messages to build the picture ofthe active calls, as shown in the example in Figure 2. These global topology views were availableboth in the lab sites and at the SUPERCOMM show floor. Figure 2 Dynamic Display of Active CallsA generic diagram illustrating the service and interface types included in the demonstration isshown in Figure 3. As seen in the diagram, all client devices were connected to the opticalnetwork via Ethernet interfaces. This illustrates the evolution of the OIF UNI  from supportingSONET/SDH services to support of Ethernet services.The 2005 demonstration was focused on dynamic Ethernet services enabled by intelligent opticalnetworks. Ethernet services were delivered across multiple carrier labs with various bandwidthcharacteristics by Switched Connections (SCs, initiated by client UNI signaling), Soft PermanentConnections (SPCs, initiated by a management system) and hybrids of both. To support dynamicEthernet calls, several key technologies were required as described below: GFP , VCAT ,LCAS , UNI 2.0 , E-NNI 1.0  (with extensions), inter-layer call and connectioncoordination. For these calls, the Ethernet signals were mapped to SONET/SDH payloads usingGFP/VCAT/LCAS standards. The test cases included mapping of both full and partial rateGigabit Ethernet signals and both co-routed and diversely routed SONET/SDH containerscarrying an Ethernet signal.
Ethernet Carrier A Carrier B Carrier C Ethernet Client Domain Domain Domain Client OIF UNI OIF E-NNI OIF E-NNI OIF UNI NE NE NE NE NE NE Ethernet SONET/SDH Ethernet UNI-N UNI-N UNI-C UNI-C Ethernet Layer Call/Connection Flow Control Plane SONET/SDH Layer Call/Connection Flow View GigE Virtual Concatenation Group (21 STS-1 or 7 VC-4) GigE GFP-F GFP-F . . . VCAT . . . VCAT Transport Plane . . . LCAS LCAS View Figure 3 Multi-Layer Control Plane and Transport plane ArchitectureIn addition, the control plane operated at multiple layers and inter-layer coordination wassupported in the optical network elements at the edges of the networks. For a SwitchedConnection, the client device control plane interfaced to the network control plane via a UNI 2.0interface supporting Ethernet transport. Using UNI 2.0, there is no need for the customerequipment to have any awareness of how the network implements support for Ethernet services(whether Ethernet mapped to SONET/SDH, or native Ethernet, or Ethernet mapped directly tooptical wavelengths). In this demonstration SONET/SDH transport was used for Layer 1transport. The edge NE performed the appropriate mapping of the Ethernet service to SONETpayload (consistent with the service parameters requested in the signaling messages) andoriginated signaling at both Layer 1 and Layer 2 in support of the request. The reverse processtook place at the egress point of the network.To illustrate the utility of the optical control plane a video application was set-up between theAT&T and Verizon labs as follows (see Figure 4): • A video server was connected to a router (client to the intelligent optical network) at each site. The server was used to transmit video streaming between sites when the Ethernet call was when available. • Whenever the inter-site Ethernet call was established, each lab could view both the video that originated from their lab and the video from the remote lab. • Whenever the Ethernet call was deleted each site could only view video originated in its own lab. • During the SUPERCOMM show, an additional DS3 link connected the AT&T labs facilities and the SUPERCOMM booth. The AT&T video streaming was available continuously at the SUPERCOMM booth. The Verizon video stream was available during the time that the Ethernet call between the routers at the two sites was established. The OIF booth demonstrations started with the Ethernet call established and both videos were visible in the booth. The Ethernet call between the two sites was then deleted and the visitors could see the change in the call map topology on one monitor (similar to
Figure 2), the video control stream from AT&T on a second monitor, and the interrupted Verizon video stream on a third monitor. Next, the Ethernet call was re-established and the call map topology reflected the change. At the same time, the visitors could see the Verizon video streaming again in real time. OIF Booth AT&T Labs Verizon Labs (Chicago, IL) (Middletown, NJ) (Waltham, MA) ATT Video VZ Video Server Server DS3 Private Line Avici Tellabs TSR 8860 UNI 2.0 UNI 2.0 Control plane GigE GigE Static Segments of Video Path Dynamic Segments of Video Path Enabled Ciena Alcatel by Control Plane CD 1677 E-NNI OC3 AT&T-VZ Figure 4 Video Application Configuration3 BackgroundOIF subscribes to the ITU-T ASON architecture, as discussed in , and has based itsImplementation Agreements for the optical UNI and the optical E-NNI on the ITU-T ASONRecommendations, especially ITU-T G.7713.2  for RSVP-based signaling. The optical UNI enables clients of optical networks to dynamically request connections without knowingnetwork internal topology, while the E-NNI  automates the establishment of these connectionsbetween optical networks. Together, UNI and E-NNI permit dynamic A-to-Z provisioning ofservices across an optical network in real time without manual intervention, resulting in faster andmore efficient operation than traditional optical networks. Link state routing protocol based onOSPF-TE based on  is used for automated network topology distribution and link statusupdates inside the network.3.1 Multi-Layer Networking for Ethernet-over-SDHThis year’s testing focused on multi-layer networking, where connections in a client layer aresupported by the dynamic establishment of connections in a server layer. In practice, carriernetworks consist of multiple technology layers, ranging from Layer 3 IP connectionless packettransport down to Layer 0 physical connectivity, such as fiber cross-connection. Often newservices arise at one layer and must be transported efficiently using a core lower layer network.
One current example of this is the support of Ethernet services, which are growing rapidly as acarrier service offering, and must be transported efficiently over the carriers’ core opticaltransport networks. A number of technologies have been developed in the transport plane toimprove the efficiency and flexibility of SONET/SDH for packet/frame transport, including GFP,VCAT and LCAS.Testing in 2005 focused on the use of the optical control plane to control connections in theoptical core network to support Ethernet layer services. As shown in Figure 5, connectionsbetween client devices at the Ethernet layer were supported by dynamically establishedconnections at the optical (SONET/SDH) layer. The number and type of connections at theSONET/SDH layer corresponded to the amount of bandwidth requested at the Ethernet layer, andan adaptation function using GFP, VCAT and LCAS was used at the originating and terminatingpoints to encapsulate the Ethernet frames into SONET/SDH paths.To create a connection, the Ethernet UNI Client (UNI-C) sends an Ethernet connection request tothe Ethernet UNI Network-side switch (UNI-N). The UNI-N is then responsible for determiningthe corresponding SONET/SDH requirements, creating the required SONET/SDH connections,and then signaling to the remote UNI-N that the underlying connections are available and are tobe used for an Ethernet client connection. The additional stage of signaling between the sourceand destination UNI-Ns carries the actual Ethernet layer connection requirements, allowing themapping at the destination UNI-N from SONET/SDH back to the Ethernet service. Both UNI-Nsthen apply Ethernet-SONET/SDH adaptation using GFP. This whole process involves multiplestages of signaling to coordinate events at different layers, making the control plane processingsignificantly more complex than previous years’ demonstrations.
OIF UNI 2.0 OIF UNI 2.0 OIF UNI 2.0 OIF E-NNI I-NNI Domain OIF I-NNI Domain OIF OIF E-NNI E-NNI E-NNI I-NNI Domain Physical Link SONET/SDH Layer Ethernet Layer Gigabit Ethernet STS-3c/VC-4 Connection 350 Mbps Connection OC48/STM16 STS-3c/VC-4 Connection 250 Mbps Connection STS-1 Connection 100 Mbps Connection Figure 5 Multi-layer Intelligent Optical Transport NetworkThe key extension to the protocols was the ability to dynamically trigger the creation of thesupporting server layer connection upon detecting that new optical capacity was needed. Whenthe ingress optical switch received a UNI 2.0 call request (i.e., for an Ethernet connection), itinitiates the process for creation of new optical connections, computes the required path acrossthe optical core and creates the connection, which is then used to carry GFP-encapsulatedEthernet frames.While there is a one-to-one relationship between Ethernet connection and SONET/SDHconnection (or VCAT group) for Ethernet Private Line services, as were tested, future EthernetVirtual Private Line and Private LAN services can allow the same SONET/SDH connection orgroup to be used for multiple Ethernet services. Standards are not complete for how multiplexingwould be supported, but candidate mechanisms are the use of the client’s VLAN tag (if present),application of a carrier VLAN tag at the UNI-N, or other tags. This is an active area ofdiscussion in IEEE, ITU-T and IETF. With Ethernet virtual services, it will be possible to reusean existing SONET/SDH pipe for future Ethernet connections as long as bandwidth is available.The network will be able to respond dynamically to new demands, either creating a new opticalconnection or reusing existing optical connections as needed.3.2 Signaling Extensions for Multi-Layer NetworkingSome of the more interesting problems that needed to be solved for multi-layer networkingincluded the following:
• How to correlate signaling at client and server layers: Since connections were established first at the server layer (SDH) and then at the client layer (Ethernet), there needed to be a way to correlate signaling at multiple layers, so that, for example, the edge switches correctly identified which SDH timeslots were to be used, and were able to exchange signaling directly between them. For the testing, a mechanism called LSP (label switched paths) Hierarchy  was used, which involves the addition of fields in the RSVP signaling to identify the signaling addresses of the edge switches and the creation of virtual interfaces corresponding to the server layer connections . • How to translate the Ethernet bandwidth request into the required SDH components: Since Ethernet bandwidth is expressed in terms of bits per second required and burst rates supported, while SDH bandwidth is expressed in terms of the size of the signal (STS-1, VC-4, VC-4-4v), there needed to be a mapping from the Ethernet bandwidth request over the UNI to the SDH bandwidth requirement at the E-NNI. In practice, such a translation would be a matter of policy determined by the service provider, since it is affected by the guarantees offered by the service provider as well as their core infrastructure. For the Demo, a mapping table was used to unambiguously map one layer to the other. • Routing across multiple layers: In theory, the addressing used for clients and network elements at one layer may be independent of that used at another, so that the routing of the connection may involve translation from addresses in the client layer to addresses used in the server layer. For example, the destination (or in OIF the TNA (Transport Network Assigned) address) for the Ethernet client must be translated to some associated endpoint in the SDH network for SDH path computation. For the Demo, a one-to-one correspondence was assumed, where in real networks a more complex translation may be required.3.3 Control Plane Support of Virtual Concatenation VCAT is an inverse multiplexing capability defined in ITU-T  that allows multiple SONET/SDH channels to be bound into a single higher rate VCAT group (VCG). For the demonstration, separate connections were set up for each component of the group, in order to create higher survivability for the group as a whole. LCAS allows failure of individual connections to be treated as reduced bandwidth in the group without actually causing failure of the entire group. A VCAT group consisting of multiple connections in the server layer was created using multiple call setups, therefore allowing each connection to follow a different path based on its individual path computation. An example of call setup for VCAT is shown in Figure 6. A coordination mechanism was supported to synchronize the establishment of the supporting VCAT connections and the client layer call. Both parallel and sequential strategies of setting up VCAT connections were considered.
Ethernet Client Layer Connection E-NNI VCAT Component Connections UNI 2.0 UNI 2.0 E-NNI E-NNI Figure 6 Setup of diverse VCAT connections3.4 Additional Transport plane TestingAdditional transport plane-only Ethernet testing was done, based on Ethernet servicespecifications developed in ITU-T and the Metro Ethernet Forum (MEF). These testsdemonstrated interoperability in the transport plane for Ethernet Virtual Services, where multipleEthernet services were transported using the same SONET/SDH VCGs. The virtual servicesdemonstrated included: Ethernet Virtual Private Line, Ethernet Virtual Private LAN and InternetAccess/Virtual Trunk. These provide a complementary aspect to the control plane testing, whichfocused on Ethernet Private Line.For Ethernet Virtual Private Line service, for example, individual client flows were tagged at theUNI-N, aggregated into a single transport link and separated at the destination based on thevalues of the VLAN tags. VLAN tags as defined in IEEE 802.1Q were used to identify anindividual service. Testing of Ethernet Virtual services was based on work being done in ITU-Tand MEF, especially ITU-T Recommendations G.8011.1 and .2 , and MEF 6 .3.5 Future WorkFindings from the interoperability testing have been compiled and provided as input to thevarious standards bodies active in optical control plane specification, to identify any areas ofpotential confusion or omission in the standards. Future testing work may be aimed at morecomplex services and topologies such as dynamic control of Ethernet virtual services, as these areincorporated into optical control plane standards work.4 ConclusionsThe 2005 OIF demonstration was the first time Ethernet adaptation and distributed optical controlplanes were brought together in an integrated fashion, and it was done on a global scale. Thecall/connection control of the UNI-N devices was the most important technical innovationdemonstrated, in two respects. First, the UNI-N provides inter-layer control plane coordination asthe client signal enters the network. The UNI-N is responsible for accepting the client connectionrequest, initiating calls in the server layer, and completing the client layer call once the serverlayer is set up. Second, the UNI-N triggers and controls the Ethernet adaptation function andmapping of client layer signals into server layer containers. This architecture minimizes theoverall network impact since the client Ethernet devices and core SONET/SDH devices are onlyconcerned with a single layer.
The OIF World Interoperability Demonstration is an essential step in the evolution process of theoptical control plane, helping to make it suitable for deployment in carrier networks. The testingdemonstrated multi-vendor support of a distributed optical control plane, its ability to controlmulti-layer end-to-end services and the overall commitment to the technology by both vendorsand carriers. The demonstration utilized real network elements from vendors whose marketpresence accounts for 64% of the 2004 worldwide revenue in the optical networking switchingand routing markets (source: RHK). The equipment was hosted in technology evaluation labs oftop-tier carriers in North America, Europe and Asia.While this event provided solutions for a number of technical issues, it also revealed others thatneed to be addressed. The knowledge gained from this interoperability demonstration is beingapplied to the OIF UNI 2.0 and E-NNI 2.0 signaling specifications. The experience also benefitscarriers in planning migrations to distributed optical control planes and anticipating theoperational considerations for multi-vendor networks.The authors would like to acknowledge all the people in the carrier labs for the tremendousefforts in putting the demonstration together and shepherding it through its stages, the staff fromthe participating vendors for the relentless work in accomplishing the interoperability, and thesupport of the OIF leadership in getting it all together.5 References1. OIF-UNI 1.0 Release 2, “OIF-UNI-01.0-R2-Common - User Network Interface (UNI) 1.0 Signaling Specification, Release 2: Common Part” and “OIF-UNI-01.0-R2-RSVP - RSVP Extensions for User Network Interface (UNI) 1.0 Signaling, Release 2”.2. ITU-T G.7041, “Generic Framing Procedure (GFP)”.3. ITU-T G.707, “Network Node Interface for the Synchronous Digital Hierarchy (SDH)”.4. ITU-T G.7042, “Link Capacity Adjustment Scheme (LCAS)”.5. OIF UNI 2.0 Signaling, “”, oif2003.293.6. OIF E-NNI 1.0 Signaling, “OIF-E-NNI-Sig-01.0 - Intra-Carrier E-NNI Signaling Specification”.7. ITU-T G.8080/Y.1304, “Architecture for the Automatically Switched Optical Network (ASON)”.8. ITU-T G.7713.2, “Distributed Call and Connection Management: Signalling mechanism using GMPLS RSVP-TE”.9. ITU-T G.7715.1, “ASON Routing Architecture and Requirements for Link State Protocols”.10. draft-ietf-mpls-lsp-hierarchy-08.txt, “LSP Hierarchy with Generalized MPLS TE”11. ITU-T G.8011/Y1307, “Ethernet over Transport – Ethernet services framework” ITU-T G.8011.1/Y1307.1, “Ethernet private line service” ITU-T G.8011.2/Y1307.2, “Ethernet Virtual Private Line Service”12. MEF 6, “Ethernet Services Definitions - Phase I”.