Carrier Ethernet Services - The Future Public Multi-Vendor Interoperability Test Berlin, September 2008
Carrier Ethernet World Congress 2008 Multi-Vendor Interoperability Test EDITOR’S NOTE INTRODUCTION This year the interoperability This year’s interoperability event focused on the hot staging test for the Future of Carrier Ethernet Services. While each Carrier Ethernet World previous event concentrated on specific topics such Congress took place in as mobile backhaul or service creation, this event parallel to the Beijing aimed to congregate the knowledge and experience Olympics. the industry gained in the last four years into a single 80 engineers from 28 parti- modern, converged network showing all that a tier- cipating vendors with over one service provider is likely to encounter. We therefore tested: Carsten Rossenhoevel 100 systems attended our Managing Director test. According to data from • Converged residential, business and Mobile Heavy Reading, more than Backhaul services 90% of the Carrier Ethernet switch and router market • Clock synchronization share were represented in this test. • Business services realized using E-Line, E-LAN The participating vendors verified 34 test areas in and for the first time E-Tree services any-to-any combinations in ten days, truly challenging the Olympic motto “Faster, Higher, • The leading access, transport and aggregation technologies Stronger“. Carrier Ethernet implementations support more functions and cover more markets today — • Microwave access and transport ranging from core to microwave to access, E-Lines to • Ethernet OAM: Fault management and perfor- E-Trees, triple play to mobile backhaul. mance monitoring It was an outstanding experience to witness the • High availability massive testing feast, a unique get-together of • Management and SLA reporting virtually all leading players with one single goal: To improve In order to construct such a large test multi-vendor interoperability of network and cover all the above test advanced Carrier Ethernet imple- TABLE OF CONTENTS areas a ten day, closed doors hot mentations. staging event was conducted at Participants and Devices ..............3 EANTC’s lab in Berlin, Germany. An EANTC panel of service providers worldwide including Network Design..........................4 Since the first Carrier Ethernet2 experts from COLT, GVT Brazil, Interoperability Test Results ...........4 World Congress in 2005, EANTC PT TELKOM Indonesia, T-Systems has organized interoperability test Ethernet Service Types .................4 and Metanoia Inc reviewed the events which are then showcases at Diverse Access Technologies ........6 the congress. test plan thoroughly to ensure the event’s scenarios are realistic Diverse Transport ........................7 Our interoperability showcases are and sound. MPLS Core .................................8 driven by three main goals: Interestingly, market forces are E-NNI ........................................9 Technical – Through participation in operating at full strength. This Mobile Backhaul.......................10 the event, vendors have the oppor- year, we once again tested three tunity to verify the interoperability of Clock Synchronization ...............14 transport technologies in the test their devices and protocol imple- event’s metro/aggregation Ethernet OAM ..........................15 mentations against the majority of networks: MPLS, PBB-TE and Resilience and Fault Detection.....18 the industry’s leading vendors. T-MPLS. These three compete to Management and SLA Reporting.21 Marketing – The participants can some extent — at our test, they all showcase the interoperability of Acronyms.................................22 proved being well suited for the their latest solutions on a unique, transport of Carrier Ethernet References ...............................23 large-scale platform. services. Standards – When fundamental Service OAM support is becoming mandatory for issues are found during the hot staging event EANTC aggregation and CPE devices; the Ethernet reports the discoveries to the standard bodies. These microwave market flourishes; mobile backhaul in turn update the standards. pushes support for backwards compatibility (ATM EANTC started the preparation for the event by pseudowires, circuit emulation) and new features inviting interested vendors to weekly conference (clock synchronization, IEEE 1588v2, E-Tree, among calls during which the technical and marketing goals others). for the event were discussed and agreed. The test This white paper summarizes in detail the plan, created by EANTC based on the test topics monumental effort that the participating vendors and suggested by the vendors, expanded on the EANTC team underwent. Enjoy the read. experience gained from previous events and was lined up with recent IEEE, IETF, ITU-T and MEF standards.
Participants and DevicesPARTICIPANTS AND DEVICES Vendor Participating DevicesVendor Participating Devices Nortel Metro Ethernet Routing Switch (MERS) 8600Actelis Networks ML658 RAD Data ACE-3205ADVA Optical FSP 150CC-825 Communications ACE-3200Networking ACE-3400Alcatel-Lucent 1850 TSS-40 ASMi-54 5650 CPAM Egate-100 7450 ESS-6 ETX-202A 7705 SAR ETX-202A/MiRICi 7750 SR7 ETX-202A/MiTOP 9500 MPR IPMUX-216/24 LA-210Calnex Solutions Paragon Sync OP-1551 RICi-16Cambridge VectaStar RICi-155GEBroadband Networks Redback Networks — SmartEdge 400Ceragon Networks FibeAir IP-MAX2 an Ericsson Company FibeAir IP-10 Rohde & Schwarz SIT SITLine ETHCiena LE-311v LE-3300 SIAE ALS MICROELETTRONICA ALFOCisco Systems 7606 7604 Spirent Spirent TestCenter ME4500 Communications Catalyst 3750-ME Symmetricom TimeProvider 5000 PTP ME-3400-2CS Grand Master 3 ME-3400-12CS TimeCesium 4000ECI Telecom SR9705 Tejas Networks TJ2030Ericsson Marconi OMS 2400 Telco Systems — T5C-XG a BATM Company T5C-24FFoundry Networks NetIron XMR 8000 T5C-24GHarris Stratex Eclipse (Gigabit) Radio T-Marc-250Networks T-Marc-254 T-Marc-340Huawei Technologies NE5000E Cluster System T-Marc-380 NE40E-4 T-Metro-200 CX600-4 Tellabs 8830 Multiservice RouterInfoVista VistaInsight for NetworksIxia XM2 IxNetworkJuniper Networks M10i MX240 Service Provider Test Plan Review MX480 The draft test plan was reviewed by a panel of global service providers in July this year. TheirNEC Corporation CX2600 feedback and comments were reflected in the final PASOLINK NEO version of the test plan. EANTC and the partici- PASOLINK NEO TE pating vendors would like to thank: COLT, GVT Brazil, PT TELKOM Indonesia, T-Systems andNokia Siemens hiD 6650 Metanoia Inc.Networks Flexi WCDMA BTS FlexiHybrid RACEL
Carrier Ethernet World Congress 2008 Multi-Vendor Interoperability Test NETWORK DESIGN INTEROPERABILITY TEST RESULTS As in previous events we set off to construct a In the next sections of the white paper we describe network that would allow all participating vendors to the test areas and results of the interoperability establish end-to-end Ethernet services with any of the event. The document generally follows the structure other vendors. One of the central design consider- of the test plan. ations for the network was to enable any device Please note that we use the term »tested« when positioned in the access network to reach any other reporting on multi-vendor interoperability tests. The access network device regardless of the other term »demonstrated« refers to scenarios where a device’s point of attachment to the network. This service or protocol was terminated by equipment proved to be especially useful for such end-to-end from a single vendor on both ends. tests as Service OAM or Mobile Backhaul. The specifics of these tests can be found in the test case sections. ETHERNET SERVICE TYPES We also aimed to build a network that would look The Metro Ethernet Forum (MEF) has defined three familiar to service providers. It is perhaps unrealistic Ethernet service types in order to allow the industry to expect that service providers will incorporate all and specifically the customers interested in the current transport technologies into their network. services to have a common language to discuss such Nevertheless the familiar network domains are likely Ethernet based services. The three service types are to exist: access, aggregation, metro and core, defined in terms of the Ethernet Virtual Connection regardless of the chosen transport technology. It is (EVC) construct: realistic, however, to expect service providers to use MPLS in the core. • E-Line – Point-to-point EVC Looking at the network from a customer’s • E-LAN – Multipoint-to-multipoint EVC perspective, we used the following network areas: • E-Tree – Rooted-multipoint EVC • Access: The devices that normally exist at the While the E-Line service type provides a service to customer premise or by NodeBs or base stations exactly two customer sites, the E-LAN and E-Tree were positioned here. We were lucky to see a service types allow the connection of more than two diverse number of access technologies for trans- customer sites. In contrast to the E-LAN service type porting Ethernet such as microwave links, which allows an any-to-any connectivity between copper, and fiber. These devices implemented customer sites, E-Tree introduces two different roles4 the UNI-C construct as defined by the MEF. for customer sites: leaf and root. An E-Tree service • Aggregation: The aggregation area of a network facilitates communication between leaves and roots, consisted of a variety of solutions meant to however, leaves can not communicate with each aggregate customer premise devices. This other directly. An E-Tree service implemented by a included Provider Bridges and H-VPLS Multi- rooted-multipoint EVC can be used to provide Tenant Unit Switches (MTU-s). When applicable multicast traffic distribution and hub-and-spoke these devices performed the UNI-N role in the topologies (e.g. DSL customers to BRAS). network. In the test network we instantiated three specific • Metro: Three different transport technologies definitions of service types: Ethernet Virtual Private were used in each of the three metro area Line (EVPL), Ethernet Virtual Private LAN (EVP-LAN), networks: MPLS, PBB-TE and T-MPLS. This and Ethernet Virtual Private Tree (EVP-Tree). All allowed each transport technology to test its own services were configured manually in the network. resiliency and Network-to-Network Interface Due to the increasingly large amount of devices and (NNI) solutions. vendors we had present at the hot staging, this • Core: As stated above, IP/MPLS was used to process was time consuming and prone to mistakes. support connectivity between the different metro A multi-vendor provisioning tool would have been area networks in order to realize end-to-end ideal for the testing and is recommended for any services. In addition, MPLS Layer 3 VPNs as service provider planning to deploy Carrier Ethernet defined in RFC 4364 were tested in the core of services. the network. The services created in the network were configured The physical network topology presented here in two ways: depicts the roles of all the devices and their respective placement in the network. Please note that • EVCs that remained within the same metro area many tests required logical connectivity between the network devices, often at an end-to-end nature, which will be • EVCs that crossed the network core shown, where applicable, using logical topologies The sections below describe the services in the in each test section. network in detail.
Ethernet Service TypesE-Tree core, two of which provided E-NNI leaf connectivity to the other metros - the Alcatel-Lucent 7750 SR7 toFor the first time at an EANTC interoperability event, T-MPLS and the ECI SR9705 to PBB-TE. The Tejasan E-Tree service instantiation was established. One TJ2030 interpreted this E-NNI connection as the rootEVP-Tree was configured with one root node within connectivity for the PBB-TE metro, and the Ericssonthe MPLS metro area and leaves throughout all Marconi OMS 2400 did the same for the T-MPLSnetwork areas. The MEF defines an E-Tree service to metro.be a rooted Ethernet service where the roots areable to communicate with all leaves, and all leaves The diagram in figure 1 shows all points whereare able to communicate with the roots, but not with E-Tree traffic was verified. The logical connectionseach other. represent something different in each area: Ethernet pseudowires in the MPLS, PBB-TE trunks in theThis service utilized each metro technology in a PBB-TE, and TMCs in the T-MPLS networks.unique way. The MPLS metro used a separate VPLSinstance to create this service, using different splithorizon groups to ensure that leaf UNIs could only E-LANcommunicate with the root UNI, but could notestablish communication between each other. The One EVP-LAN was configured in the network withCisco ME4500 implemented the root UNI-N and customer ports in all three metro areas. Thehanded the service off to the Nokia Siemens construction of the EVP-LAN service used differentNetworks hiD 6650 which propagated the tree into mechanisms in each metro area. These mechanismsthe MPLS metro. The Juniper MX480 configured a are described in details in the diverse transportleaf using MPLS towards the Cisco 7606 which section.treated this connection as the root for the corenetwork. Three leaves were configured within the Cisco ME-3400-2CS Cisco ME4500 Telco Systems T-Marc-340 Nokia Siemens Networks hiD 6650 MPLS 5 ECI SR9705 Telco Systems MTU-s MTU-s T-Metro-200 Telco Systems Huawei Juniper Redback Tellabs T-Metro-200 CX600-4 MX480 SmartEdge 8830 400 MTU-s MTU-s Cisco Juniper 7604 MX480 Telco Systems Ciena Cisco T-Metro-200 LE-311v Ceragon 7606 ADVA FibeAir IP-10 FSP 150CC-825 Ericsson Alcatel-Lucent ECI 7750 SR7 SR9705 Tejas TJ2030 Marconi OMS 2400 Foundry Ericsson NetIron XMR 8000 Tejas TJ2030 Marconi OMS T-MPLS PBB-TE 2400 Alcatel-Lucent 1850 TSS-40 Nortel Ciena Ericsson MERS 8600 LE-3300 Marconi OMS Telco Systems 2400 T-Marc-380 ADVA Cambridge Harris Stratex FSP 150CC-825 VectaStar Eclipse Access MTU Root UNI Leaf UNI Device MTU-s Switch Aggregation Metro/Core Root/Leaf Logical Path Device Device E-NNI Propagation Figure 1: E-Tree logical connections
Carrier Ethernet World Congress 2008 Multi-Vendor Interoperability Test E-Line DIVERSE ACCESS TECHNOLOGIES The E-Line service type configured in the network The different services in the test network used a used Virtual LAN (VLAN) IDs to distinguish between variety of access technologies to reach the simulated the various services. In some cases, much like real last mile customer access device. Most services used world networks, a switch positioned at the customer fiber (multi-mode) and copper based Gigabit site would add a Service VLAN tag (S-VLAN) to the Ethernet. One UNI was implemented over a single Ethernet traffic provided by the customer, therefore, strand fiber cable using IEEE 802.3ah defined allowing the customer to maintain its private VLAN 1000BASE-BX10 between the Cisco ME4500 and addressing scheme and separate the customer VLAN the Cisco ME-3400-2CS. Two Actelis ML658 space from the provider’s. devices used G.SHDSL.bis to connect the aggre- gation area to the access. RAD demonstrated a wide variety of access technologies including EFM bonding of four G.SHDSL.bis pairs between the ASMi-54 and LA-210. In addition, RAD demon- strated Ethernet over PDH connectivity with the ETX-202A with MiRICi E1/T1 over a single E1 link T-MPLS and the RICi-16 over 16 bonded E1 links, both aggregated by the Egate-100. The PDH to channelized STM-1 was performed by the OP-1551. MPLS PBB-TE Several Ethernet access links comprised of two Ethernet links with a microwave signal in between. These systems are described in more detail below. Microwave for Access and Transport In recent years we have seen an increased interest in our interoperability events from vendors offering UNI-C Microwave Access Device microwave connectivity and network solutions. UNI-C Access Device Microwave solutions alleviate the need to roll out6 physical wire infrastructure and are especially UNI-N Aggregation Device prevalent in such areas as cellular backhaul, emerging markets, large corporation networks, UNI-N Metro Device hospitals, and mobile-fixed operators. User Network Interface (UNI) This event enjoyed the participation of the following microwave products: Alcatel-Lucent 9500 MPR, Figure 2: E-Line service creation Cambridge VectaStar, Ceragon FibeAir IP-10 and FibeAir IP-MAX2, Harris Stratex Eclipse, NEC All vendor devices successfully participated in PASOLINK NEO, Nokia Siemens Networks Flexi- creation of E-Line services. From the number of Hybrid, and SIAE MICROELETTRONICA ALS and combinations tested, we are confident that an any-to- ALFO. In addition, Cambridge Broadband Networks any combination of endpoints is possible. provided a point-to-multipoint microwave system Three of the E-Line services created between the with which providers can connect either multiple three metro clouds were encrypted using Rohde & customer offices or multiple base stations via Schwarz SITLine ETH. The encryption device was Ethernet or E1 lines. situated between the UNI-C (which was emulated by Since the radios rely on a signal through the air Spirent TestCenter) and Alcatel-Lucent 7705 SAR, some weather events such as rain and heavy fog Telco Systems T-Marc-380 and Telco Systems can cause the signal to degrade effectively T-Metro-200 all of which were serving as UNI-N decreasing the range or capacity of the link. Radio devices. Once the encryption connections were devices can recognize the decrease in air-link established we verified that the EVCs were indeed capacity and some solutions can distinguish which encrypted and that the connection remained stable. frames should be prioritized and further transported versus which frames will be dropped. The Alcatel- Lucent 9500 MPR, Cambridge VectaStar, Ceragon FibeAir IP-10, Harris Statex Eclipse, and SIAE MICROELETTRONICA ALFO showed this function- ality by decreasing the modulation scheme in Quadrature Amplitude Modulation (QAM) which caused traffic loss only to best effort frames and no or minimal loss to prioritized traffic with unaffected latency.
Diverse TransportServices relying on microwave equipment will need of a single Virtual Private LAN Services (VPLS)to be made aware when the microwave signal is too instance utilizing both VPLS PEs and H-VPLS MTUweak to transmit traffic. The link state propagation switches established between the following devices:function disables the Ethernet link state for all ports Alcatel-Lucent 7450 ESS-6, Ciena LE-311v, Ciscoassociated with the microwave link. This function- 7604 and Catalyst 3750-ME, ECI SR9705, Huaweiality was demonstrated by the Ceragon FibeAir CX600-4, Ixia XM2 IxNetwork, Juniper MX240 andIP-10, Harris Stratex Eclipse and the SIAE MX480, Nokia Siemens Networks hiD 6650,MICROELETTRONICA ALFO. These devices also Redback SmartEdge 400, Tellabs 8830, and Telcoshowed their capability to propagate an incoming Systems T-Metro-200. This VPLS instance used LDPloss of signal on a tributary Ethernet port across the for signaling statically configured peers asmicrowave link and switching off the appropriate described in RFC 4762. These devices also estab-physical port on the other side of the radio lished Ethernet pseudowires using LDP to facilitateconnection. point-to-point Ethernet services.Cambridge Broadband Networks demonstrated A separate VPLS instance was used to test BGP-their ability to share point-to-multipoint link capacity based Auto-Discovery, which was successfully estab-between several end stations. In the demonstration lished between the Cisco 7606 and the ECIthree end stations were defined to share a 45 Mbps SR9705. A total of four vendors were interested inwireless link to a central controller. Cambridge testing BGP-based Auto-Discovery, one of whichBroadband Networks showed that when capacity uncovered an interoperability issue during the testsbetween one base station and controller was not where packets captures were taken to be furtherused, the remaining base stations could use the studied in their labs.extra capacity. In order to test the interoperability of VPLS implemen-Over the last few years we have seen an impressive tations which use BGP for signaling as described inincrease in the features built into microwave RFC 4761, another separate VPLS instance wastransport. While historically microwave solutions configured. This was tested between the followingwere used to provide a virtual wire, we see more devices with BGP-based Auto-Discovery enabled:and more intelligence built into the solutions — on Huawei CX600-4 and Huawei NE40E-4, andseveral products a complete Ethernet switch Juniper MX240 and Juniper MX480. The Juniperfunctionality. MX480 performed an interworking function between this BGP signaled VPLS domain and an LDP signaled VPLS domain with the Cisco 7604.DIVERSE TRANSPORT 7The Carrier Ethernet architecture specified by the Provider Backbone BridgeMEF is agnostic to the underlying technology used to Traffic Engineering (PBB-TE)provide Carrier Ethernet services. The creation andsupport of such services is, however, an essential One of the potential solutions to delivering MEFcomponent of the interoperability test event. Mainly defined services using Ethernet technologies only isthree technologies compete for Carrier Ethernet the IEEE defined Provider Backbone Bridge TrafficTransport: MPLS, PBB-TE and T-MPLS. During this Engineering (PBB-TE). The technical specification isevent we had the opportunity to verify all three defined in 802.1Qay which is working its waytechnologies. The following sections describe test through the standard process and is in draft versionresults for each technology in detail. 3.0 at the time of the testing. The standard extends the functionality of the Provider Backbone Bridges (802.1ah) adding a connection-oriented forwardingMPLS mode by creating point-to-point trunks. These trunksMPLS is defined in a set of protocols standardized deliver resiliency mechanisms and a configurableby the Internet Engineering Task Force (IETF) and the level of performance.IP/MPLS Forum. MPLS is positioned to deliver layer The vendors participating in the PBB-TE transport2 and layer 3 services including Ethernet services as domain included Ciena LE-311v2, Ciena LE-3300,defined by the MEF while being agnostic to the Ixia XM2 IxNetwork, Nortel MERS 8600, and Tejasunderlying transport technology. TJ2030.The tests in this area were based on previous In the PBB-TE Metro network we were able to test theexperience gained from EANTC’s Carrier Ethernet establishment of E-Line, E-LAN, and E-Tree services.World Congress and MPLS World Congress interop- The establishment of E-Line services was straight-erability test events and reached a larger number of forward as we tested it in several previous events.participants than in previous events, including a total E-LAN and E-Tree services creation was tested testedof 12 vendors testing MPLS implementations. The for the first time within the PBB-TE cloud. For theMPLS metro domain operated independently from E-LAN service Ciena LE-3300 and Tejas TJ2030the MPLS core network. switches established bridging instances per PBB-TEThe MPLS metro network was built solely for the trunk and C-VLAN/S-VLAN IDs. Every PBB-TE edgepurpose of Carrier Ethernet services. Multipoint-to- device established a trunk for each particular UNI tomultipoint services were facilitated with the creation one of the bridges. A few issues related to usage of different Ethertype values in CFM messages,
Carrier Ethernet World Congress 2008 Multi-Vendor Interoperability Test padding, and different interpretation of CCM using a multipoint architecture similar to VPLS. On intervals were discovered in the initial configuration one particular E-Line service, both Alcatel-Lucent phase of PBB-TE trunks, however, these issues were 1850 TSS-40 and Ericsson Marconi OMS 2400 resolved quickly. were able to successfully test Quality of Service In addition, the Nortel MERS 8600 and the Spirent (QoS) by distinguishing between three different TestCenter tested one of the latest additions to classes of service within the same Ethernet service Ethernet - Provider Link State Bridging (PLSB) – a pre- and only drop low priority traffic when interfaces standard implementation of the IEEE 802.1aq were oversubscribed. (Shortest Path Bridging) which is in draft version 0.3. Since the T-MPLS standards do not define a control The protocol uses the IETF defined IS-IS protocol for plane protocol, the T-MPLS connections between distributing Backbone MAC addresses and Service vendors were manually configured. Ericsson used IDs of participating nodes across the network. Once two proprietary management tools (ENEA and the network topology has been learned, IS-IS is used DiToNe) to setup the T-MPLS network configuration to establish loop-free multipoint-to-multipoint services. on their devices. The forwarding plane uses PBB (802.1ah), however since the other devices in the PBB-TE network did not support PLSB the three Nortel MERS 8600 devices T-MPLS to MPLS-TP Migration were able to use a re-encapsulation of either PBB-TE Following the approval of the first version of the trunks or VLAN tags to peer within the PBB-TE ITU recommendations on T-MPLS, the IETF and network. The Nortel MERS 8600 devices and the ITU-T jointly agreed to work together to extend Spirent TestCenter emulated nodes successfully MPLS protocols to meet transport network learned the appropriate B-MAC addresses, and requirements in order to ensure a smooth forwarded the respective traffic accordingly. convergence of MPLS-based packet transport Tejas Networks demonstrated a logical Ethernet LAN technology. A Joint Working Group (JWT) was network with an IEEE 802.1ad based Ethernet Ring formed between the IETF and the ITU to achieve Protection Switching (ERPS). This ring based control mutual alignment of requirements and protocols protocol being standardized under ITU-T G.8032 is and to analyse the different options for T-MPLS a protection mechanism which offers carriers a standard progress. On the basis of the JWT deterministic sub-50 ms network convergence on a activity, it was agreed that the future standard- fiber failure as opposed to the conventional loop- ization work will focus on defining a transport breaking mechanisms like Rapid Spanning Tree profile of MPLS (named MPLS-TP) within IETF Protocol (RSTP). ERPS convergence time is and in parallel aligning the existing T-MPLS8 independent to the number of nodes in the network, Recommendations within ITU-T to the MPLS-TP thereby vastly enhancing the scalability of a carrier work in IETF. network. We measured failover and restoration of At their Dublin meeting in July 2008, the IETF below 35 ms for the demonstrated ERPS. has initiated the work on MPLS-TP. Due to the fact that IETF MPLS-TP standard or drafts do not exist yet, we tested the implementations based Transport MPLS (T-MPLS) on the T-MPLS ITU-T Recommendations currently This test marked the third T-MPLS interoperability in force and its relevant drafts. testing at EANTC. The following devices successfully It is our intention also to include the first imple- participated in the T-MPLS area during the event: mentations of MPLS-TP drafts at our next event. Alcatel-Lucent TSS-40, Ericsson Marconi OMS 2400, and Ixia XM2 IxNetwork. The T-MPLS standards specify the networking layer for packet transport networks based on MPLS data MPLS CORE plane and designed for providing SONET/SDH-like Since MPLS is used by the majority of service OAM and resiliency for packet transport networks. providers as core technology it is only logical that Alcatel-Lucent and Ericsson successfully tested the when providers add Carrier Ethernet services to their creation of E-Line, E-LAN and E-Tree services, the last product offering the MPLS core will be used. We of which was a first at an EANTC interoperability followed this approach and used the MPLS core to event. Both participants constructed T-MPLS paths connect between three different Ethernet transport (TMP) which are end-to-end tunnels that aggregate metro areas. The core area was constructed using T-MPLS channels (TMC) representing the services. the following edge devices, all of which successfully The TMPs and TMCs were transported over different established multiple VPLS domains and CVirtual physical layer types including 1 Gbit Ethernet, Private Wire Services (VPWS) using LDP for various 10 Gbit Ethernet, ITU-T G.709, and SDH STM-16. Ethernet services: Alcatel-Lucent 7750 SR7, Cisco The Alcatel-Lucent 7705 SAR was used as a non- 7606, ECI SR9705, Foundry NetIron XMR 8000, T-MPLS switch in the aggregation area, interfacing to Huawei NE40E-4, Juniper M10i, and Tellabs 8830. the T-MPLS domain by means of statically configured All devices were physically connected to Huawei MPLS labels. NE5000E cluster system P router (P for Provider, as The E-LAN and E-Tree services were configured opposed to PE for Provider Edge) through which all services were tunneled through by default.
External Network to Network Interface (E-NNI)In addition to providing transport for Ethernet MPLS Metro Connectivity to the Coreservices, all edge devices in the core established anIP/MPLS L3VPN service using BGP (based on RFC Several options exist to allow connectivity between4364). The Alcatel-Lucent 7750 SR7 and Cisco two MPLS areas. The preferred options were MPLS7606 terminated Ethernet pseudowires into this VPN based, but one option used IEEE 802.1ad Providerproviding the potential to offer layer 3 services to Bridging tags, or simply 802.1Q VLAN tags tocustomers which are not reachable otherwise. transport services between the two areas. The Label Edge Router (LER) in the MPLS core would strip the MPLS header from traffic before it forwarded theEXTERNAL NETWORK TO Ethernet frames to the LER in the MPLS metro. The S-Tag or VLAN tag would then signal to the MPLSNETWORK INTERFACE (E-NNI) metro device which pseudowire to forward the T-MPLS PBB-TE frames onto. Devices using VLAN tags were Alcatel- Tejas Lucent 7450 ESS-6, ECI SR9705, and Foundry Alcatel-Lucent Ciena TJ2030 NetIron XMR 8000. Ericsson 1850 TSS-40 LE-3300 Marconi The other option used to connect between adminis-OMS 2400 tratively separated MPLS core and metros is referred to as pseudowire (PW) stitching. This involves the creation of two pseudowires, one in each domain, Alcatel-Lucent ECI Nortel and then interconnecting them either within one 7750 SR7 SR9705 MERS 8600 device, or with a third pseudowire between the two edge devices. Vendors who took this approach Huawei Tellabs 8830 chose the latter. In this case two MPLS labels must be NE40E-4 signaled: the inner label (PW label) signaled by Foundry LDP, and the transport label (PSN label) signaled by Juniper NetIron XMR 8000 M10i either eBGP (IPv4+label) or LDP. To facilitate the Cisco transmission of LDP sessions, either a separate OSPF 7606 area was enabled between the two edge devices or a static route was used. Devices taking the PW stitching approach were Cisco 7606, Juniper M10i, Juniper MPLS Alcatel-Lucent Juniper MX480, and Redback SmartEdge 400. MX480 7450 ESS-6 Redback ECI SmartEdge 400 9 SR9705 PBB-TE Connectivity to the Core Metro/Core Device As PBB-TE and 802.1ad are both part of the IEEE Provider Bridging domain of technologies, it is not Provider Bridging (802.1ad) surprising that Provider Bridging tags were LDP-based E-NNI supported across the board in the PBB-TE metro eBGP (IPv4+label)-based E-NNI domain. All services crossing the core into the VLAN (802.1Q) PBB-TE cloud used S-Tags (Service Tags) to distin- Static MPLS Configuration guish each service between a core edge router and a PBB-TE switch. These devices included Ciena Figure 3: E-NNI to the core LE-3300, ECI SR9705, Nortel MERS 8600, Tejas TJ2030, and Tellabs 8830. One PBB-TE trunk was configured between Nortel MERS 8600 and TejasAs we described above three different technologies TJ2030 and traversed the MPLS core.were used in the metro areas. The problem thatevery service provider then faces is to connect themetro area with the existing network core. In our test T-MPLS Connectivity to the Corenetwork, much like in most service providernetworks, the core used MPLS for transport and Two options were used to establish services over anservices. Therefore, we required mechanisms to MPLS core into a T-MPLS network. The first was toallow services originating on one metro area to use 802.1ad S-Tags, similarly to the MPLS metro.cross the core and be received on other metro The second option used was pseudowire stitching.areas. The T-MPLS edge device terminated TMCs comingThe following subsections describe the specific from the edges of the network and stitched them toNetwork to Network Interface (E-NNI) solutions used an MPLS Ethernet pseudowire which was estab-in the network. lished with the neighboring core edge device. This was done using LDP for both MPLS labels, which ran over a separate OSPF area than the core. This option was tested between Alcatel-Lucent 1850 TSS-40 and the Alcatel-Lucent 7750 SR7, which also had some services configured over statically configured MPLS pseudowires.
Carrier Ethernet World Congress 2008 Multi-Vendor Interoperability Test MOBILE BACKHAUL Carrier Ethernet Network Traditionally, the interface between mobile base Requirements stations and base station controllers has been based on a number of parallel TDM circuits (for GSM, Mobile Network TDM Circuit Emulation Sub-second Resiliency Sub 50-ms Resiliency CDMA) or ATM connections (for the first versions of Service Types ATM Pseudowires UMTS and CDMA 2000) carried on E1 or T1 links. E-Lines Services E-LAN Services E-Tree Services Several market studies show that the transport network costs account for 20–30% of a mobile operator’s operational expenditure (OPEX). SGSN BSC Traditional GSM X X MSC Mobile Core Traditional 3G (UMTS X X Backhaul Network Rel. 99 / CDMA2000) RNC Hybrid 3G offload X X X GGSN Radio Access Network (ATM-based voice; IP tunneled data) MSC — Mobile Service Switching Center Full packet-based 3G a X X SGSN — Serving GPRS Support Node GGSN — Gateway GPRS Support Node Long-Term Evolution X X BSC — Base Station Controller (LTE, 4G) RNC — Radio Network Controller Mobile WiMAX a X X Figure 4: Mobile Backhaul scope a. Can be used as a fallback With the advent of high-speed data transport (HSPA) in 3G networks, with WiMAX and LTE on the Circuit Emulation Services (CES) horizon, the amount of data traffic in mobile10 There is a number of specifications defining circuit networks has vastly grown and will continue to do so. Mobile operators are considering mobile emulation services which could be used to support backhaul over Carrier Ethernet networks, as these Mobile Backhaul services. During our event we provide enough bandwidth for any predicted tested implementations and observed demonstrations increase in data traffic and are more cost effective of MEF8 and RFC 4553 specification for E1 inter- faces. than the current TDM networks. During the tests and demonstrations the devices were The main issue and test focus for Mobile Backhaul connected either back-to-back, back-to-back with an transport is the migration path from TDM/ATM to impairment generator of Calnex Paragon Sync converged packet based services. Thousands of emulating a network behavior between the two base stations will not be upgraded immediately or devices under test, or over the whole test network. not at all. Migration paths vary widely depending We accepted a test or a demonstration if the two on the specific service provider environment. devices performing circuit emulation were able to In this test event, we verified a number of migration pass E1 data over the packet based network and the scenarios, focusing TDM and ATM transport over deviation of the E1 signal received from the network Carrier Ethernet as well as clock synchronization. compared to its input signal was within 50 parts per The following table provides an overview of the million (ppm) over 10 minutes. transport requirements imposed by different mobile As shown in the CES tests back-to-back figure in total network technologies. 5 products from three different vendors passed the tests: Alcatel-Lucent 9500 MPR, NEC CX2600, NEC PASOLINK NEO TE, RAD IPMUX-216/24, and RAD MiTOP-E1/T1 hosted by the RAD ETX-202A. All tests were performed using MEF8 for encapsulation and adaptive clocking for clock synchronization. The test between Alcatel-Lucent 9500 MPR and RAD IPMUX-216/24 was performed once with and once without the optional RTP (Real Time Protocol) header. All other tests were performed without RTP. As described in RFC 4197 section 4.3.3, the usage of RTP relaxes the tolerance requirement for the internal clocks of the devices performing CES and therefore
Mobile Backhauldecreases the probability of jitter buffer overflow or Alcatel-Lucent demonstrated MEF8 CES with differ-underflow. ential clocking by using RTP (Real Time Protocol)In addition, CES was demonstrated over the whole header between Alcatel-Lucent 9500 MPR andtest network as shown in the diagram ., and tested Alcatel-Lucent 9500 MPR devices over T-MPLS metroback-to-back with the Calnex Peragon Sync network. In the same demonstration Alcatel-Lucentimpairment tool, as shown in Figure 6: CES tests showed its proprietary solution to synchronize twoacross the network. The Calnex Paragon Sync microwave endpoints over the air by transportingemulated the jitter of a network path with 10 nodes the clock information from the Alcatel-Lucent 9500and 40% traffic utilization for a more realistic MPR performing the CES to another Alcatel-Lucentscenario. 9500 MPR over the airFigure 6: CES Across the Network. . Alcatel-Lucent NEC PASOLINK Alcatel-Lucent Alcatel-Lucent 9500 MPR NEO TE 9500 MPR 9500 MPR Alcatel-Lucent RAD RAD ACE-3400 RAD ACE-3205 9500 MPR IPMUX-216/24 Alcatel-Lucent Calnex RAD ETX202A RAD ACE-3200 RAD ACE-3205 9500 MPR Paragon Sync with MiTOP E1/T1 Alcatel-Lucent Calnex NEC CX2600 9500 MPR Paragon Sync NEC CX2600 NEC CX2600 RAD Calnex Alcatel-Lucent 9500 MPR Access network TDM service IPMUX-216/24 Paragon Sync UNI TDM link RAD NEC CX2600IPMUX-216/24 11 Metro network Access Device TDM service TDM link Figure 6: CES tests across the network Access Device Figure 5: CES tests back-to-back ATM Transport over MPLS As described in the introduction, ATM transportA number of microwave solutions participated in this services over a Packet Switched Network (PSN) isevent and have demonstrated their ability to another key requirement for the migration of Mobiletransport E1 TDM data over their microwave links, Backhaul to packet switched networks. During thenamely Alcatel-Lucent 9500 MPR, Ceragon FibeAir event two implementations of ATM transport overIP-10, and NEC PASOLINK NEO. The Alcatel-Lucent MPLS as specified in RFC 4714 were tested and9500 MPR microwave solution performed at the demonstrated.same time circuit emulation service. Successful interoperability was tested betweenRAD Data Communications demonstrated IETF circuit Nokia Siemens Networks Flexi WCDMA BTS andemulation (SAToP, RFC 4553) over MPLS which is RAD ACE-3200. The two devices encapsulated ATMvery similar to the MEF8 specification. In one case data into a statically configured MPLS pseudowirethe Circuit Emulation Service was demonstrated and sent the resulting MPLS packets over an IP tunnelbetween RAD ACE-3200 and RAD ACE-3205 in across the PSN. The ATM pseudowire was used toMPLS metro network. The Ceragon FibeAir IP-10 transport an ATM circuit configured between Nokiamicrowave solution was connected both between Siemens Networks RACEL, a Radio Networkthe RAD ACE-3200 and E1 source, and also Controller (RNC) and mobile core network emulator,between the RAD ACE-3200 and an MPLS metro and Nokia Siemens Networks Flexi WCDMA BTS.edge device. Ceragon Networks and RAD Data In addition, RAD Data Communications demon-Communications demonstrated a hybrid mobile strated a statically configured ATM pseudowirebackhaul network operation, effectively combining between the RAD ACE-3205 and RAD ACE-3400native TDM transport and Ethernet encapsulated devices. This pseudowire was tunneled over theCES. In another test case SAToP CES was demon- PBB-TE network.strated between RAD ACE-3400 and RADACE-3205 in PBB-TE network.
Mobile Backhaul Physical Topology Diagram Harris Stratex Alcatel-Lucent 9500 MPR Spirent Eclipse TestCenter Rohde & Schwarz SITLine ETH RAD ADVA FSP RICi-155GE 150CC-825 RAD RICi-155GE Alcatel-Lucent 7705 SAR Alcatel-Lucent SIAE MICROELETTRONICA ALS 1850 TSS-40 InfoVista VistaInsight for Networks Ericsson NEC PASOLINK NEO Marconi OMS 2400 IXIA XM2 IxNetwork T-MPLS Metro Alcatel-Lucent Nokia Siemens Networks 1850 TSS-40 FlexiHybrid Huawei NE5000E Cluster System Alcatel-Lucent Ericsson Ericsson 7750 SR7 Gaming Client Alcatel-Lucent 9500 MPR Marconi OMS 2400 Marconi OMS 2400 Huawei Cambridge NE40E-4 Telco Systems IXIA XM2 T-Marc-254 VectaStar IxNetwork Juniper 12 M10i Huawei Video Client CX600-4 Spirent Telco Systems TestCenter T5C-24G RAD Juniper RICi-155GE Juniper MX480 Cisco MX480 ME3400-12CS Symmetricom TimeProvider Harris Stratex Redback 5000 PTP Grand Master Eclipse SmartEdge 400 Telco Systems Cisco 7604 T-Metro 200 ADVA FSP MPLS MTU-s Ciena 150CC-825 Spirent LE-311v Ceragon SIAE TestCenter MTU-s FibeAir IP-10 MICROELETTRONICA ALS Telco Systems Ceragon T5C-24F FibeAir IP-10 Juniper MX240 NEC Telco Systems CX2600 T-Marc-250 Ceragon MTU-s RAD FibeAir IP-10 ETX-202A + MiRICi E1/T1 RAD ACE-3205 RAD RAD RAD ASMi-54 ETX-202A ACE-3200 RADApplication Demonstrations + MiRICi E1/T1 RAD LA-210 ETX-202A Gaming Clients running on E-LAN service Video Equipment running on E-Tree service Symmetricom TimeCesium 4000
Device Types Gaming Client Metro/Core Network Device Actelis IXIA XM2 Aggregation Device RAN BS ML658 IxNetwork InfoVista VistaInsight for Networks RAN NC Tester RAD Video Client Access Device IPMUX-216/24 Harris Stratex P Router Eclipse Radio Transmission Device Cluster System Rohde & Schwarz SITLine ETH Alcatel-Lucent Actelis 95000 MPR ML658 Telco Systems RAD T-Marc 380 ACE-3400 RAD RAD ETX-202A RICi-16 Tejas Tejas TJ2030 RAD TJ2030 OP-1551 NEC Nortel PASOLINK NEO TE Tejas MERS 8600 RAD Tejas TJ2030 Egate-100 TJ2030 Spirent TestCenter PBB-TE Metro ADVA FSP Ciena LE-3300 NEC PASOLINK NEO RAD 150CC-825 ACE-3205 ECI SR9705 Nortel MERS 8600 Ciena LE-311v SIAE MICROELETTRONICA ALFO Tellabs 8830 Nortel Telco Systems IXIA XM2 IxNetwork MERS 8600 T5C-XG Telco Systems Spirent T-Marc-254 Foundry TestCenter NetIron XMR 8000 Alcatel-Lucent IXIA XM2 7450 ESS-6 IxNetwork Cambridge VectaStar Cisco 7606 Ceragon ADVA FSP FibeAir IP-MAX2 150CC-825 Telco Systems T-Metro 200 Gaming Client ECI SR9705 RAD Nokia Siemens Networks MTU-s Connection Types hiD 6650 ETX-202A Gigabit Ethernet Cisco RAD Fast Ethernet Huawei ME4500 ETX-202AMetro CX600-4 TDMoNxE1/STM-1 Nokia Siemens Networks Tellabs Flexi WCDMA BTS ATMoNxE1/STM-1 Cisco Telco Systems 8830 Catalyst 3750-ME T-Marc-340 SHDSL Cisco RAD G.SHDSL.bis MTU-s ACE-3200 ME-3400-2CS Wireless Telco Systems Video Source 10 Gbit G709 T-Metro 200 STM-16 Alcatel-Lucent NEC PASOLINK NEO 7705 SAR MTU-s 10 Gbit Eth 10 MHz Clock Nokia Siemens Networks NEC RACEL Rohde & Schwarz PASOLINK NEO TE IXIA XM2 SITLine ETH NEC Network Areas IxNetwork CX2600 Aggregation network RADIPMUX-216/24 Alcatel-Lucent Metro network Calnex 5650 CPAM Access network Paragon Sync Core network UNI E-NNI
Carrier Ethernet World Congress 2008 Multi-Vendor Interoperability Test Clock Synchronization Introduction Precision Time Protocol (PTP); Nokia Siemens Networks’ Flexi WCDMA BTS received the PTP Across all mobile network technologies, clock packets as a slave clock over an E-Line across the synchronization is a topic of interest and major backhaul network. Calnex Solutions connected their concern today. Base stations within a mobile Paragon Sync in between the master and slave operator’s domain require a common clock for three clock, witnessing the protocol exchange and inten- reasons: tionally dropping packets. • General operation and frequency stability. Base stations need to keep their transmit frequencies and time slots very stable to avoid interferences. • Base station hand over. Voice calls shall not drop when the cell phone is moving from one cell’s coverage area to the next. Clock frequencies of the two base stations need to be synchronous to ensure that the phone can continue sending within its pre-assigned mobile network slots nearly uninterruptedly. • Common frequency transmission. In some mobile technologies, adjacent base stations transmit Figure 7: 1588v2 Synchronization using identical frequencies, leading to a large measurement common frequency coverage area and allowing the communication of end systems with multiple base stations at the same time (MIMO). This The master and slave clocks interoperated success- network service requires phase synchronization fully using the subset of IEEE 1588v2 Precision Time of base station clocks to ensure that their signals Protocol required for frequency synchronization — do not extinguish each other. the unidirectional SYNC messages. Via an E1 output The easiest way of providing a common clock is to of the base station and using a Symmetricom use GPS. However, mobile operators do not always software, we examined the frequency accuracy of prefer this solution due to technical or political the base station. Figure 7 shows the first 85 minutes reasons. Sometimes base stations do not have of a clock deviation measurement of the Nokia visibility of the sky (pico cells in buildings, tunnels) or Siemens Networks Flexi WCDMA BTS synchronized the GPS receiver installation would be too via IEEE 1588v2 to the Symmetricom TimeProvider14 cumbersome (femtocells at home). 5000 PTP Grand Master clock. We started the measurement after the first five minutes of device Network Type operation — the amount of time the devices require Synchronization Synchronization for the initial clock synchronization. As shown in the diagram there is a peak of frequency deviation in Frequency the first two minutes of the measurement, and Phase another peak at around 20–30 minutes. In any case, the deviation never exceeded 3.6 ppb (parts-per- billion). The deviation was most often measured at GSM X around 0.6 ppb. The measured results demonstrates the accurate synchronization of the Flexi WCDMA 3G (UMTS, CDMA2000) X BTS, and the test goal to achieve a frequency 4G (LTE including MBMS) X X deviation of below 16 ppb was fulfilled. Although we did not conduct long term measurements as 4G (Mobile WiMAX) X X required in ITU standards due to a lack of time at the hot staging; the same test will be conducted live at CEWC for visitors to witness the clock accuracy Vendors offer a number of network-based clock maintained throughout the conference. synchronization mechanisms. They can be selected For the same reasons, PTP impairments generated by depending on the precision requirements. For the Calnex Paragon Sync did not show visible mobile backhaul, the base station frequency needs effects. The base station has an internal temperature- to be accurate to below 50 ppb (ITU-T G.812). The controlled quartz as a fallback clock when the network clock needs to be three times more accurate incoming IEEE 1588v2 signal is lost. This clock is to reach this goal — 16 ppb (ITU-T G.8261 draft). accurate enough for a couple of days of operation; we lacked the time to wait for it to degrade. Clock Synchronization Test Results Adaptive Clocking. Another solution for some IEEE1588v2. For the first time in a public Mobile frequency synchronization scenarios is adaptive Backhaul test, we verified interoperability of IEEE clocking. In this solution, the clock is regenerated 1588v2 based clock synchronization implementa- from the frequency of bits arriving on an emulated tions at this event. Symmetricom provided a TimePro- TDM circuit. Assuming that the transmitter sends vider 5000 PTP Grand Master implementing the packets at a known rate and precise intervals, the
Ethernet OAMadaptive mode is usually accomplished on slaves by First Mile (EFM), and Connectivity Faulteither measuring packets inter-arrival time or Management (CFM), standardized by the IEEEmonitoring a buffer fill level (some adaptive clock under 802.3ah and 802.1ag respectively, Y.1731,recovery mechanisms may also use timestamps). standardized by ITU-T, and E-LMI, specified by MEF.Adaptive clock works only for constant bit rate Over the past years we have seen a significantservices. increase in support in this area – in the first eventAs described in the "Circuit Emulation Service" four vendors tested their CFM implementations andchapter, the Alcatel-Lucent 9500 MPR, NEC five vendors tested their EFM code. At our currentCX2600, NEC PASOLINK NEO TE, RAD event 12 vendors tested their EFM and CFM imple-IPMUX-216/24, and RAD MiTOP-E1/T1 hosted by mentations.the ETX-202A have successfully demonstrated and Our service provider panel placed a high value ontested adaptive clocking. both Ethernet OAM test areas and with the inclusionIn addition we measured the value of the frequency of both EFM and CFM in the new MEF 20 “Userdeviation as demonstrated between the RAD Network Interface (UNI) Type 2 ImplementationACE-3200 and RAD ACE-3205 over a long Agreement” technical specifications a clearmeasurement time. We connected the Symmetricom continuous need for testing has been establishedTimeCesium 4000 Master Clock to the RADACE-3200, and verified that the frequency accuracywas better than 16 ppb. Link OAMETHERNET OAM Link OAM is the name used by the Metro Ethernet Forum (MEF) to refer to clause 57 of the IEEE 802.3EANTC interoperability events have integrated standards where OAM is defined for Ethernet in theEthernet Operations, Administration and First Mile. The protocol monitors the health andManagement (OAM) testing since 2006. Several operations of the UNI’s physical layer. The MEFdifferent protocols fall under the category of Ethernet requires the usage of Link OAM between the UNI-NOAM. In this event we have tested Ethernet in the and UNI-C starting from UNI type 2.2 and recom- Cisco RAD 7604 Cisco RICi 16 ME-3400-12CS RAD ETX-202A Cisco Catalyst 3750-ME 15 RAD LA-210 RAD MTU-s ADVA FSP 150CC-825 RICi 155GE Actelis Ciena ML658 LE-311v MTU-s Spirent TestCenter Alcatel-Lucent 7450 ESS-6 IXIA XM2 Ceragon IxNetwork FibeAir IP-10 Huawei Telco Systems CX600-4 T-Marc-340 Tellabs 8830 Telco Systems Foundry T5C-24F NetIron XMR 8000 Telco Systems Juniper MTU-s MX480 Juniper Telco Systems T-Marc-250 MX240 T-Marc-380 Discovery followed by Access MTU Tester Loopback Mode Device MTU-s Switch Dying Gasp Messages Aggregation Metro/Core Device Device Figure 8: Ethernet Link OAM test results