Timing over packet demarcation


Published on

There is a pressing need to distribute accurate timing, i.e., frequency and/or Time of Day (ToD), across Packet Switched Networks (PSNs) for applications such as cellular backhaul. This paper reviews the main issues involved in timing over packet (ToP) demarcation and provides best practices for ToP demarcation and performance monitoring.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Timing over packet demarcation

  1. 1. White PaperTiming over PacketDemarcationAn Innovative Approach toSynchronization Performance Monitoringin Packet NetworksAlon Geva, Advanced Technologies ManagerYaakov Stein, Chief ScientistRAD Data Communications The Access Company
  2. 2. AbstractThere is a pressing need to distribute accurate timing, i.e., frequency and/orTime of Day (ToD), across Packet Switched Networks (PSNs) for applicationssuch as cellular backhaul. Highly accurate delivery of timing over PSNs ischallenging, due to their asynchronous nature and the need to overcomeimpediments, such Packet Delay Variation (PDV). This challenge increases foroperators leasing transport services from carriers, as operators have littlecontrol over the timing packets as they traverse the carrier’s network and, asa result, timing recovery accuracy cannot be guaranteed. While Ethernet OAMand performance monitoring tools are available for carriers and wholesaleprovides, no such demarcation functionality is presently defined specificallyfor the timing distribution service.This paper reviews the main issues involved in timing over packet (ToP)demarcation and provides best practices for ToP demarcation andperformance monitoring.
  3. 3. White Paper: Timing over Packet DemarcationContentsIntroduction ........................................................................................................................... 2Legacy Synchronization Performance Monitoring ............................................................ 4Synchronization over Packet Networks using PTP ........................................................... 5PTP Performance Monitoring Today ................................................................................... 7PTP Demarcation - The Future in PTP Performance Monitoring .................................... 9Useful Practices of ToP Demarcation ............................................................................... 12Summary .............................................................................................................................. 15References ........................................................................................................................... 16© 2011 RAD Data Communications Ltd 1
  4. 4. White Paper: Timing over Packet Demarcation Introduction There is a pressing need to distribute accurate timing, by which we mean frequency and/or Time of Day (ToD), across Packet Switched Networks (PSNs) for applications and devices that require it. One of the most demanding cases relates to cellular backhaul. The second and third generations of cellular networks’ base stations require provisioning of frequency with error less than ±50 ppb (parts per billion) compared to the network’s Primary Reference Clock (PRC). Base stations belonging to the next generation of cellular networks, such as the UMTS-TDD and the TD-SCMA, further require provisioning of ToD with error less than 1.5 microseconds compared to Universal Time Coordinated (UTC) (see [1]). Future cellular networks will have even stricter requirements in order to enable new features such as soft handoff between neighboring base stations and Multiple Input Multiple Output (MIMO). ToD accuracy here will be required to be at a level of hundreds of nanoseconds! Delivery of timing of such accuracy over PSNs is challenging, due to their asynchronous nature. In particular, PSNs based on a conventional Ethernet physical layer are not designed with timing distribution in mind, and require additional mechanisms for its distribution. Packet-based timing distribution protocols send packets containing timing information from a timing master, usually referred to as Primary Reference Clock (PRC) or Primary Reference Time Clock (PRTC), to a timing slave located close to the application device requiring the precise timing. The Network Time Protocol (NTP) is the most prevalent synchronization distribution protocol when high accuracy over a well engineered network is required (e.g., for cellular backhaul). A major impediment to the proper function of packet-based timing distribution protocols is the Packet Delay Variation (PDV) that timing packets experience when traveling through the PSN. This PDV is primarily due to the timing packets being queued in network elements along their path. The queuing delay at each network element depends on the momentary traffic load when the timing packet needs to be queued, and thus varies randomly over time. Hence, the overall PDV, being the sum of all the queuing delays, varies non-deterministically. In particular, the aforementioned symmetry assumption cannot hold at any given time, and special filtering mechanisms must be employed to mitigate this effect. Despite these mechanisms, end-to-end frequency and time distribution performance cannot always be guaranteed to meet end-application requirements. 2 © 2011 RAD Data Communications Ltd
  5. 5. White Paper: Timing over Packet DemarcationThe problem becomes more severe when the service provider relies on a carrier’s infrastructure todeliver its traffic to the end application. In such cases, the service provider has little control over thetiming packets as they traverse the carrier’s network and timing recovery accuracy cannot beguaranteed.Service providers continuously monitor Quality of Service (QoS) parameters experienced by user datatraffic in order to ensure conformance to an agreed upon Service Level Agreement (SLA). This isusually accomplished by network demarcation devices running Operation Administration andMaintenance (OAM) protocols (such as ITU-T Recommendation Y.1731 for Ethernet) to monitor theprovided service. OAM functionality may consist of Fault Management (FM) and /or PerformanceManagement (PM) functions. Fault Management consists of defect detection and reportingmechanisms, such as Continuity Check (CC), Connectivity Verification (CV), Forward Defect Indication(FDI), and Backward Defect Indication (BDI). Performance Management involves measurement ofmetrics such as Packet Loss Ratio (PLR), one-way delay (1wD), Packet Delay Variation (PDV),throughput, and availability.OAM mechanisms have been defined for many network and service types, from low-rate TDM links,through SONET/SDH networks, to Ethernet Virtual Connection (EVC) services. However, no suchdemarcation functionality is presently defined specifically for the timing distribution service. As aresult, service providers rely on performance metrics derived from the timing slave to infer the timingservice’s performance. This can be problematic for several reasons. First, timing slaves, employingdifferent timing recovery algorithms, may display different performance, leading to inconsistentresults. Second, timing slaves are typically located at the far edge of the service provider’s network,and thus their performance does not capture the individual behavior of specific segments in theservice provider’s network, but rather a sum of all degradations along the path from master to slave.Third, the timing slave may be embedded in a network element (e.g., a switch or a router), or end-application device (e.g., a cellular base station), and not provide visibility to the performancefunctions that need to be monitored.An alternative that addresses all of the aforementioned shortcomings would be to place astandardized timing slave at, or close to, the required demarcation points in the network, and use itto monitor the performance at the desired locations. However, as will be described in the nextsection, this option would necessitate provisioning a dedicated timing distribution service from themaster to each such location, incurring additional bandwidth and logistics support. Furthermore, thisdedicated timing distribution flow will, in general, experience network impairments different from thetiming flow that it is intended to monitor, introducing an inherent error and inconsistency. © 2011 RAD Data Communications Ltd 3
  6. 6. White Paper: Timing over Packet Demarcation Legacy Synchronization Performance Monitoring Timing distribution, just like any other service delivered over a telecommunication network, needs to be continuously monitored in order to validate its correct behavior and conformance to the required performance level. The on-going migration from synchronous Time Division Multiplexing (TDM) networks, which inherently distribute frequency, to asynchronous Packet Switched Networks (PSNs) introduces major challenges for service providers. Not only must they now employ protocols to distribute frequency, and in many cases ToD, they must also be able to monitor the quality of such timing services. Frequency distribution in synchronous networks is a mature technology that relies on the stringent frequency accuracy requirements of data symbols transmitted over physical links. Frequency information is automatically distributed downstream from a master clock to the end-application on a hop-by-hop basis. Each network element along the path recovers the frequency of the symbols it receives, enslaves a local oscillator to this frequency, and uses this oscillator as its frequency reference for downstream transmission. This results in a timing (or synchronization) chain, consisting of local node clocks connected by physical links. At the top of the timing chain usually stands a highly accurate and stable autonomous clock called the Primary Reference Clock (PRC), usually an autonomous atomic clock. Thus, the frequency at every point along the timing chain is said to be ‘PRC traceable’. Monitoring frequency accuracy and stability at a specific network element may be accomplished by comparing the frequency of its local clock to a trusted frequency reference, such as another stand- alone atomic clock or frequency output from a GPS receiver. The comparison is often carried out by continuous phase comparison of the two signals. The outcome of this continuous phase comparison can then be used to calculate frequency quality metrics, such as the Maximum Time Interval Error (MTIE) and Time Deviation (TDEV) ([2]). Measurement equipment designed for this task is readily available, and is used by service providers to monitor and validate the quality of frequency distribution at various locations within their network, and thus to trigger corrective actions whenever anomalous behavior is detected. A legacy SONET/SDH frequency monitoring system is presented in Figure 1. The monitored clock signal is introduced into a frequency monitoring device (wander analyzer) that is also referenced with an accurate reference clock (i.e. a PRC). The outcome phase error measurement (Time Interval Error - TIE) is offloaded to a PC (or any other metric calculation device) for deriving MTIE and TDEV frequency accuracy/stability metrics. These metrics may then be used to verify the performance of the frequency distribution system and detect abnormalities. 4 © 2011 RAD Data Communications Ltd
  7. 7. White Paper: Timing over Packet Demarcation Figure 1: Typical SONET/SDH frequency monitoring systemSynchronization over Packet Networks using PTPDistribution of timing (frequency and/or time of day) in packet networks is achieved by a timing-over-packet protocol. Such protocols function by sending timing packets between a timing master and atiming slave. The timing master is usually connected to a highly accurate and stable frequency/timereference (PRC or PRTC). The timing packets must be formatted as valid packets for the network, andare transported from master to slave in the same fashion as all packets in the network.When it comes to very accurate distribution of frequency and time over packet networks, the mostprominent timing-over-packet protocol is the Precision Time Protocol (PTP) version 2 (also known asIEEE 1588v2 or IEEE 1588-2008 - [3]). The 2nd version, ratified in 2008, is Telecom oriented in thetrue sense and includes various features and mechanisms making it an attractive apparatus forTelecom applications. © 2011 RAD Data Communications Ltd 5
  8. 8. White Paper: Timing over Packet Demarcation In PTP, timing information is carried by three aspects of the timing packets, namely timestamps carried explicitly in the packets, the precise transmission time, and the precise reception time of the packets (by the master or slave device). The difference between the latter two properties is sensitive to Packet Delay Variation (PDV) that obscures the master-to-slave and slave-to-master propagation latencies, complicating the timing recovery process. Figure 2 depicts the packets transactions for PTP. The transaction is initiated by the PTP master (which is timed by a PRC/PRTC) sending a timing packet to the PTP slave. Timestamp t1 is recorded by the PTP master as the first timing packet passes a well-defined (master) reference point on its way out to the network. Timestamp t2 is recorded by the PTP slave as soon as the first timing packet passes the slave’s reference point after reaching the PTP slave. It should also be noted that t1 is explicitly carried by the first timing packet, and thus becomes known to the PTP slave (if the PTP master is incapable of accurately timestamping the outgoing packet on-the-fly, a follow-up packet containing t1 can be sent). Figure 2: PTP packets transaction In the reverse direction, timestamp t3 is recorded by the PTP slave as the second timing packet passes a well-defined (slave) reference point on its way out to the network, and timestamp t4 is recorded by the PTP master as the second timing packet passes a well-defined (master) reference point on its way in from the network. 6 © 2011 RAD Data Communications Ltd
  9. 9. White Paper: Timing over Packet DemarcationAfter the second timing packet has been received (by the master) and timestamp t4 is recorded, a thirdtiming packet is sent by the PTP master to the PTP slave, informing it with the value of t4. At this timethe PTP slave knows all four timestamps {t1, t2, t3, t4} and is able (under symmetry assumptions) tocalculate the network delay between the master and itself, and thereafter derive the instantaneoustime-error of its local clock. As before, the slave can correct its local clock’s frequency and/or time tomatch that of the master.PTP Performance Monitoring TodaySimilar to what we discussed for TDM networks, in order to monitor timing performance at some pointalong the network path between master and slave, a comparison must be carried out between aphysical timing signal (i.e., a clock signal) recoverable at that point and a trusted timing reference.However, as timing distribution over PSN is not necessarily performed on a hop-by-hop basis,intermediate network elements do not necessarily perform timing recovery and do not necessarilymaintain local clocks. To produce the required physical timing signal at the desired location, themonitoring system itself could perform timing recovery, essentially reproducing the process carried outby the slave. Alternatively, the timestamp information collected at intermediate locations could be usedto directly calculate packet network performance metrics (specifically PDV-based metrics) that could beused to predict the attainable timing performance. The use of such PDV-related metrics is relativelynew, and is presently the subject of intensive research. Several promising metrics such as minTDEV andMAFE (currently planned to be incorporated into the next version of ITU-T Recommendation G.8260([4]) have been proposed and shown to possess desirable properties.Unlike synchronization monitoring in TDM networks that relies on the existing timing flow, producing aphysical timing signal at desired intermediate locations cannot be performed by simply tapping onto thecontent of existing timing over packet flows. One way to overcome this is to introduce a dedicatedtiming packet flow from the timing master to the intermediate location, since existing timing-over-packet protocols require active interaction between master and slave. Such a PTP timing monitoringarrangement is depicted in Figure 3, where the PTP master serves both the PTP slave and the ToPmonitoring device using two separate timing flows. In this scheme, the ToP monitoring device appears,from the timing master’s point-of-view, to be just another PTP slave. The overall timing bandwidthincrease is directly proportional to the number of active PTP monitors installed in the network.Moreover, as separate independent timing flows are used for the PTP slaves and the ToP monitoringdevices, the true PDV experienced by the PTP slaves is not really being monitored. © 2011 RAD Data Communications Ltd 7
  10. 10. White Paper: Timing over Packet Demarcation Figure 3: PTP performance monitoring using an intermediate slave device A second approach would be to use standard network OAM protocols (e.g. ITU-T Y.1731[5]) to monitor network PDV between two demarcation points. Such an approach is presented in Figure 4 where standard OAM packets sent from one MEP (Management Entity Group End-Point), located at one edge of the network, to a seconds MEP, located at the opposite end, are used to measure relevant PDV information and compute the required PDV metrics based on that. The problem is that, here too, the measurements are being carried based on synthetic OAM packets rather than the original PTP flow we want to monitor. Moreover, any Transparent Clock function located between the demarcation points will not be taken into account as it will be completely agnostic to the OAM packets. Figure 4: PTP performance monitoring using standard OAM techniques 8 © 2011 RAD Data Communications Ltd
  11. 11. White Paper: Timing over Packet DemarcationPTP Demarcation - The Future in PTP Performance MonitoringThe new patent pending approach enables monitoring quality and achievable performance of a timingdistribution service in packet networks at any given point in the network, without requiringintroduction of additional traffic.At the heart of the new approach is a network device, hereinafter referred to as a Timing-over-PacketDemarcation Function (ToPDF). The ToPDF contains two network interfaces, to be called the Master-Facing Interface (MFI) and the Slave-Facing Interface (SFI), which enable it to be connected as a“bump on the wire” at any desired location in the network. The network interfaces can be of anytype depending on the specific physical and link layer technologies employed. The ToPDF iscompletely transparent and does not modify monitored packets in any way. Furthermore, the ToPDFdoes not interact with other network elements, and does not generate any additional traffic load onthe network.The ToPDF has the ability to identify incoming and outgoing timing packets (from both its networkinterfaces), to extract any information from these packets, and to record accurate timestamps. Basedon the extracted information and the timestamps, the ToPDF can perform calculations in order toderive the following parameters: • Timing status of the network • Timing metrics that might be useful in predicting the currently achievable timing distribution quality over a given path • Actual recovered frequency and time achieved in this specific location of the networkSuch information can be then used by the service provider to better assess the condition of thetiming distribution in its network.The ToPDF can be placed at any desired location in the network. In some cases the ToPDF would belocated at (or near) a Customer Edge (CE) switch or router, forming a demarcation point between theprovider network and the customer network. In others, it will be placed at the border between twoservice provider networks, or at strategic points inside a single network. © 2011 RAD Data Communications Ltd 9
  12. 12. White Paper: Timing over Packet Demarcation We further assume that one or more timing packet flows traversing the ToPDF in both directions (going from master to slave and back). The ToPDF is able to detect the dedicated timing packets that traverse it in both directions, but does not alter them in any way (and certainly does not alter non- timing packets). However, the ToPDF performs the following non-intrusive operations on timing packets: • Extract information from the timing packets, in particular, flow identifiers and timestamps • Timestamp the timing packets as they traverse well-defined ingress/egress reference points, based on its local clock It should be noted that such non-intrusive detection, data-extraction and timestamping mandates the use of dedicated hardware processing, in order to minimize the additional delay introduced by the ToPDF and make it completely fixed and symmetrical (This requirement can be easily accomplished for the PTP case, using an End-to-End Transparent Clock). In the following discussion, a “timing flow” will mean any persistent flow between a timing master and a timing slave that distributes timing information from master to slave. Moreover, a “raw timestamp set” will mean a specific combination of four timestamps {t1, t2, t3, t4} that contains the complete timing information exchanged between a given timing master and a given timing slave. Figure 5 presents the ToP demarcation concept with a PTP distribution system for a typical cellular backhaul scenario where the cellular service provider is buying wholesale transport services from a local carrier. In this example, the PTP master is part of the carrier network (part of the wholesale service) and the PTP slave is located at the base station (part of the customer network). The ToPDF is embedded in the carrier network’s PE router that delimits the two networks, with access to the PTP distribution timing flows. Thus, it can identify a specific timing flow and generate its own local timestamps (based on its local clock) referenced to some pre-defined timestamping reference point on its MFI. Thus, when the entire packet transaction between the PTP master and PTP slave is over and the slave has the required set of timestamps {t1, t2, t3, t4} to calculate instantaneous time-error between itself and the master, the ToPDF will hold the required set of timestamps {t1, t’2, t’3, t4} to calculate instantaneous time-error between itself and the master. Additionally, using this set of timestamps, the ToPDF can now derive all the required 1-way/2-way network delay and PDV metrics. 10 © 2011 RAD Data Communications Ltd
  13. 13. White Paper: Timing over Packet Demarcation Figure 5: PTP ToP demarcationThe ToPDF will also easily handle any intermediate TC nodes by looking at the Correction Field (CF) inthe header of the traversing PTP packets. As described in Figure 6, by extracting the CF values from allthe PTP messages comprising a complete PTP transaction (Sync, Delay Request and Delay Response),the ToPDF has all the information it needs to take the TC delay compensation for both themaster ToPDF ( ) and ToPDF master ( ) network paths. © 2011 RAD Data Communications Ltd 11
  14. 14. White Paper: Timing over Packet Demarcation Figure 6: PTP ToP demarcation with TCs Useful Practices of ToP Demarcation In many cases, end-to-end PTP frequency and time distribution is the only real possibility for the near and interim future, resulting in a PDV-sensitive solution whose bottom line performance cannot be always guaranteed. This poses a nasty ‘headache’ for the service provider: how can the time/frequency distribution service be used in such a way that would guarantee the required level of performance? More importantly, how can we monitor the service in order to be able to warn that the service has been degraded, isolate problems and tell where in the network the degradation occurred? The answer to these questions might seem difficult at first glance. Clearly, as was already discussed, packet timing flows cannot be naturally monitored by legacy time/frequency-error monitoring devices. Moreover, with standard end-to-end PTP distribution, only the slave device at the end of the distribution chain can be monitored, leaving the entire network section under a thick curtain of uncertainty, and in many other cases even that is not true in reality (for example, when the cellular base station does not offer any external monitoring/probing capabilities for the integrated PTP slave). 12 © 2011 RAD Data Communications Ltd
  15. 15. White Paper: Timing over Packet DemarcationThe good news is that using the new ToP demarcation concept the above questions can be mappedinto the following almost equivalent one: Where in our network should we deploy ToP demarcationentities in order to get: a. Best coverage of the network (the timing distributing part) b. Optimal dissection of the network for best detection of fault conditionsUnderstandably, ToP demarcation entities may be located in many different network locations alongthe timing flow(s) distribution path and, indeed, such practice may be extremely useful in order toisolate timing problems. For example, Figure 7 depicts a PTP timing flow (from a PTP master locatedat the 1st provider network to a PTP slave located in the customer network) that traverses threenetworks: The first network is operated by provider #1, the second network is operated by provider#2, and the third network is operated by the customer. By placing ToPDFs at the (two) demarcationpoints of these networks, as well as another ToPDF at the edge of the customer network (its SFIdirectly connected to the ‘First-Mile’ link), one can estimate the contribution of each of the networksto the timing distribution degradation, and in particular one can isolate the source of majorperformance degradations.The last ToPDF, located close to the end-application device that comprises the PTP slave, experiencesPDV similar to the PTP slave and thus should provide a reliable estimate of the overall end-to-endtiming recovery performance. This can be exploited by the cellular service-provider to appreciate thecurrent condition of the PTP slave even when it is embedded in the end-application device (cellularbase station in this case) and its status/recovery outcome is not directly accessible.The second ToPDF, located at the demarcation point between the provider networks and thecustomer network, experiences the PDV introduced by the provider networks only. Bidirectional PDVmetrics computable by ToPDF2, at this point, can be used by the providers to ensure the quality ofservice they provide, and by the customer to monitor the service it is being provided.The first ToPDF, located at the demarcation between the first and second provider networks,experiences PDV introduced by provider #1’s network only. Bidirectional PDV metrics computable byToPDF1, at this point, can be used by the two providers to predict the timing performance that wouldhave been attained by a PTP slave served only by the first provider. © 2011 RAD Data Communications Ltd 13
  16. 16. White Paper: Timing over Packet Demarcation Figure 7: Multiple ToP demarcation entities along a synchronization distribution chain The PTP performance monitoring approach described here is only one possible practice out of many that could be used to closely monitor PTP time/frequency distribution. Clearly, the selected approach needs to match the underlying network and timing distribution architectures. The appealing thing about the ToP demarcation technology is its high adaptation capability to almost any given network device, from large aggregation switches down to SFPs (see Figure 8), as well as to any timing distribution topology. All that is really needed is to connect the network- element/module containing the ToPDF to a ‘bump on the wire’ on the desired network link. Figure 8: ToPDF elements 14 © 2011 RAD Data Communications Ltd
  17. 17. White Paper: Timing over Packet Demarcation From this point on, the ToPDF will continuously monitor the selected timing flows, sending its findings and collected data to a central NMS/PM portal that will synchronize, integrate and analyze all collected data (from all the ToPDFs), and supplying the user with concurrent, on-going and comprehensive information about the synchronization distribution status in its network (Figure 9). Figure 9: Multiple ToP demarcation entities along a synchronization distribution chainSummaryRAD has developed an innovative PTP performance monitoring technique allowing carriers and serviceprovides to closely monitor the Timing over Packet distribution in their networks, making sure it isworking well and supplying the required level of performance.Contrary to existing solutions that are only end-to-end, RAD’s PTP PM solution can provide deepinsight to the contribution of each network segment to the overall service degradation, allowing theuser to quickly isolate problems and take proactive measures to solve them. For more information,please contact market@rad.com. © 2011 RAD Data Communications Ltd 15
  18. 18. White Paper: Timing over Packet Demarcation References [1] 3GPP TS 25.402 version 5.2.0 Release 5, [2] ITU-T Recommendation G.810 (08/96), definitions and terminology for synchronization networks. [3] IEEE 1588-2008 (July 2008), IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems. [4] ITU-T Recommendation G.8260 (08/2010), Definitions and terminology for synchronization in packet networks. [5] ITU-T Recommendation Y.1731 (07/2011), OAM functions and mechanisms for Ethernet based networks. 16 © 2011 RAD Data Communications Ltd
  19. 19. www.rad.com International Headquarters North America Headquarters RAD Data Communications Ltd. RAD Data Communications Inc. 24 Raoul Wallenberg St. 900 Corporate Drive Tel Aviv 69719 Israel Mahwah, NJ 07430 USA Tel: 972-3-6458181 Tel: (201) 529-1100, Fax: 972-3-6498250 Toll free: 1-800-444-7234 E-mail: market@rad.com Fax: (201) 529-5777 www.rad.com E-mail: market@radusa.com www.radusa.com The RAD name and logo is a registered trademark of RAD Data Communications Ltd. © 2011 RAD Data Communications Ltd. All rights reserved. Subject to change without notice. CatalogThe Access Company no. 802478 Version 11/2011