SlideShare a Scribd company logo
1 of 21
Download to read offline
DesignCon 2016
Are There Any Rules of Thumb When
It Comes to 100Gb/S Board Design?
A Walkthrough from Physical Domain
to Channel Operating Margin (COM)
Testing.
Jacov Brener, Marvell Israel Ltd
jacovb@marvell.com
Liav Ben-Artsi, Marvell Israel Ltd
liav@marvell.com
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 2 of 21
Abstract
The 4x25Gb/s design presents significant challenges, one of which is channel qualification; therefore the
COM was introduced [1]. COM is the outcome of an algorithm which purpose is to evaluate the channel
and its ability to provide a good enough media for the 100Gb/s link. Although COM is adequate for
channel evaluation it doesn't provide any guidance as of how to get the design there [2] .Looking at the
past, engineers referred to their knowhow and rules of thumb, but do these rules of thumb apply now?
Our goal is to provide examples of design guidelines for the case above and emphasize the limitations in
which these guidelines hold. This work is not intended to substitute the COM algorithm, rather save time
and effort on the road to a compatible interconnect.
Author’s Biographies
Jacov is a Senior Signal Integrity Engineer at Marvell Israel Ltd, starting this year. For the previous 10
years Jacov has worked in Intel's Communication and Storage Infrastructure Group, most of the time as
package designer and signal/power integrity focal point in a high-speed PHY team.
Liav is a Senior Signal Integrity Manager at Marvell Israel Ltd. Liav has worked at Marvell for the last
15 years, most of the time as a signal integrity engineer, focusing on signal integrity methodologies,
guidelines, test plan definitions and training. He holds several patents in the field as well as several
others pending. Liav took an active part in IEEE802.3bj standard committee.
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 3 of 21
Table of Contents
1. Introduction to COM............................................................................................................................... 4
1.1. The algorithm and the procedure..................................................................................................... 4
1.2. Reference system channel................................................................................................................ 4
1.3. DUT host board................................................................................................................................ 6
1.4. Work scope ...................................................................................................................................... 7
2. Minimizing reflections to increase COM ............................................................................................... 8
2.1. The impact of Si termination ........................................................................................................... 8
2.2. Balancing host board card impedance discontinuities..................................................................... 9
3. Key guidelines to lower Xtalk in a high-density board ........................................................................ 13
3.1. Via-to-via FEXT............................................................................................................................ 13
3.2. NEXT due to reference plane resonance ....................................................................................... 17
4. The importance of correct de-skewing and its impact on MC.............................................................. 18
5. Summary and Conclusion..................................................................................................................... 20
Table of Figures
Figure 1: Frequency domain properties of cable assembly test with case #1 (a) package and case #2 package (b)..................5
Figure 2: DUT host board schematic drawing............................................................................................................................6
Figure 3: Frequency domain properties of the host board for TX lane (a) and RX lane (b).......................................................7
Figure 4: COM dependence on TX & RX load impedances ........................................................................................................8
Figure 5: Via structure #1 (a) and vias structure #2 (b) ...........................................................................................................10
Figure 6: Differential Return Loss (RLdiff) of via structure #1 and structure #2 .....................................................................10
Figure 7: Differential Impedance (Zdiff) of via structure #1 and structure #2 .........................................................................10
Figure 8: RLdiff of systems with structures #1 and structures #2 .............................................................................................11
Figure 9: Differential Insertion Loss (ILdiff) of systems with structures #1 and structures #2.................................................11
Figure 10: SBR response & Equalization effect of systems with structures #1 and structures #2 ............................................12
Figure 11: SBR Bathtub curves of systems with structures #1 and structures #2......................................................................12
Figure 12: Typical via placement..............................................................................................................................................14
Figure 13: E-filed of via structure without shield vias (a) and with them (b)............................................................................14
Figure 14: Xtalk of via structure without shield vias and with them .........................................................................................15
Figure 15: FEXT of systems with via structures without shield vias and with them..................................................................15
Figure 16: Xtalk SBR of systems with via structures without shield vias and with them...........................................................16
Figure 17: Bathtub curves of Xtalk only (a) and Xtalk with ISI effect (b) of systems with via structures without shield vias and
with them....................................................................................................................................................................................16
Figure 18: E-filed of via structure without terminating vias (a) and with them (b) ..................................................................17
Figure 19: E-filed of via structure without terminating vias on PCB edge ...............................................................................17
Figure 20: Xtalk of via structure without terminating vias and with them ................................................................................18
Figure 21: De-skew technique #1 (a) and de-skew technique #2 (b).........................................................................................19
Figure 22: MC of a trace with de-skew technique #1 and de-skew technique #2......................................................................19
Figure 23: MC of a system with de-skew technique #1 and de-skew technique #2 ...................................................................19
Table 1: COM of systems with structures #1 and structures #2 ................................................................................................12
Table 2: COM of systems with via structures without shield vias and with them......................................................................16
Table 3: COM of systems without terminating vias and with them ...........................................................................................18
Table 4: COM of system in the initial and optimized phases.....................................................................................................20
Table 5: Rules-of-Thumb for mitigation of different types of signal distortion .........................................................................20
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 4 of 21
1. Introduction to COM
1.1. The algorithm and the procedure
The 4x25Gb/s design presents significant challenges to today's technology and was defined by IEEE
802.3bj working group. Within this group several new methodologies were developed for the task. One
of the methodologies is called COM which stands for Channel Operating Margin.
COM is the outcome of a time domain algorithm, provided within the specification as a set of equations
and procedures, which purpose is to evaluate the quality of the channel and its ability to provide a good
enough media for the 100Gb/s link. Basically the COM will report how much margin in dB a channel
has to perform well given a reference transmitter and receiver [3].
COM algorithm implementation example [4] was also provided by members of the same working group
and was used in this work in order to evaluate the host board (sometime referred to as line card)
performance along with a specific reference system channel.
In order to use COM one must have channel response (THRU) and may have additional crosstalk
(Xtalk) profiles (all as Touchstone S-parameters model). Once those are in place some algorithm setting
may be adjusted and channel evaluation may begin. The procedure in which COM operates can be
summarized in 3 basic steps:
1. TX (transmitter) & RX (receiver) equalization: TX Finite Impulse Response (FIR) filter, RX
Continuous Time Linear Equalizer (CTLE) , RX Decision Feedback Equalizer (DFE)
2. Xtalk addition: Far-End Xtalk (FEXT), Near-End Xtalk (NEXT)
3. Result analysis: frequency domain plots, pulse responses, bathtubs, COM
Once the COM is given it’s the final and only figure of merit to be used to evaluate a specific channel’s
performance, and this figure of merit will be used as a measure to evaluate out test cases’ result.
1.2. Reference system channel
In order to evaluate the quality of a host board a complete end-to-end system is needed, therefore a
reference channel such as a backplane or a cable should be taken along with the host board model. The
reference system channel for our work was chosen to be a 5m 24AWG cable connected to a zQuad
Small Form-factor Pluggable Plus (zQSFP+) connector a.k.a. cable assembly [5].
Testing the above cable assembly to 100GBASE-CR4 specification with a reference analytical host
board model yields COM=4.3dB (for a worst case reference package model, also provided as part of the
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 5 of 21
COM equations) while the pass/fail criteria is COM≥3dB [6]. Figure 1 shows the frequency domain
properties of the cable assembly concatenated with a reference board model and 2 cases of packages:
1. Case #1: package trace length of 12mm
2. Case #2: package trace length of 30mm (was the worst case package in our case)
(a) (b)
Figure 1: Frequency domain properties of cable assembly test with case #1 (a) package and case #2 package (b)
As can be seen this cable assembly introduces very high -30dB loss and -54dB Xtalk at Fbaud/2 to the
system, which yield an Insertion loss to Crosstalk Ratio (ICR) of 24dB at Fbaud/2.
Figure 1 and COM results clearly show that this kind of cable assembly stresses the receiver
significantly even without introducing a real host board instead of the analytical model.
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 6 of 21
1.3. DUT host board
The role of the host board is to connect the Integrated Circuit (IC) chip transmitting/receiving 4x25Gb/s
signal to the zQSFP+ connector. Figure 2 describes the schematic drawing of the board.
Figure 2: DUT host board schematic drawing
The board was designed with Panasonic Megtron6 dielectric and Hyper Very Low Profile (HVLP)
copper. The traces connecting the Ball Grid Array (BGA) footprint of the IC to the zQSFP+ connector
are 5in long, running on inner layers (striplines) in order to reduce FEXT.
Board model was constructed using cascaded models of its components. The components were extracted
using a 3D full wave commercial filed solver. Figure 3 shows the frequency domain properties of the
modeled board, both for TX and RX ends. From the plots it can be seen that the board’s loss at Fbaud/2 is
~-7.3dB which is exactly the full budget allocated for it by the specification [6] so this board should
stress the cable assembly test to its maximum loss wise.
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 7 of 21
(a) (b)
Figure 3: Frequency domain properties of the host board for TX lane (a) and RX lane (b)
1.4. Work scope
Loss has long been considered as a major factor that influences the system’s performance [7].
Reflections and Xtalk are also recognized as types of distortion yet the guide lines to minimize them
usually include trace impedance and trace to trace separation only. Mode Conversion (MC) is another
type of noise which looks relatively easy to handle yet sometimes it could be misleading.
In our work we’ll focus on finding new design rules and rules of thumb in order to minimize these
causes of signal distortion. Our assumption is that a common designer engineer is well familiar with the
loss mechanisms of the system and is capable to minimize it, so our drive will be to help him in other
aspects of the host board design.
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 8 of 21
2. Minimizing reflections to increase COM
2.1. The impact of Si termination
Silicon (Si) or on-die termination has been here for ages [8] and its value for high-speed signals is
traditionally 50Ω single-ended or 100Ω differential. Often a question arises what is the optimized
impedance for a high-speed serial link. In theory, if all channel’s components are tuned to differential
impedance of 100Ω and no significant loss is present clearly the best value for the Si termination from
signal integrity point of view is 50Ω single-ended. However in practice not all channel components
reach 100Ω target and there’s significant loss introduced in the channel, for those reasons 50Ω single-
ended impedance at TX&RX might not be the best case.
In order to answer the question above our reference system channel was tested against COM in various
TX&RX termination values in a cable assembly test as in 1.2. The results are plotted in Figure 4.
Figure 4: COM dependence on TX & RX load impedances
Notice, that the optimum values for this channel are 35Ω for TX and 55Ω for RX. Those values are the
result of equilibrium between the reflection mechanism which drives the impedances towards 100Ω and
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 9 of 21
the loss mechanism which pushes the TX termination down and RX termination up according to simple
voltage division.
In any case, 50Ω Si termination isn’t the optimized value for any channel and cannot be taken as a rule
of thumb for any given Si nor should it be taken as a reference impedance for any channel.
2.2. Balancing host board card impedance discontinuities
Matching vias, capacitor pads and other impedance discontinuities is critical in order to reduce
reflections in time domain or Return Loss (RL) and Insertion Loss Deviation (ILD) in frequency
domain. Those channel parameters impact the signal quality as seen at the RX end directly. This has
been known for a long time. Engineer’s target was always to design as much as transparent interconnect
as possible. The important question that needs to be answered in this procedure is “transparent in what
way?”
Common knowledge, also written in some references [9], suggests trying to match those discontinuities
to 100Ω differential impedance, probably since it’s the common reference impedance of the TX & RX
as well as a common system target impedance. Although in 2.1 we have seen that this is not always the
case, nor it’s the optimal case, let’s assume that the TX & RX terminations are indeed 100Ω. Even
though the terminations are 100Ω, the system itself will most likely not have a flat 100Ω profile.
Connectors, BGA and other mechanical components cannot be perfectly matched to 100Ω and they
usually have more capacitive nature thus the “effective impedance” of the system is usually lower than
100Ω.
Let’s examine 2 cases of via structures and see how they perform in this kind of system. Figure 5 shows
the mechanical drawing of these 2 cases. As can be seen the signal-signal and signal-GND distances in
structure #1 are greater than structure #2 so we’d expect structure #1 to be less capacitive and more
inductive in nature thus to have higher impedance compared to structure #2. As mentioned before, it is
common that vias have a differential impedance less than 100Ω, therefore our expectation is that
structure #1 will have a somewhat lower impedance then 100Ωs and structure #2 will have even lower
impedance than that.
Figure 6 shows the differential RL of the 2 vias compared to 100Ω reference impedance. It can be
clearly seen that in this setup structure #1 behaves in a better way then structure #2.
Going back to our previous discussion regarding “effective impedance”, let’s assume that these vias are
placed somewhere near the BGA area so it “sees” the board+system(cable)+board+BGA on one side
and BGA at the other side. Let’s also guess that the “effective impedance” of the
board+system(cable)+board+BGA side 90Ω and the BGA’s side only is 80Ωs. This assumption is
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 10 of 21
usually a good initial guess since Si capacitance along with package discontinuities lower the effective
impedance by a great deal while long board traces and system traces or cables hold the controlled
impedance which is usually closer to 100ohm.
(a) (b)
Figure 5: Via structure #1 (a) and vias structure #2 (b)
Figure 6: Differential Return Loss (RLdiff) of via structure #1 and structure #2
Figure 7: Differential Impedance (Zdiff) of via structure #1 and structure #2
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 11 of 21
Figure 7 shows the Time-Domain Reflectometry (TDR) impedance for this case. Here we can conclude
the opposite regarding which structure is better compared to the conclusion that was derived from Figure
6. It’s clear that structure #2 provides a smoother transition between impedances then structure #1 since
its impedance is “located” just between the 2 “effective impedances” we assumed. This smother
transition is what should make the via more transparent.
So which structure is better #1 based on Figure 6 or #2 based on Figure 7? Figure 8 gives a thick hint to
the answer for this question. The RL of the whole system with the host boards and cable assembly with
structure #2 gives better RL especially up to 10GHz which is within the critical bandwidth for our
25Gbps design. Note that in both cases the IL stays almost the same up to a very high frequency as can
be seen in Figure 9, but when looking a bit more at the IL one can notice that structure #2 has a smother
frequency domain behavior as well as slighter lower loss.
Figure 8: RLdiff of systems with structures #1 and structures #2
Figure 9: Differential Insertion Loss (ILdiff) of systems with structures #1 and structures #2
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 12 of 21
The final answer is of course supplied by the COM and summarized in Table 1. For the given system via
structure #2, placed in multiple places on the host board, increased the COM by 0.75dB taking it above
the PASS/FAIL criteria! Recalling the COM result of the cable assembly itself with a reference
analytical host board, an actual real board routing with via structure #2 decreased the COM by a mere
0.65dB.
Systems with COM [dB]
Structures #1 2.9 (FAIL)
Structures #2 3.65 (PASS)
Table 1: COM of systems with structures #1 and structures #2
The reasons for the COM difference between the 2 systems can be seen in Figure 10 and Figure 11.
Structure #2 is more transparent thus system’s RL is better. The main tap of the single bit response
(SBR) is higher since the channel is less reflective. At the end it leads to the fact that structure #2 opens
the bathtub by ~20% compared to structure #1!
Figure 10: SBR response & Equalization effect of systems with structures #1 and structures #2
Figure 11: SBR Bathtub curves of systems with structures #1 and structures #2
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 13 of 21
All the figures of merit along the way led us to think that structure #2 is better except the RL in Figure 6.
This is due to the fact that the RL was plotted with reference to the standard 100Ω ports to not to the real
“effective impedance” of the system.
Of course each system is different and has its own “effective characteristic impedance” yet in general
the following design guide lines can be derived:
1. Usually the “effective impedance” of the system is lower than 100Ω, and that was our case also
2. Match the impedance discontinuities to the “effective impedance” of the system seen form each
side of the impedance discontinuity, not necessarily 100Ω as indicated in former rules of thumb
3. In today’s technology almost minimum distances of signal-signal and signal-ground (keeping 5-
10mil additional spacing) yielded the best results for our tested case
3. Key guidelines to lower Xtalk in a high-density board
3.1. Via-to-via FEXT
Xtalk between vias has been a challenge for a long time and has been discussed extensively such as in
[10]. The importance of via-to-via Xtalk increases as we proceed to 25Gbps signals. This happens due to
the fact that trace-to-trace Xtalk is no longer the most significant limitation as we go from µ-strip (outer
layers traces) to strip-line (inner layer traces), in which far-end Xtalk is significantly lower. And so, the
role of via-to-via Xtalk becomes more significant.
Just as in the trace-to-trace Xtalk case, via-to-via Xtalk is usually about FEXT. Common practice is to
route TX & RX traces on different layers so they run in parallel as we look at their X-section. Due to
this practice trace-to-trace NEXT is usually negligible (since those signals have a solid reference plane
between them) yet FEXT is needed to be accounted for. Moreover, when signal need to change layers,
and vias has a wider footprint compared to the overall X-section of the differential pair then these via
structures are placed perpendicular to the direction of the traces. This practice can be seen in Figure 12.
Once again, as in the trace case, it’s usually better to separate between TX & RX vias so assuming each
via’s nearest neighbors are of the same type we get a row of TX or RX vias, making the via Xtalk a
FEXT issue. However this is not yet the most challenging case. When dealing with a high lane count i.e.
a high density board the distance between these adjacent rows is rapidly decreasing. As can be seen in
Figure 12, although the distance between the rows is larger than the one between vias in the same row,
they don’t have the reference via to act as guards and defend them from one another.
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 14 of 21
Figure 12: Typical via placement
A proof for this concern can be found in the E-field distribution shown in Figure 13 (a). As can be seen,
a very strong field exists around the signal vias which penetrates to the adjacent row much easier then to
the adjacent via at the same row. Note, that even the farthest via on the adjacent row in this case suffers
from more Xtalk then the closet via on the same row, since the latter is separated from the aggressor
with a reference via.
(a) (b)
Figure 13: E-filed of via structure without shield vias (a) and with them (b)
The solution to this phenomenon is to stitch a shield via row between the signals’s via rows so the Xtalk
won’t be able to penetrate to these victim vias. Figure 13 (b) shows the effects of the added shielding
vias. Note that the E-field has dropped by ~40dB between the vias and now all the vias has a fair
immunity to Xtalk.
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 15 of 21
The degradation of FEXT in vias due to those shielding vias is ~10dB at Fbaud/2 as plotted in Figure 14
and the overall effect on the system FEXT is 4-8dB through the whole spectrum of significance as
plotted in Figure 15. This effect has of course a profound impact on the performance by decreasing the
FEXT SBR peak from 0.5mV to 0.3mV, as depicted on Figure 16, which is a ~40% improvement!
Figure 14: Xtalk of via structure without shield vias and with them
Figure 15: FEXT of systems with via structures without shield vias and with them
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 16 of 21
Figure 16: Xtalk SBR of systems with via structures without shield vias and with them
Furthermore as we look at system level performance, the bathtub curves depicted on Figure 17 show that
eye opening only due to Xtalk is improved by ~15% while a combination of Xtalk and Inter Symbol
Interference (ISI) improved by ~23%.
(a) (b)
Figure 17: Bathtub curves of Xtalk only (a) and Xtalk with ISI effect (b) of systems with via structures without shield vias and
with them
Finally as we look at the COM result, it becomes clear: The shielding vias are a must. COM increases by
0.25dB only due to placing shielding vias.
Systems with via structures COM [dB]
without shield vias 3.1
with shield vias 3.35
Table 2: COM of systems with via structures without shield vias and with them
A general recommendation therefore would be not to base an expectation of Xtalk purely on the distance
and symmetry of the vias.
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 17 of 21
3.2. NEXT due to reference plane resonance
Plane resonance due to unterminated power/ground plane is not new [11] yet till now it was relatively
small. Plane resonance occurs when a signal reaches the end of a plane, bounces of it and propagates out
of the edge to the cavity. With the bitrate increase, leading to shorter wave length effects, and the
decrease of noise margins its impact on a high-speed signal cannot be neglected any longer. This is
especially true in the case of noise source as NEXT which has to be very low in order to maintain good
enough Signal-to-Noise Ratio (SNR).
Consider a case where a structure of TX vias is placed near a similar structure of RX vias, while both
structure are placed the edge of a reference plane. Figure 18 (a) shows the E-field going out of the left
TX via pair to the right RX via pair. Note the high E-field near the aggressor via but also the same order
of magnitude of the field near the edges of the reference plane. Figure 18 (b) shows the same structure
and the E-field outcome yet in this case terminating vias are placed and stitch adjacent reference planes.
It can be clearly seen the field drops dramatically at the whole cavity, especially at the plane edges.
Figure 19 examines the same phenomena on the edge of a Printed Circuit Board (PCB) plane and
reveals the same nature of the E-filed on the edge, just in a smaller magnitude.
(a) (b)
Figure 18: E-filed of via structure without terminating vias (a) and with them (b)
Figure 19: E-filed of via structure without terminating vias on PCB plane edge
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 18 of 21
The tremendous impact on NEXT is plotted in Figure 20 which predicts ~8dB impact on the Xtalk, only
from this tiny structure! Furthermore, when adding these additional NEXT sources to our previously
optimized channel, COM decreases by 0.1dB only due to this small section as shown in Table 3.
Although 0.1dB decrease in COM may look negligible to some designers, this performance degradation
can easily be avoided by stitching the edge of the reference plain with terminating vias at distance of
125mil and below, which corresponds to quarter wave length of Fbaud/2 in a typical dielectric used for
25Gbps applications.
Figure 20: Xtalk of via structure without terminating vias and with them
Systems type COM [dB]
without terminating vias 3.15
with terminating vias 3.25
Table 3: COM of systems without terminating vias and with them
4. The importance of correct de-skewing and its impact on MC
The impact of skew in channel delay has been discussed extensively [12] as well as also its impact on
system level performance [13], however the best way to de-skew signal’s path has remained
controversial. One of the great thumb rules states “de-skew where skew happens” yet it’s not clear
where exactly skew happens and how much.
Consider 2 de-skewing techniques as shown in Figure 21. From first glance both of them look alike and
follow the rule of thumb above, by de-skewing the total trace length where it begins. In second look, one
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 19 of 21
can notice, that the skew occurs in 2 locations – large skew at the left and small skew at the right of the
trace edges, just near the pins/vias.
(a)
(b)
Figure 21: De-skew technique #1 (a) and de-skew technique #2 (b)
The difference in the MC between these 2 de-skewing schemes is shown in Figure 22. Clearly the 2nd
de-skewing method (b) is better and decreases the MC by 2-3dB.
Figure 22: MC of a trace with de-skew technique #1 and de-skew technique #2
The overall impact of applying technique #2 on the entire system instead of technique #1 on the total
system’s MC is depicted in Figure 23 and it is 6-8dB.
Figure 23: MC of a system with de-skew technique #1 and de-skew technique #2
Once again, similar to 3.2, this performance degradation can be easily avoided by proper analysis of
“where and by how much skew happens”.
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 20 of 21
5. Summary and Conclusion
In our work we have walked the path from physical design, through frequency-domain modeling
towards final COM verdicts. We have shown that sometimes old rules of thumb don’t work, moreover in
the 25Gbps case they will seldom succeed to work. New insights on system’s impedance and its impact
on COM have been raised as we consistently compared physical structures using COM.
Different chip impedance impact channel performance substantially and 100Ω termination on either side
is not the optimal value for all systems. Different chip impedance impacts the “effective impedance” of
the system substantially, so rules of thumb are only good in some well-defined impedance environment.
Proper via treatment can increase system’s COM by ~1dB as shown in Table 4, 0.75dB out of it by via
matching to the “effective impedance” and 0.25dB by reference via shielding. These steps may avoid
system’s failure just as was done in our case of a 5m cable.
System design phase COM [dB]
Initial 2.35 (FAIL)
Optimized 3.35 (PASS)
Table 4: COM of system in the initial and optimized phases
Simple-to-implement guide lines, such as reference plane termination and proper de-skewing, can
prevent unnecessary system performance degradation.
In summary, our work indicated simple rules of thumb that worked and improved our 25Gbps system.
Those are listed in Table 5
Distortion type Mitigation Rule-of-Thumb
RL Decrease distances of signal-signal and signal-ground vias almost to minimum
Xtalk
Shield signal vias from all sides, yet bit farther then signal-ground distance
Stich termination vias at the end of reference planes at distance ≤120mil
MC De-skew where skew happens by the same amount it happens at the location
Table 5: Rules-of-Thumb for mitigation of different types of signal distortion
Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16
Page 21 of 21
References
[1] R. Mellitz, A. Ran, M. P. Li and V. Ragavassamy, "Channel Operating Margin (COM): Evolution
of Channel Specifications for 25 Gbps and Beyond," in DesignCon, 2013.
[2] X. Dong, M. Mo, F. Rao, W. Jin and G. Zhang, "Relating COM to Familiar S-Parameter Parametric
to Assist 25Gbps System Design," in DesignCon, 2014.
[3] M. Brown, M. Dudek, A. Healey, L. Ben Artsi, R. Mellitz, C. Moore, A. Ran and P. Zivny, "The
state of IEEE 802.3bj 100 Gb/s Backplane Ethernet," in DesignCon, 2014.
[4] A. Ran, "COM Update," 6 November 2014. [Online]. Available:
http://www.ieee802.org/3/bj/public/tools/ran_com_3bj_3bm_01_1114.zip.
[5] M. Shanbhag, "QSFP 5 m 24AWG Cable Assembly," 16 January 2014. [Online]. Available:
http://www.ieee802.org/3/100GCU/public/ChannelData/shanbhag_3bj_01_0114.zip.
[6] IEEE, "802.3bj Clause 92," 2014. [Online]. Available:
https://standards.ieee.org/findstds/standard/802.3bj-2014.html.
[7] H. Johnson and M. Graham, High-Speed Signal Propogation: Advanced Black Magic, 1st ed.,
Prentice Hall, 2003.
[8] H. Johnson and M. Graham, High speed digital design: a Handbook of black magic, 1st ed.,
Prentice Hall, 1993, pp. 223-248.
[9] E. Bogatin, Signal and Power Integrity – Simplified, 2nd ed., Prentice Hall, 2009, p. 401.
[10] G. Blando, J. Miller, D. Winterberg and I. Novak, "Cross Talk in Via Pin Fields, Including the
Impact of Power Distribution Structures," in DesignCon, 2009.
[11] V. S. Pandit, W. H. Ryu and M. J. Choi, Power Integrity for I/O Interfaces: With Signal
Integrity/Power Integrity Co-Design, 1st ed., Prentice Hall, 2010, pp. 238-247.
[12] E. Kunz, J. Y. Choi, V. Kunda, L. Kocubinski, Y. Li, J. Miller, G. Blando and I. Novak, "Sources
and Compensation of Skew in Single-Ended and Differential Interconnects," in DesignCon, 2014.
[13] M. Rowlands, P. Patel and P. Casher, "System Performance as a Function of Common Mode
Metrics," in DesignCon, 2012.

More Related Content

Similar to Paper are thereanyrulesofthumb_brener

Chen Zeng (201375033)
Chen Zeng (201375033)Chen Zeng (201375033)
Chen Zeng (201375033)Chen Zeng
 
Tree structured partitioning into transform blocks and units and interpicture...
Tree structured partitioning into transform blocks and units and interpicture...Tree structured partitioning into transform blocks and units and interpicture...
Tree structured partitioning into transform blocks and units and interpicture...LainAcarolu
 
EELE 5331 Digital ASIC DesignLab ManualDr. Yushi Zhou.docx
EELE 5331 Digital ASIC DesignLab ManualDr. Yushi Zhou.docxEELE 5331 Digital ASIC DesignLab ManualDr. Yushi Zhou.docx
EELE 5331 Digital ASIC DesignLab ManualDr. Yushi Zhou.docxtoltonkendal
 
ImplementationOFDMFPGA
ImplementationOFDMFPGAImplementationOFDMFPGA
ImplementationOFDMFPGANikita Pinto
 
Enterprise Routing Protocol Migration
Enterprise Routing Protocol MigrationEnterprise Routing Protocol Migration
Enterprise Routing Protocol MigrationJeffrey Sicuranza
 
Data over dab
Data over dabData over dab
Data over dabDigris AG
 
Invited Paper for ASM 2004
Invited Paper for ASM 2004Invited Paper for ASM 2004
Invited Paper for ASM 2004stephen_mcparlin
 
disertation_Pavel_Prochazka_A1
disertation_Pavel_Prochazka_A1disertation_Pavel_Prochazka_A1
disertation_Pavel_Prochazka_A1Pavel Prochazka
 
3D Functional Tolerancing And Annotation CATIA
3D Functional Tolerancing And Annotation CATIA3D Functional Tolerancing And Annotation CATIA
3D Functional Tolerancing And Annotation CATIALeslie Schulte
 
S Pii Plus+C+Library+Programmer+Guide
S Pii Plus+C+Library+Programmer+GuideS Pii Plus+C+Library+Programmer+Guide
S Pii Plus+C+Library+Programmer+Guideguestd2fe1e
 
S Pii Plus+C+Library+Programmer+Guide
S Pii Plus+C+Library+Programmer+GuideS Pii Plus+C+Library+Programmer+Guide
S Pii Plus+C+Library+Programmer+Guideguestd2fe1e
 
DESIGN AND IMPLEMENTATION OF AN IMPROVED CARRY INCREMENT ADDER
DESIGN AND IMPLEMENTATION OF AN IMPROVED CARRY INCREMENT ADDERDESIGN AND IMPLEMENTATION OF AN IMPROVED CARRY INCREMENT ADDER
DESIGN AND IMPLEMENTATION OF AN IMPROVED CARRY INCREMENT ADDERVLSICS Design
 
Robust link adaptation in HSPA Evolved
Robust link adaptation in HSPA EvolvedRobust link adaptation in HSPA Evolved
Robust link adaptation in HSPA EvolvedDaniel Göker
 
G.723.1 2017
G.723.1 2017G.723.1 2017
G.723.1 2017haidersim
 

Similar to Paper are thereanyrulesofthumb_brener (20)

ASIC_Design.pdf
ASIC_Design.pdfASIC_Design.pdf
ASIC_Design.pdf
 
Chen Zeng (201375033)
Chen Zeng (201375033)Chen Zeng (201375033)
Chen Zeng (201375033)
 
Tree structured partitioning into transform blocks and units and interpicture...
Tree structured partitioning into transform blocks and units and interpicture...Tree structured partitioning into transform blocks and units and interpicture...
Tree structured partitioning into transform blocks and units and interpicture...
 
EELE 5331 Digital ASIC DesignLab ManualDr. Yushi Zhou.docx
EELE 5331 Digital ASIC DesignLab ManualDr. Yushi Zhou.docxEELE 5331 Digital ASIC DesignLab ManualDr. Yushi Zhou.docx
EELE 5331 Digital ASIC DesignLab ManualDr. Yushi Zhou.docx
 
ImplementationOFDMFPGA
ImplementationOFDMFPGAImplementationOFDMFPGA
ImplementationOFDMFPGA
 
Enterprise Routing Protocol Migration
Enterprise Routing Protocol MigrationEnterprise Routing Protocol Migration
Enterprise Routing Protocol Migration
 
22024582
2202458222024582
22024582
 
Tutorial for EDA Tools:
Tutorial for EDA Tools:Tutorial for EDA Tools:
Tutorial for EDA Tools:
 
Tutorial for EDA Tools
Tutorial for EDA ToolsTutorial for EDA Tools
Tutorial for EDA Tools
 
Aashto08
Aashto08Aashto08
Aashto08
 
Thesis
ThesisThesis
Thesis
 
Data over dab
Data over dabData over dab
Data over dab
 
Invited Paper for ASM 2004
Invited Paper for ASM 2004Invited Paper for ASM 2004
Invited Paper for ASM 2004
 
disertation_Pavel_Prochazka_A1
disertation_Pavel_Prochazka_A1disertation_Pavel_Prochazka_A1
disertation_Pavel_Prochazka_A1
 
3D Functional Tolerancing And Annotation CATIA
3D Functional Tolerancing And Annotation CATIA3D Functional Tolerancing And Annotation CATIA
3D Functional Tolerancing And Annotation CATIA
 
S Pii Plus+C+Library+Programmer+Guide
S Pii Plus+C+Library+Programmer+GuideS Pii Plus+C+Library+Programmer+Guide
S Pii Plus+C+Library+Programmer+Guide
 
S Pii Plus+C+Library+Programmer+Guide
S Pii Plus+C+Library+Programmer+GuideS Pii Plus+C+Library+Programmer+Guide
S Pii Plus+C+Library+Programmer+Guide
 
DESIGN AND IMPLEMENTATION OF AN IMPROVED CARRY INCREMENT ADDER
DESIGN AND IMPLEMENTATION OF AN IMPROVED CARRY INCREMENT ADDERDESIGN AND IMPLEMENTATION OF AN IMPROVED CARRY INCREMENT ADDER
DESIGN AND IMPLEMENTATION OF AN IMPROVED CARRY INCREMENT ADDER
 
Robust link adaptation in HSPA Evolved
Robust link adaptation in HSPA EvolvedRobust link adaptation in HSPA Evolved
Robust link adaptation in HSPA Evolved
 
G.723.1 2017
G.723.1 2017G.723.1 2017
G.723.1 2017
 

Paper are thereanyrulesofthumb_brener

  • 1. DesignCon 2016 Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? A Walkthrough from Physical Domain to Channel Operating Margin (COM) Testing. Jacov Brener, Marvell Israel Ltd jacovb@marvell.com Liav Ben-Artsi, Marvell Israel Ltd liav@marvell.com
  • 2. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 2 of 21 Abstract The 4x25Gb/s design presents significant challenges, one of which is channel qualification; therefore the COM was introduced [1]. COM is the outcome of an algorithm which purpose is to evaluate the channel and its ability to provide a good enough media for the 100Gb/s link. Although COM is adequate for channel evaluation it doesn't provide any guidance as of how to get the design there [2] .Looking at the past, engineers referred to their knowhow and rules of thumb, but do these rules of thumb apply now? Our goal is to provide examples of design guidelines for the case above and emphasize the limitations in which these guidelines hold. This work is not intended to substitute the COM algorithm, rather save time and effort on the road to a compatible interconnect. Author’s Biographies Jacov is a Senior Signal Integrity Engineer at Marvell Israel Ltd, starting this year. For the previous 10 years Jacov has worked in Intel's Communication and Storage Infrastructure Group, most of the time as package designer and signal/power integrity focal point in a high-speed PHY team. Liav is a Senior Signal Integrity Manager at Marvell Israel Ltd. Liav has worked at Marvell for the last 15 years, most of the time as a signal integrity engineer, focusing on signal integrity methodologies, guidelines, test plan definitions and training. He holds several patents in the field as well as several others pending. Liav took an active part in IEEE802.3bj standard committee.
  • 3. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 3 of 21 Table of Contents 1. Introduction to COM............................................................................................................................... 4 1.1. The algorithm and the procedure..................................................................................................... 4 1.2. Reference system channel................................................................................................................ 4 1.3. DUT host board................................................................................................................................ 6 1.4. Work scope ...................................................................................................................................... 7 2. Minimizing reflections to increase COM ............................................................................................... 8 2.1. The impact of Si termination ........................................................................................................... 8 2.2. Balancing host board card impedance discontinuities..................................................................... 9 3. Key guidelines to lower Xtalk in a high-density board ........................................................................ 13 3.1. Via-to-via FEXT............................................................................................................................ 13 3.2. NEXT due to reference plane resonance ....................................................................................... 17 4. The importance of correct de-skewing and its impact on MC.............................................................. 18 5. Summary and Conclusion..................................................................................................................... 20 Table of Figures Figure 1: Frequency domain properties of cable assembly test with case #1 (a) package and case #2 package (b)..................5 Figure 2: DUT host board schematic drawing............................................................................................................................6 Figure 3: Frequency domain properties of the host board for TX lane (a) and RX lane (b).......................................................7 Figure 4: COM dependence on TX & RX load impedances ........................................................................................................8 Figure 5: Via structure #1 (a) and vias structure #2 (b) ...........................................................................................................10 Figure 6: Differential Return Loss (RLdiff) of via structure #1 and structure #2 .....................................................................10 Figure 7: Differential Impedance (Zdiff) of via structure #1 and structure #2 .........................................................................10 Figure 8: RLdiff of systems with structures #1 and structures #2 .............................................................................................11 Figure 9: Differential Insertion Loss (ILdiff) of systems with structures #1 and structures #2.................................................11 Figure 10: SBR response & Equalization effect of systems with structures #1 and structures #2 ............................................12 Figure 11: SBR Bathtub curves of systems with structures #1 and structures #2......................................................................12 Figure 12: Typical via placement..............................................................................................................................................14 Figure 13: E-filed of via structure without shield vias (a) and with them (b)............................................................................14 Figure 14: Xtalk of via structure without shield vias and with them .........................................................................................15 Figure 15: FEXT of systems with via structures without shield vias and with them..................................................................15 Figure 16: Xtalk SBR of systems with via structures without shield vias and with them...........................................................16 Figure 17: Bathtub curves of Xtalk only (a) and Xtalk with ISI effect (b) of systems with via structures without shield vias and with them....................................................................................................................................................................................16 Figure 18: E-filed of via structure without terminating vias (a) and with them (b) ..................................................................17 Figure 19: E-filed of via structure without terminating vias on PCB edge ...............................................................................17 Figure 20: Xtalk of via structure without terminating vias and with them ................................................................................18 Figure 21: De-skew technique #1 (a) and de-skew technique #2 (b).........................................................................................19 Figure 22: MC of a trace with de-skew technique #1 and de-skew technique #2......................................................................19 Figure 23: MC of a system with de-skew technique #1 and de-skew technique #2 ...................................................................19 Table 1: COM of systems with structures #1 and structures #2 ................................................................................................12 Table 2: COM of systems with via structures without shield vias and with them......................................................................16 Table 3: COM of systems without terminating vias and with them ...........................................................................................18 Table 4: COM of system in the initial and optimized phases.....................................................................................................20 Table 5: Rules-of-Thumb for mitigation of different types of signal distortion .........................................................................20
  • 4. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 4 of 21 1. Introduction to COM 1.1. The algorithm and the procedure The 4x25Gb/s design presents significant challenges to today's technology and was defined by IEEE 802.3bj working group. Within this group several new methodologies were developed for the task. One of the methodologies is called COM which stands for Channel Operating Margin. COM is the outcome of a time domain algorithm, provided within the specification as a set of equations and procedures, which purpose is to evaluate the quality of the channel and its ability to provide a good enough media for the 100Gb/s link. Basically the COM will report how much margin in dB a channel has to perform well given a reference transmitter and receiver [3]. COM algorithm implementation example [4] was also provided by members of the same working group and was used in this work in order to evaluate the host board (sometime referred to as line card) performance along with a specific reference system channel. In order to use COM one must have channel response (THRU) and may have additional crosstalk (Xtalk) profiles (all as Touchstone S-parameters model). Once those are in place some algorithm setting may be adjusted and channel evaluation may begin. The procedure in which COM operates can be summarized in 3 basic steps: 1. TX (transmitter) & RX (receiver) equalization: TX Finite Impulse Response (FIR) filter, RX Continuous Time Linear Equalizer (CTLE) , RX Decision Feedback Equalizer (DFE) 2. Xtalk addition: Far-End Xtalk (FEXT), Near-End Xtalk (NEXT) 3. Result analysis: frequency domain plots, pulse responses, bathtubs, COM Once the COM is given it’s the final and only figure of merit to be used to evaluate a specific channel’s performance, and this figure of merit will be used as a measure to evaluate out test cases’ result. 1.2. Reference system channel In order to evaluate the quality of a host board a complete end-to-end system is needed, therefore a reference channel such as a backplane or a cable should be taken along with the host board model. The reference system channel for our work was chosen to be a 5m 24AWG cable connected to a zQuad Small Form-factor Pluggable Plus (zQSFP+) connector a.k.a. cable assembly [5]. Testing the above cable assembly to 100GBASE-CR4 specification with a reference analytical host board model yields COM=4.3dB (for a worst case reference package model, also provided as part of the
  • 5. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 5 of 21 COM equations) while the pass/fail criteria is COM≥3dB [6]. Figure 1 shows the frequency domain properties of the cable assembly concatenated with a reference board model and 2 cases of packages: 1. Case #1: package trace length of 12mm 2. Case #2: package trace length of 30mm (was the worst case package in our case) (a) (b) Figure 1: Frequency domain properties of cable assembly test with case #1 (a) package and case #2 package (b) As can be seen this cable assembly introduces very high -30dB loss and -54dB Xtalk at Fbaud/2 to the system, which yield an Insertion loss to Crosstalk Ratio (ICR) of 24dB at Fbaud/2. Figure 1 and COM results clearly show that this kind of cable assembly stresses the receiver significantly even without introducing a real host board instead of the analytical model.
  • 6. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 6 of 21 1.3. DUT host board The role of the host board is to connect the Integrated Circuit (IC) chip transmitting/receiving 4x25Gb/s signal to the zQSFP+ connector. Figure 2 describes the schematic drawing of the board. Figure 2: DUT host board schematic drawing The board was designed with Panasonic Megtron6 dielectric and Hyper Very Low Profile (HVLP) copper. The traces connecting the Ball Grid Array (BGA) footprint of the IC to the zQSFP+ connector are 5in long, running on inner layers (striplines) in order to reduce FEXT. Board model was constructed using cascaded models of its components. The components were extracted using a 3D full wave commercial filed solver. Figure 3 shows the frequency domain properties of the modeled board, both for TX and RX ends. From the plots it can be seen that the board’s loss at Fbaud/2 is ~-7.3dB which is exactly the full budget allocated for it by the specification [6] so this board should stress the cable assembly test to its maximum loss wise.
  • 7. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 7 of 21 (a) (b) Figure 3: Frequency domain properties of the host board for TX lane (a) and RX lane (b) 1.4. Work scope Loss has long been considered as a major factor that influences the system’s performance [7]. Reflections and Xtalk are also recognized as types of distortion yet the guide lines to minimize them usually include trace impedance and trace to trace separation only. Mode Conversion (MC) is another type of noise which looks relatively easy to handle yet sometimes it could be misleading. In our work we’ll focus on finding new design rules and rules of thumb in order to minimize these causes of signal distortion. Our assumption is that a common designer engineer is well familiar with the loss mechanisms of the system and is capable to minimize it, so our drive will be to help him in other aspects of the host board design.
  • 8. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 8 of 21 2. Minimizing reflections to increase COM 2.1. The impact of Si termination Silicon (Si) or on-die termination has been here for ages [8] and its value for high-speed signals is traditionally 50Ω single-ended or 100Ω differential. Often a question arises what is the optimized impedance for a high-speed serial link. In theory, if all channel’s components are tuned to differential impedance of 100Ω and no significant loss is present clearly the best value for the Si termination from signal integrity point of view is 50Ω single-ended. However in practice not all channel components reach 100Ω target and there’s significant loss introduced in the channel, for those reasons 50Ω single- ended impedance at TX&RX might not be the best case. In order to answer the question above our reference system channel was tested against COM in various TX&RX termination values in a cable assembly test as in 1.2. The results are plotted in Figure 4. Figure 4: COM dependence on TX & RX load impedances Notice, that the optimum values for this channel are 35Ω for TX and 55Ω for RX. Those values are the result of equilibrium between the reflection mechanism which drives the impedances towards 100Ω and
  • 9. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 9 of 21 the loss mechanism which pushes the TX termination down and RX termination up according to simple voltage division. In any case, 50Ω Si termination isn’t the optimized value for any channel and cannot be taken as a rule of thumb for any given Si nor should it be taken as a reference impedance for any channel. 2.2. Balancing host board card impedance discontinuities Matching vias, capacitor pads and other impedance discontinuities is critical in order to reduce reflections in time domain or Return Loss (RL) and Insertion Loss Deviation (ILD) in frequency domain. Those channel parameters impact the signal quality as seen at the RX end directly. This has been known for a long time. Engineer’s target was always to design as much as transparent interconnect as possible. The important question that needs to be answered in this procedure is “transparent in what way?” Common knowledge, also written in some references [9], suggests trying to match those discontinuities to 100Ω differential impedance, probably since it’s the common reference impedance of the TX & RX as well as a common system target impedance. Although in 2.1 we have seen that this is not always the case, nor it’s the optimal case, let’s assume that the TX & RX terminations are indeed 100Ω. Even though the terminations are 100Ω, the system itself will most likely not have a flat 100Ω profile. Connectors, BGA and other mechanical components cannot be perfectly matched to 100Ω and they usually have more capacitive nature thus the “effective impedance” of the system is usually lower than 100Ω. Let’s examine 2 cases of via structures and see how they perform in this kind of system. Figure 5 shows the mechanical drawing of these 2 cases. As can be seen the signal-signal and signal-GND distances in structure #1 are greater than structure #2 so we’d expect structure #1 to be less capacitive and more inductive in nature thus to have higher impedance compared to structure #2. As mentioned before, it is common that vias have a differential impedance less than 100Ω, therefore our expectation is that structure #1 will have a somewhat lower impedance then 100Ωs and structure #2 will have even lower impedance than that. Figure 6 shows the differential RL of the 2 vias compared to 100Ω reference impedance. It can be clearly seen that in this setup structure #1 behaves in a better way then structure #2. Going back to our previous discussion regarding “effective impedance”, let’s assume that these vias are placed somewhere near the BGA area so it “sees” the board+system(cable)+board+BGA on one side and BGA at the other side. Let’s also guess that the “effective impedance” of the board+system(cable)+board+BGA side 90Ω and the BGA’s side only is 80Ωs. This assumption is
  • 10. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 10 of 21 usually a good initial guess since Si capacitance along with package discontinuities lower the effective impedance by a great deal while long board traces and system traces or cables hold the controlled impedance which is usually closer to 100ohm. (a) (b) Figure 5: Via structure #1 (a) and vias structure #2 (b) Figure 6: Differential Return Loss (RLdiff) of via structure #1 and structure #2 Figure 7: Differential Impedance (Zdiff) of via structure #1 and structure #2
  • 11. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 11 of 21 Figure 7 shows the Time-Domain Reflectometry (TDR) impedance for this case. Here we can conclude the opposite regarding which structure is better compared to the conclusion that was derived from Figure 6. It’s clear that structure #2 provides a smoother transition between impedances then structure #1 since its impedance is “located” just between the 2 “effective impedances” we assumed. This smother transition is what should make the via more transparent. So which structure is better #1 based on Figure 6 or #2 based on Figure 7? Figure 8 gives a thick hint to the answer for this question. The RL of the whole system with the host boards and cable assembly with structure #2 gives better RL especially up to 10GHz which is within the critical bandwidth for our 25Gbps design. Note that in both cases the IL stays almost the same up to a very high frequency as can be seen in Figure 9, but when looking a bit more at the IL one can notice that structure #2 has a smother frequency domain behavior as well as slighter lower loss. Figure 8: RLdiff of systems with structures #1 and structures #2 Figure 9: Differential Insertion Loss (ILdiff) of systems with structures #1 and structures #2
  • 12. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 12 of 21 The final answer is of course supplied by the COM and summarized in Table 1. For the given system via structure #2, placed in multiple places on the host board, increased the COM by 0.75dB taking it above the PASS/FAIL criteria! Recalling the COM result of the cable assembly itself with a reference analytical host board, an actual real board routing with via structure #2 decreased the COM by a mere 0.65dB. Systems with COM [dB] Structures #1 2.9 (FAIL) Structures #2 3.65 (PASS) Table 1: COM of systems with structures #1 and structures #2 The reasons for the COM difference between the 2 systems can be seen in Figure 10 and Figure 11. Structure #2 is more transparent thus system’s RL is better. The main tap of the single bit response (SBR) is higher since the channel is less reflective. At the end it leads to the fact that structure #2 opens the bathtub by ~20% compared to structure #1! Figure 10: SBR response & Equalization effect of systems with structures #1 and structures #2 Figure 11: SBR Bathtub curves of systems with structures #1 and structures #2
  • 13. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 13 of 21 All the figures of merit along the way led us to think that structure #2 is better except the RL in Figure 6. This is due to the fact that the RL was plotted with reference to the standard 100Ω ports to not to the real “effective impedance” of the system. Of course each system is different and has its own “effective characteristic impedance” yet in general the following design guide lines can be derived: 1. Usually the “effective impedance” of the system is lower than 100Ω, and that was our case also 2. Match the impedance discontinuities to the “effective impedance” of the system seen form each side of the impedance discontinuity, not necessarily 100Ω as indicated in former rules of thumb 3. In today’s technology almost minimum distances of signal-signal and signal-ground (keeping 5- 10mil additional spacing) yielded the best results for our tested case 3. Key guidelines to lower Xtalk in a high-density board 3.1. Via-to-via FEXT Xtalk between vias has been a challenge for a long time and has been discussed extensively such as in [10]. The importance of via-to-via Xtalk increases as we proceed to 25Gbps signals. This happens due to the fact that trace-to-trace Xtalk is no longer the most significant limitation as we go from µ-strip (outer layers traces) to strip-line (inner layer traces), in which far-end Xtalk is significantly lower. And so, the role of via-to-via Xtalk becomes more significant. Just as in the trace-to-trace Xtalk case, via-to-via Xtalk is usually about FEXT. Common practice is to route TX & RX traces on different layers so they run in parallel as we look at their X-section. Due to this practice trace-to-trace NEXT is usually negligible (since those signals have a solid reference plane between them) yet FEXT is needed to be accounted for. Moreover, when signal need to change layers, and vias has a wider footprint compared to the overall X-section of the differential pair then these via structures are placed perpendicular to the direction of the traces. This practice can be seen in Figure 12. Once again, as in the trace case, it’s usually better to separate between TX & RX vias so assuming each via’s nearest neighbors are of the same type we get a row of TX or RX vias, making the via Xtalk a FEXT issue. However this is not yet the most challenging case. When dealing with a high lane count i.e. a high density board the distance between these adjacent rows is rapidly decreasing. As can be seen in Figure 12, although the distance between the rows is larger than the one between vias in the same row, they don’t have the reference via to act as guards and defend them from one another.
  • 14. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 14 of 21 Figure 12: Typical via placement A proof for this concern can be found in the E-field distribution shown in Figure 13 (a). As can be seen, a very strong field exists around the signal vias which penetrates to the adjacent row much easier then to the adjacent via at the same row. Note, that even the farthest via on the adjacent row in this case suffers from more Xtalk then the closet via on the same row, since the latter is separated from the aggressor with a reference via. (a) (b) Figure 13: E-filed of via structure without shield vias (a) and with them (b) The solution to this phenomenon is to stitch a shield via row between the signals’s via rows so the Xtalk won’t be able to penetrate to these victim vias. Figure 13 (b) shows the effects of the added shielding vias. Note that the E-field has dropped by ~40dB between the vias and now all the vias has a fair immunity to Xtalk.
  • 15. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 15 of 21 The degradation of FEXT in vias due to those shielding vias is ~10dB at Fbaud/2 as plotted in Figure 14 and the overall effect on the system FEXT is 4-8dB through the whole spectrum of significance as plotted in Figure 15. This effect has of course a profound impact on the performance by decreasing the FEXT SBR peak from 0.5mV to 0.3mV, as depicted on Figure 16, which is a ~40% improvement! Figure 14: Xtalk of via structure without shield vias and with them Figure 15: FEXT of systems with via structures without shield vias and with them
  • 16. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 16 of 21 Figure 16: Xtalk SBR of systems with via structures without shield vias and with them Furthermore as we look at system level performance, the bathtub curves depicted on Figure 17 show that eye opening only due to Xtalk is improved by ~15% while a combination of Xtalk and Inter Symbol Interference (ISI) improved by ~23%. (a) (b) Figure 17: Bathtub curves of Xtalk only (a) and Xtalk with ISI effect (b) of systems with via structures without shield vias and with them Finally as we look at the COM result, it becomes clear: The shielding vias are a must. COM increases by 0.25dB only due to placing shielding vias. Systems with via structures COM [dB] without shield vias 3.1 with shield vias 3.35 Table 2: COM of systems with via structures without shield vias and with them A general recommendation therefore would be not to base an expectation of Xtalk purely on the distance and symmetry of the vias.
  • 17. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 17 of 21 3.2. NEXT due to reference plane resonance Plane resonance due to unterminated power/ground plane is not new [11] yet till now it was relatively small. Plane resonance occurs when a signal reaches the end of a plane, bounces of it and propagates out of the edge to the cavity. With the bitrate increase, leading to shorter wave length effects, and the decrease of noise margins its impact on a high-speed signal cannot be neglected any longer. This is especially true in the case of noise source as NEXT which has to be very low in order to maintain good enough Signal-to-Noise Ratio (SNR). Consider a case where a structure of TX vias is placed near a similar structure of RX vias, while both structure are placed the edge of a reference plane. Figure 18 (a) shows the E-field going out of the left TX via pair to the right RX via pair. Note the high E-field near the aggressor via but also the same order of magnitude of the field near the edges of the reference plane. Figure 18 (b) shows the same structure and the E-field outcome yet in this case terminating vias are placed and stitch adjacent reference planes. It can be clearly seen the field drops dramatically at the whole cavity, especially at the plane edges. Figure 19 examines the same phenomena on the edge of a Printed Circuit Board (PCB) plane and reveals the same nature of the E-filed on the edge, just in a smaller magnitude. (a) (b) Figure 18: E-filed of via structure without terminating vias (a) and with them (b) Figure 19: E-filed of via structure without terminating vias on PCB plane edge
  • 18. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 18 of 21 The tremendous impact on NEXT is plotted in Figure 20 which predicts ~8dB impact on the Xtalk, only from this tiny structure! Furthermore, when adding these additional NEXT sources to our previously optimized channel, COM decreases by 0.1dB only due to this small section as shown in Table 3. Although 0.1dB decrease in COM may look negligible to some designers, this performance degradation can easily be avoided by stitching the edge of the reference plain with terminating vias at distance of 125mil and below, which corresponds to quarter wave length of Fbaud/2 in a typical dielectric used for 25Gbps applications. Figure 20: Xtalk of via structure without terminating vias and with them Systems type COM [dB] without terminating vias 3.15 with terminating vias 3.25 Table 3: COM of systems without terminating vias and with them 4. The importance of correct de-skewing and its impact on MC The impact of skew in channel delay has been discussed extensively [12] as well as also its impact on system level performance [13], however the best way to de-skew signal’s path has remained controversial. One of the great thumb rules states “de-skew where skew happens” yet it’s not clear where exactly skew happens and how much. Consider 2 de-skewing techniques as shown in Figure 21. From first glance both of them look alike and follow the rule of thumb above, by de-skewing the total trace length where it begins. In second look, one
  • 19. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 19 of 21 can notice, that the skew occurs in 2 locations – large skew at the left and small skew at the right of the trace edges, just near the pins/vias. (a) (b) Figure 21: De-skew technique #1 (a) and de-skew technique #2 (b) The difference in the MC between these 2 de-skewing schemes is shown in Figure 22. Clearly the 2nd de-skewing method (b) is better and decreases the MC by 2-3dB. Figure 22: MC of a trace with de-skew technique #1 and de-skew technique #2 The overall impact of applying technique #2 on the entire system instead of technique #1 on the total system’s MC is depicted in Figure 23 and it is 6-8dB. Figure 23: MC of a system with de-skew technique #1 and de-skew technique #2 Once again, similar to 3.2, this performance degradation can be easily avoided by proper analysis of “where and by how much skew happens”.
  • 20. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 20 of 21 5. Summary and Conclusion In our work we have walked the path from physical design, through frequency-domain modeling towards final COM verdicts. We have shown that sometimes old rules of thumb don’t work, moreover in the 25Gbps case they will seldom succeed to work. New insights on system’s impedance and its impact on COM have been raised as we consistently compared physical structures using COM. Different chip impedance impact channel performance substantially and 100Ω termination on either side is not the optimal value for all systems. Different chip impedance impacts the “effective impedance” of the system substantially, so rules of thumb are only good in some well-defined impedance environment. Proper via treatment can increase system’s COM by ~1dB as shown in Table 4, 0.75dB out of it by via matching to the “effective impedance” and 0.25dB by reference via shielding. These steps may avoid system’s failure just as was done in our case of a 5m cable. System design phase COM [dB] Initial 2.35 (FAIL) Optimized 3.35 (PASS) Table 4: COM of system in the initial and optimized phases Simple-to-implement guide lines, such as reference plane termination and proper de-skewing, can prevent unnecessary system performance degradation. In summary, our work indicated simple rules of thumb that worked and improved our 25Gbps system. Those are listed in Table 5 Distortion type Mitigation Rule-of-Thumb RL Decrease distances of signal-signal and signal-ground vias almost to minimum Xtalk Shield signal vias from all sides, yet bit farther then signal-ground distance Stich termination vias at the end of reference planes at distance ≤120mil MC De-skew where skew happens by the same amount it happens at the location Table 5: Rules-of-Thumb for mitigation of different types of signal distortion
  • 21. Are There Any Rules of Thumb When It Comes to 100Gb/S Board Design? DC16 Page 21 of 21 References [1] R. Mellitz, A. Ran, M. P. Li and V. Ragavassamy, "Channel Operating Margin (COM): Evolution of Channel Specifications for 25 Gbps and Beyond," in DesignCon, 2013. [2] X. Dong, M. Mo, F. Rao, W. Jin and G. Zhang, "Relating COM to Familiar S-Parameter Parametric to Assist 25Gbps System Design," in DesignCon, 2014. [3] M. Brown, M. Dudek, A. Healey, L. Ben Artsi, R. Mellitz, C. Moore, A. Ran and P. Zivny, "The state of IEEE 802.3bj 100 Gb/s Backplane Ethernet," in DesignCon, 2014. [4] A. Ran, "COM Update," 6 November 2014. [Online]. Available: http://www.ieee802.org/3/bj/public/tools/ran_com_3bj_3bm_01_1114.zip. [5] M. Shanbhag, "QSFP 5 m 24AWG Cable Assembly," 16 January 2014. [Online]. Available: http://www.ieee802.org/3/100GCU/public/ChannelData/shanbhag_3bj_01_0114.zip. [6] IEEE, "802.3bj Clause 92," 2014. [Online]. Available: https://standards.ieee.org/findstds/standard/802.3bj-2014.html. [7] H. Johnson and M. Graham, High-Speed Signal Propogation: Advanced Black Magic, 1st ed., Prentice Hall, 2003. [8] H. Johnson and M. Graham, High speed digital design: a Handbook of black magic, 1st ed., Prentice Hall, 1993, pp. 223-248. [9] E. Bogatin, Signal and Power Integrity – Simplified, 2nd ed., Prentice Hall, 2009, p. 401. [10] G. Blando, J. Miller, D. Winterberg and I. Novak, "Cross Talk in Via Pin Fields, Including the Impact of Power Distribution Structures," in DesignCon, 2009. [11] V. S. Pandit, W. H. Ryu and M. J. Choi, Power Integrity for I/O Interfaces: With Signal Integrity/Power Integrity Co-Design, 1st ed., Prentice Hall, 2010, pp. 238-247. [12] E. Kunz, J. Y. Choi, V. Kunda, L. Kocubinski, Y. Li, J. Miller, G. Blando and I. Novak, "Sources and Compensation of Skew in Single-Ended and Differential Interconnects," in DesignCon, 2014. [13] M. Rowlands, P. Patel and P. Casher, "System Performance as a Function of Common Mode Metrics," in DesignCon, 2012.