Host Interfacing Board to Universal Base Board interconnections in server node

1. Host to UBB Interfacing

3. OAM connector 0
● 54V/48V and 3.3V input power
● x16 SerDes to connect to host
● 3x16 SerDes for the accelerator to accelerator communication
○ x16 may split into sub-links like 2* x8 or 4* x4 or 16* x1

4. OAM connector 1
● 54V/48V and 12V (optional) input power
● Other single-ended signals like JTAG, GPIOs etc.
● Upto four SerDes for the accelerator to accelerator communication
or other purposes:
○ SerDes 4, 5, 6 and 7 are up to x16 lanes which can split into x8, x4 or x1
● SerDes 7 may be defined for different use cases:
○ This link could be the 7th SerDes for some cases to have a fully
connected interconnect between the modules
○ It could be the 2nd link to the host for the ASIC(s) on the module, eg. a
full x16 link, 2x8, or 4x4 links.
○ Or it could be a unique defined link by some ASICs Eg., it could be a
downstream port for the ASIC on the module.

5. HIB (Host Interface Board Example)

6. ● The Host Interface Board (OAI-HIB) provides eight x16 highspeed Links such
as PCIe Gen-4 to UBB
● HIB provides Clock, Reset, and PowerGood to UBB
● HIB provides security (root of trust– RoT), management, and control interface
to UBB (OAI-SCM)

7. PCIe Switch
A PCIe switch basically functions as an I/O controller and magnifies the host
devices’ capabilities to support more devices.

8. Case Study - PI7C9X3G1632GP PCIe Gen3 Switch
PI7C9X3G1632GP is a PCIe Gen3 packet switch that supports 32 lanes of Gen3
SERDES in 2-port to 16-port configurations.

9. Case Study - PI7C9X3G1632GP PCIe Gen3 Switch
Functional Modes of operation
● Base Mode (Fan-out Mode)
● Switch Partition Mode (Fan-out Mode)
● Cross-Domain End-Point (CDEP Mode)

10. Case Study - PI7C9X3G1632GP PCIe Gen3 Switch
● Base Mode (Fan-out Mode)

11. Case Study - PI7C9X3G1632GP PCIe Gen3 Switch
● Switch Partition Mode (Fan-out Mode)

12. Case Study - PI7C9X3G1632GP PCIe Gen3 Switch
● Cross-Domain End-Point Mode

13. Case Study - PI7C9X3G1632GP PCIe Gen3 Switch
Mode Selection
PORTCFG_x[2] PORTCFG_x[1] PORTCFG_x[0] Functional Mode
0 0 0 1Port-16Lane Configuration
0 0 1 2Port-16Lane Configuration
0 1 0 3Port-16Lane Configuration
0 1 1 4Port-16Lane Configuration
1 0 0 5Port-16Lane Configuration
1 0 1 8Port-16Lane Configuration

14. Appendix

15. PCIe Lane Bifurcation
PCIe bifurcation is a feature that allows the division of data lanes in a PCIe slot. It
does not affect overall speed but does allow a larger slot to act like multiple
smaller slots.

16. ● Bridge
○ In networking, a bridge joins the two lowest
layer of the OSI reference model - Physical
and Data Link between two or more LAN
segments
○ Bridges may or may not have intelligence,
allowing all traffic to pass between the
ports, or inspecting the frame header
information to decide if the frame should
pass.
Bridge vs. Switch
● Switch
○ The switch is also used to denote a
bridging capability, however, a switch often
has multiple ports while a bridge typically
contains only two.
○ A switch is a “multi-port bridge”.

17. PCIe Bridge vs. Switch (Examples)
● Bridge
○ A bridge that passes one PCI Express
port’s data to a single USB 2.0 port could
be considered a simple transparent bridge
that makes an electrical conversion (from
PCIe to USB) and passes the data payload
between ports.
○ Alternatively, the bridge might include
intelligence such that only USB destination
data are allowed to pass while other data
present at the PCIe port is blocked.
● Switch
○ Switches can also bridge multiple ports
and media flavors (such as PCI to PCIe) or
single x4 PCIe lanes to multiple x1 lanes-
and all permutations thereof.

18. Hot Plug Types
● Surprise hot plug
○ Surprised serial hot-plug
○ Surprised parallel hot-plug
● Managed hot plug
○ Managed serial hot-plug

19. https://studylib.net/doc/18315930/hot-and-surprise-plug-recommendations-for-
enterprise

20. PCI Express

21. The PCI Express electrical interface is measure by the number of simultaneous
lanes.

22. PCI Express Pinout
Pin Side B Side A Description
1 +12 V PRSNT1# Must connect to farthest PRSNT2#
pin
2 +12 V +12 V
Main power pins
3 +12 V +12 V
4 Ground Ground
5 SMCLK TCK
SMBus and JTAG port pins
6 SMDAT TDI
7 Ground TDO
8 +3.3 V TMS
9 TRST# +3.3 V
10 +3.3 V aux +3.3 V Aux power & Standby Power
11 WAKE# PERST# Link reactivation; fundamental reset
Key Notch

23. PCI Express Pinout
(x1)
Pin Side B Side A Description
12 CLKREQ#[24] Ground Clock Request Signal
13 Ground REFCLK+ Reference clock differential pair
14 HSOp(0) REFCLK−
Lane 0 transmit data, + and −
15 HSOn(0) Ground
16 Ground HSIp(0)
Lane 0 receive data, + and −
17 PRSNT2# HSIn(0)
18 Ground Ground
PCI Express x1 cards end at pin 18