The document outlines the collaborative efforts of various organizations, including CXL, PCI SIG, DMTF, OFA, OCP, and SNIA, to develop interoperable management standards for CXL fabric, memory, and devices. It discusses the complexities of CXL management using APIs and protocols while emphasizing the need for open standards and community involvement in advancing functionalities. The paper highlights ongoing projects and initiatives aimed at ensuring a seamless integration of disaggregated composability and industry-standard components.

1. What are CXL, PCI SIG, DMTF, OFA,
OCP & SNIA
doing to make CXL fabric,
memory & device management
interoperable?
CXL Fabric Management Standards

2. CXL Fabric Management Standards
Jeff Hilland, Distinguished Technologist HPE, President DMTF

3. • Provide a roadmap of Open Communities solving CXL fabric management
• Show how these communities are working together to ensure an
interoperable, multi-vendor solution is possible
• Any help the community could provide to these communities would be
appreciated
Agenda

4. • How do we get to
commercially available off
the shelf disaggregated
composability using
industry standard
components
• Without writing millions
of lines of custom code
to make it work
• While still allowing for
differentiation
The Approach:
• A multitude of open
The Infrastructure Management
Dilemma:
BMC
Host w/ FW, OS/VMM
Mgmt
SW
CXL Switch
CXL Controller
Media/Device
CXL 2/3
FM API
Agent
I2C
UART
PCIe
(B)MC
(DC-SCM)
Ethernet
FM API
Agent
Enclosure
Boundary
(sometimes)
I2C
UART
PCIe
Ethernet
?
?
?
?
?
?
?

5. • CXL management is largely via the CCI & FM API
• CCI (Component Command Interface) is used to manage CXL Memory
• Capacity/Label Info, Health & Alerts, Media/Poison, Security, QoS, etc.
• FM API are commands used to manage Switches
• Also device status registers (Event/Log)
• Accessed through the mailbox registers or MCTP
• Two mailboxes (primary, secondary): one for system SW, one for FW (only
primary is mandatory)
• MCTP is for sideband/out-of-band management
• This traffic is not currently comprehended by BMCs and is entirely different
than the rest of platform management
CXL

6. • PCI cards typically have both PCI VDM for management traffic
• This is what MCTP uses for In-Band management
• Flit mode requires changes (Non-flit mode is specified in PCI Spec)
• PCI signaling still has the I2C/I3C, USB pins for side band signaling
• This is used for out-of-band management using MCTP
• PCI devices already support not only MCTP but many support most of the
other PMCI payloads (next slide)
• Since cxl.io is basically cxl encapsulated pci, cxl.io devices can be
managed with PCI VDM traffic over CXL
PCI SIG

7. • PMCI standards
• PMCI is a suite of standards used for
platform management
• MCTP is the low level transport
• SPDM flows over MCTP
• PLDM flows over MCTP too
• PLDM has Type 2, FW Update,
RDE (Redfish Device Enablement),
File Transfer and other payloads
• FRU is going through a modernization
• Work In Progress is out, moving from a binary with extensions to JSON payload
• Every modern BMCs supports most of these protocols
• There is a split in the industry between those that prefer Redfish Device
Enablement vs PLDM Type 2
• https://www.dmtf.org/standards/pmci
DMTF - PMCI
KCS
PCIe VDM
Physical
Layer
Management Component Transport Protocol (MCTP)
Transport
Layer
MCTPover KCS
MCTPHost Interface
SMBus/I2C Serial
RMII Based Transport
(RBT) MCTPover Serial
PLDM over
MCTP
MCTPover PCIe VDM
SPDM over
MCTP
NC-SI over MCTP
NVMe-MI over
MCTP
MCTP
Control
Type = 0
PLDM
Type = 1
NC-SI Control
Type = 2
Ethernet
(NC-SI Passthru)
Type = 3
NVMe-MI
Type = 4
SPDM
Type = 5
MCTPover SMBus/I2C
NC-SI
Passthru
0: Messaging Control & Discovery
1: SMBIOS DataTransfer
2: Monitoring & Control
4: FRU Data
5: Firmware Update
6: Redfish DeviceEnablement
3: BIOS Control & Configuration
Upper
Layer
PLDM
Platform Level Data Model
(PLDM) Message Types
MCTP Message Types
= Physical Layer
= Binding
= Transport
= Data Model/Message
= PLDM Message
= Host Interface
Secured
Messages
Type = 6
Sec d Msgs over
MCTP
Vendor Defined
Messages
Type = 7E/7F
0: MCTP Control
1: PLDM
2: NC-SI Control
4: NVMe-MI
5: SPDM
7E/7F: Vendor Defined
3: NC-SI Passthru
MCTP Message Types
Encapsulation
= Encapsulation
NC-SI
Control
3F: OEM Specific

8. • Redfish is the industry platform management standard
• REST protocol with a JSON payload, specified by schemas
• Suite of self validation based on profiles.
• Started as just sever management, but now convers storage, composability,
fabrics, DCIM (power/cooling), DPUs, aggregators, operating systems,
workloads, containers and CXL
• CXL support released starting in the 2020.3 release
• White paper, presentation, YouTube
• Updates keep coming. 3.0 support complete in 2023.2
• Also has all of the device support, eventing, security, etc.
• CXL Mapping Spec (“Rosetta Stone”) now available
• Maps the CCI & FM API in 3.0 to Redfish
• https://redfish.dmtf.org
DMTF - Redfish
PCIe Devices
D1
PCIe Functions
F1
CXL Logical Devices
LD1

9. • OFA Sunfish group is
working on an Open
Fabric Management
Framework
• Code to manage
multiple fabrics
• Represent
components on the
fabric.
• Represent
management of the
fabric itself as well.
• Interface is based on
Redfish/Swordfish
OFA (Open Fabrics Alliance) -
Sunfish

10. • OCP Hardware Management
• Working in multiple areas on platform management
proscriptive standards
• Profiles – Redfish System & Component Profiles to
validate conformance for
Systems, Devices, Memory and Fabrics (et al)
• Combined with the Redfish validation tools, the profiles can be
used to programmatically validate implementations
• SMC – Satellite Management Controller Specification
• Requirements for fabric attached devices
(MCTP, PLDM, FW Update and RDE requirements)
• OCP CMS – Composable Memory Systems
• Meeting to solve data plane & solutions issues
• Deep Dive on CXL Redfish model in the archives
OCP Efforts
HW MGMT

11. • SNIA – Swordfish
• Extension of Redfish, leveraging
the core of Redfish storage &
memory
• Has advanced memory pooling,
grouping & service models that
align memory as storage with
other traditional storage media
• Has advanced NVMe, block, file
& object support
• https://www.snia.org/swordfish
SNIA

12. • Every
community
has a part
• Closely
working
together to
enable
solutions
The Answer:
BMC
Host w/ FW, OS/VMM
Mgmt
SW
CXL Switch
CXL Controller
Media/Device
(B)MC
(DC-SCM)

13. • Get involved in OCP: HW Mgmt, Profile sub-project & CMS
• Get involved in any of the Open Communities
• Contribute code to, for example, Sunfish
• Never too late to help.
Call to Action

14. Thank you!