This document discusses techniques for improving CPU access to data from IO devices, including Direct Cache Access (DCA), PCIe Transaction Layer Processing Hints (TPH), and Data Direct I/O (DDIO). DCA allows the CPU to access cache data from IO devices to avoid memory access, but requires driver intervention. PCIe TPH and DDIO aim to prefetch or retain IO write data in the cache without driver involvement. DDIO is specific to Intel platforms and accelerates local socket performance, while DCA does not differentiate sockets. The document provides details on how each technique functions and references further reading.

1. © 2011 VMware Inc. All rights reserved
Revisit DCA, PCIe TPH and DDIO
VMware
Hisaki Ohara
Dec. 2014

2. Terminology
 Direct cache access
• Technology such that CPU can hit cache data of IO device on demand to
2
avoid memory access
 DCA
• Adopted since Xeon 5100 (Woodcrest)
• As a part of Intel IOAT
• OS/driver intervention required
 PCIe TLP Processing Hints (TPH)
• PCI-SIG: PCI Express 2.1 Protocol Extensions
 DDIO
• Adopted since Xeon E5 (SandyBridge)
• Likely to be a subset of PCIe TPH
• No OS/driver intervention required

3. DCA
 Hardware Prefetch on DMA Write
3
• Memory access cannot be avoided
• Need to avoid cache eviction before CPU uses
 DCA flag := CPUID(EAX:1):ECX[18]
• Require driver’s intervention
 TLP format for DCA
82599 datasheet: Fig.7.29
82599 datasheet: Fig.7.30 Available for multiple sockets

4. PCIe TLP Processing Hint (TPH)
 “TPH is a performance feature that allows CPU to prefetch or keep
certain PCIe writeback data in LLC for quick core consumption.”
 Steering Tag (ST) field: Identify target resources
• Three modes to derive ST value
4
• No ST Mode
• Interrupt Vector Mode
• Device Specific Mode

5. 5

6. 6

7. 7

8. PCIe Port after SandyBridge generation
8

9. DDIO: How to enable/disable
9

10. Differences of DDIO from DCA
 DDIO based on PCIe specification
• Not certain the usage of ST on DDIO from the public information
 DDIO accelerate only for local socket, while DCA does not matter
• I don’t think that DCA contributed the performance for remote sockets
10

11. References
 PCI Express Base Specification Revision 3.0
 PCI-SIG Engineering Change Notice (ECN): TLP Processing Hints
• https://www.pcisig.com/specifications/pciexpress/specifications/ECN_TPH_11
11
Sept08.pdf
 IDF 2009: TCIS006: PCI Express 3.0 Technology: Device
Architecture Optimizations on Intel Platforms
 Intel Xeon Processor E5-2600 v2 Datasheet, Vol.2
• http://www.intel.com/content/www/us/en/processors/xeon/xeon-e5-v3-
datasheet-vol-2.html
 Intel Data Direct I/O: Frequently Asked Questions
• http://www.intel.com/content/dam/www/public/us/en/documents/faqs/data-direct-
i-o-faq.pdf