In this deck from the HPC Advisory Council Spain Conference, Dan Olds from OrionX discusses the High Performance Interconnect (HPI) market landscape, plus provides ratings and rankings of HPI choices today. "The HPI market is the very high-end of the networking equipment market where high bandwidth and low latency are non-negotiable. It started out as a specialist proprietary segment but has blossomed into an indispensable, large, and growing area. Products in this category are used to build extreme-scale computing systems. They are typically not used for traditional telco, enterprise, or service provider networking needs. In this talk, we’ll take a look at the technologies and performance of their high-end technology and the coming battle between onloading vs. offloading interconnect architectures." Watch the video presentation: http://wp.me/p3RLHQ-fON Learn more: http://orionx.net/wp-content/uploads/2016/06/HPI-Environment-OrionX-Constellation-DataCenter-20160626.pdf Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

1. High Performance Interconnects:
Landscape, Assessments &
Rankings
Dan Olds
Partner, OrionX
September 21st, 2016

2.  Very top end of networking market – when
you absolutely need high bandwidth and
low latency
 Without HPI, you don’t really have a
cluster – or at least one that works very
well
 Performance has been rising at 30%
annually
 Spending on HPI is also rising significantly
High Performance Interconnects (HPI)
©2016 OrionX 2

3. Current HPI line
Multi-Rack
10G
40G
100G
1G
100G
Link Speed
TCP/IP
InfiniBand
Specialized
OPA
Net Protocol App Comm
MPI
JDBC
RMI
IIOP
SOAP
etc.
Single-Rack
HPI
market segment

4. Three Types of HPI
©2016 OrionX 4
 Ethernet
– Sold by a host of providers, Cisco, HPE, Juniper, plus many others
– Tried and true interconnect, easiest to implement
– While it has the bandwidth of others, latency is pretty high (ms rather than ns)
 Proprietary
– Primarily sold by Cray, SGI, and IBM plus a few others
– Have to purchase a system in order to get their brand of HPI
– Intel is a new entrant in this segment of the market, although without an accompanying system
 InfiniBand
– Mellanox has emerged as the de-facto leader
– Highest performance based on published numbers:100Gb/s, 150m messages/s, 90ns latency

5. Key Differences in HPI: Product Maturity/Position
©2016 OrionX 5
Ethernet
 Ethernet has been around longer than any HPI, but surpassed in performance
– Still many installations, but has lost much of its share at the high end
– Latency (measured in ms, not ns) is the problem, not bandwidth

6. Key Differences in HPI: Product Maturity/Position
©2016 OrionX.net 6
 Intel Omni Path Architecture
– Intel and Omni-Path are in their infancy still, very few installations
– Handful of customers (although some big names), few, if any, in production
– Claims bandwidth/latency/message rate same or better than InfiniBand (covered later)

7. Key Differences in HPI: Product Maturity/Position
©2016 OrionX.net 7
 InfiniBand
– Has been in the HPI market since early 2000’s
– Thousands of customers, millions of nodes
– Now makes up almost half of the TOP500 list
– Synonymous with Mellanox these days

8. Key Differences in HPI technology
©2016 OrionX 8
Onload vs. Offload
 Onload: main CPU handles all network processing chores, adapter and switches just pass
the messages, examples
– Intel Omni-Path Architecture, Ethernet
– Also PC servers, old UNIX systems where CPUs handled every task and received interrupts
on communications
 Offload: HCA and switches handle all network processing tasks, very little or no need for
main CPU cycles, allows CPU to continue processing applications, examples
– Mellanox InfiniBand
– Mainframes with communication assist processors used to allow CPU to process
applications, not communications

9. Offload Details
©2016 OrionX.net 9
 Network protocol load includes:
– Link Layer: packet layout, packet forwarding,
flow control, data integrity, QoS
– Network layer: adds header, routing of
packets from one subnet to another
– Transport layer: in-order packet delivery,
divides data into packets, receiver
reassembles packets, sends/receives
acknowledgements
– MPI operations: scatter, gather, broadcast,
etc.
 With offload, ALL of these operations are
handled by the adapter hardware, example:
InfiniBand HBA

10. Onload Details
©2016 OrionX.net 10
 Network protocol load includes:
– Link layer packet layout, packet forwarding,
flow control, data integrity, QoS
– Network layer: header, routing of packets
from one subnet to another
– Transport layer: in-order packet delivery,
divides data into packets, receiver
reassembles packets, sends/receives
acknowledgements
– MPI operations: scatter, gather, broadcast,
etc.
 With onload, ALL of these operations are
performed by the host processor, using host
memory

11. Onload vs. Offload
©2016 OrionX.net 11
 Onload vs. Offload
isn’t a big deal when
the cluster is
small…

12. Onload vs. Offload
©2016 OrionX.net 12
 But it will become a
very large deal when
the cluster becomes
larger
 Will particularly be a
problem on scatter,
gather type collective
problems when head
node will be overrun
trying to process
messages

13. Onload vs. Offload
©2016 OrionX.net 13
 As node count increases, performance of Onload will drop
– Higher node count = more messaging, pressure on head node
 Node counts are increasing significantly
 Dedicated hardware ASICs
– Much faster than general purpose CPUs
 MPI not highly parallel
– With Onload, this means that speed is
limited to slowest core speed
– Has no bearing on Offload speed

14. FUD War Rampant
©2016 OrionX.net 14
 Cost of HPI in cluster budget is
typically ~25% of total
 Prices in high tech typically
don’t increase over time
 Price points for new products
typically are the same as the
former high end product they
replace…ex: high end PCs
From: The Next Platform, “Intel Stretches Deep Learning On
Scalable System Framework”, 5/10/16

15. More FUD War…..
©2016 OrionX.net 15
 All images provided by Intel, all from The Next Platform story “Intel Stretches Learning on Scalable System Framework” May
10th, 2016
 What else do these images have in common?

16. FUD Wars – Behind the Numbers
©2016 OrionX.net 16
 It’s all in the fine print,
right?
 Here’s Intel’s fine print for
the graphs on the last
slide….
 “dapl” is key, it’s an Intel
MPI mechanism that
doesn’t allow for offload
operations ala InfiniBand
…..48 port (B0 silicon). IOU Non-posted Prefetch disabled in BIOS.
Snoop hold-off timer = 9. EDR based on internal testing: Intel MPI
5.1.3, shm:dapl fabric, RHEL 7.2 -genv
I_MPI_DAPL_EAGER_MESSAGE_AGGREGATION off. Mellanox
EDR ConnectX-4 Single Port Rev 3 MCX455A HCA. Mellanox
SB7700 – 36 Port EDR InfiniBand switch. MLNX_OFED_LINUX-3.2-
2.0.0.0 (OFED-3.2-2.0.0). IOU Non-posted Prefetch enabled in BIOS.
1. osu_latency 8 B message. 2. osu_bw 1 MB message. 3.
osu_mbw_mr, 8 B………
Software and workloads used in performance tests may have been
optimized for performance only on Intel microprocessors.

17. “dapl” Strikes Again!
©2016 OrionX.net 17
 100% CPU core
utilization on a Offload
HCA?!!
 Does anyone believe
this?!!
 This means that about
half of the Top500
systems are
absolutely useless
 “dapl” is a key
component of this
‘benchmark’ once
again

18. FUD Aside, here are the numbers….
©2016 OrionX.net 18
Intel OPA Mellanox InfiniBand
Bandwidth 100 Gb/sec 100 Gb/sec
Latency .93 .85 or less
Message rate 89 million/sec* 150 million/sec
* this number, provided by Intel, has dropped from >150 million in 2015

19. HPI Roadmaps
©2016 OrionX.net 19
 InfiniBand roadmap
shows EDR now
(100Gb/s) and HDR in
2017 (200Gb/s)
 Can’t find a solid Intel
OPA roadmap
 Ethernet roadmap
shows 200Gb/s in
2018-19

20. Major HPI Choices
©2016 OrionX.net 20
Vendor
Market Customer Product
Presence Trends Overall Readiness Needs Overall Capabilities Roadmap Overall
Mellanox 9 9 9 8 9 8.5 9 10 9.5
Ethernet vendors 7 7 7 9 6 7.5 7 6 6.5
Intel 6 8 7 6 7 6.5 7 8 7.5

21. Mellanox
Intel
Ethernet vendors
Vendor Market Product Customer
Ethernet 7 6.5 7.5
Mellanox 9 9.5 8.5
Intel 7 7.5 6.5
OrionX Constellation

22. ©2016 OrionX 22
OrionX Constellation™ reports