This document summarizes a presentation on MEMS optical switching in datacenters. It discusses using MEMS switches to build a hybrid optical/electrical network fabric that is more cost-effective than a fully electrical fabric. The hybrid fabric uses MEMS switches to route large "elephant" traffic flows optically while routing smaller "mice" flows electrically. Algorithms are used for dynamic topology management to optimize the optical topology based on traffic patterns. Evaluation shows the hybrid fabric can improve performance of HPC workloads while being up to 30% cheaper than an electrical fat-tree network. Challenges include edge routing and traffic engineering in reconfigurable optical networks.

1. © 2009 IBM Corporation
© 2013 IBM Corporation
MEMS Optical Switching in the Datacenter
Silicon Photonics for Next Generation Computing Systems
HiPEAC Computer Systems Week
October 2013
Kostas Katrinis – IBM Research, Ireland

2. 2
Outline
● Scope & Background
● Motivation & Challenges
● Hybrid Network Architecture
● Data Plane
● Control-Plane
● System Evaluation
● Use Cases
● Conclusion
Part I - Introduction
Part II – Arch & Tech
Part III – Evaluation
Part IV – Use Cases

3. 3
Scope
Part I - Background
● Target Markets
● (Cloud) Datacenters - Θ(10K) Servers
● HPC Clusters (82% in Nov'12 Top-500)
● Target Systems:
● Data Network Fabric

4. 4
DC Traffic Trends
Part I - Background
● 76% of the traffic is
intra-datacenter *
● Total DC traffic CAGR
33% to 2015 *
● Traffic percentage exiting the rack
is high (up to 90%) **
● ...and we expect it to increase (scale-out workloads)
* Cisco Global Cloud Index: Forecast and Methodology, 2011–2016
** Benson et al., “Network Traffic Characteristics of Data Centers in the Wild”, IMC'10

5. 5
Design Trade-offs (Performance)
Performance
Cost
Highly
oversubcr.
High
Bisection
Trade-off
Part I - Background
● We need high-capacity between any two points
in the DC
● and at various scales (incremental deployment)
● ... we need $$$

6. 6
Motivating Example
Item Item List Price (USD)** Qty Total List Price (USD)
BNT G8264 (64-port switch) 30,000 5,120 153,600,000
BNT SFP+ SR Transceiver 665 262,164 174,325,760
MM Fiber Cable 28 131,072 3,670,016
Estimated List Prices
**Source: ibm.com
Fabric Price
331M USD
≈ Compute Price
(@5K/server)
Part I - Motivation
● Full-bisection fat-tree @ 65k servers
● Building block: 64-port Ethernet switches (ala VL2*)
● Denser switches will not help you (e.g. 288-port Mellanox
Vantage)
Greenberg et al., “VL2: A Scalable and Flexible Data Center Network”, SIGCOMM 2009

7. 7
Motivating Example (cont.)
Total Price
331m USD
AND.....
#Cables to route
131,000
Can you count the birds in the nest?
Part I - Motivation

8. 8
Paradigm Shift - Switch Light
Tiltable Mirrors implemented via MEMS (Micro-Electrical Mechanical Systems)
+ High-radix (320 ports you can buy, 1024 feasible)
+ No transceivers
+ Decreasing $/port
+ 50x less Watts/port vs. electronics
+ Can switch up to ~1Tbps
+ Protocol Agnostic
Electronic Switch (Ethernet) Optical MEMS switch
Price/Port (USD) 1100 (includes TxRx cost) 350
Bandwidth/Port 10Gbps “Rate-free”
Power/Port (W) 10 0.2
Requires TxRxs Yes No
x3
x∞
x50
Part I - Motivation

9. 9
MEMS Switch in the DC
Part I - Challenges
● Repurposing is not free:
● 10-200ms switching latency vs. sub-μsec Ethernet
switch (point-to-point “circuits”)
● L2 spanning-tree forwarding bad option for ROI
(applies to electronic redundant topologies too!)
● Traffic Engineering (becomes dynamic Topology
Management?) is important
● Collectives?

10. 10
Related Approaches
Codename Affiliation Targets Working Prototype Comments
Helios UCSD/Google HPC/DC Yes First-principles, lacking integration,
no edge routing, supporting
infrastructure (e.g. monitoring)
c-Through CMU/Rice/Intel DC No (Emulation) Reconfiguration algorithms, traffic
splitting, problems not addressed
at scale
OSA
(previously
Proteus)
Northwestern/UIUC/NEC DC Yes (with Wavelength-
Division Multiplexing)
Mostly pursuing multiple
wavelengths/fiber
Plexxi Plexxi DC Product offering Not a re-configurable architecture,
low-bisection ring between racks
Part II – Architecture

11. 11
Hybrid Fabric Architecture
Part II – Architecture

12. 12
High-level Functionality
Part II – Architecture
● Bijective TE:
● Mice are routed via the 1G electronic fabric
● Elephants are routed via the 10G optical fabric
● Optical Fabric is reconfigurable
● Centralized control optimizes topology against
traffic pattern and demand volume

13. 13
Multi-hop & Multi-path Data Plane
Part II – Data Plane
● Our simulation work showed that multi-hop reduces
overhead of slow switching latency
● Relaxes the impact of slowly movable p2p circuits
● Larger topology space (not just bipartite graphs)
● Multi-path as throughput booster (utilization)
Multi-hop: Rack-2 reaches
Rack-4 via Rack-3 TOR switch

14. 14
VLAN-based Forwarding
● Routing over 802.1p overlays
● TOR ports along a multi-hop path are assigned the
same VLAN-ID
● Paths “touching” common TOR switch(es) use
distinct VLAN-IDs
● Dynamic VLAN-ID assignment/revoking via central
controller
Part II – Data Plane

15. 15
Server-based Path Selection Part II – Data Plane
● OVS based
● Mice flows per default via eth0, elephant flowspec
pushed by the controller to OVS

16. 16
VLAN forwarding - A bird-eye view
● Not clean: re-purposing a feature to cancel
another feature (spanning-trees)
● Not infinitely scalable (4094 IDs)
● Server support is off datacenter
provider/networking vendor premise in some
models (e.g. IaaS)
● Tenant is the master of the server
● VLAN tagging is slow (coming up...)
Part II – Data Plane

17. 17
VLANs vs. Openflow Performance
Part II – Data Plane
● All measurements at IBM G8264 (7.6.1 firmware)
● At 32 ports switching, OF is 2x faster
● VLAN tagging latency has a 700ms “DC” component
● OF support is work-in-progress
802.1p Openflow

18. 18
Controller Loop
Part II – Control Plane

19. 19
Dynamic Topology Management
● Input:
● Traffic Matrix (bytes)
● Optical physical topology
● Circuit state (used/not-used)
● Output:
● Optical Topology (optical cross-connections)
● Mapping of multi-hop paths to circuits
● Goal:
● Maximize optical throughput (volume of TM routed
optically)
Part II – Control Plane

20. 20
Dynamic Topology Mgmt Algorithms
● Showed that the problem is NP-complete
(reduction to circular arrangement problem)
● Heuristic approaches:
● High-Demand First (HDF): cluster demand based
on proximity and fit as much demand as possible to
optical fabric available capacity
● Simulated-Annealing (SA): couple HDF loops with
SA optimization
● ILP modelling for optimality sense at lower
scale
Part II – Control Plane

21. 21
Topology Mgmt Algos Evaluation
● Hop-bytes as throughput measure here (lower is
better)
● SA-100 best in throughput vs. performance trade-off
Part II – Control Plane

22. 22
Cost Competitiveness
● Comparison vs. fat-tree at various over-subscription levels
(parameter β)
● Hybrid is 30% cheaper at full-bisection
● Competitiveness diminishes but hybrid is a winner throughout
Part III – Cost Eval.

23. 23
Proof-of-Concept Prototype Part III – Perf. Eval.

24. 24
Evaluation Scenarios
● 4 racks, 40 servers (10 servers/rack)
● Equi-cost comparisons vs. fat-tree
● For a given hybrid network setup (parameter β),
evaluate application performance against electronic
fat-tree
● HPC Workload Input
● NAS Parallel Benchmarks
● FFTW
Part III – Perf. Eval.

25. 25
Evaluation Results Set-1
● Comparison vs. 1:25 fat-tree
● 25% improvement for most workloads
● At least as good for 2 cases
Part III – Perf. Eval.

26. 26
Evaluation Results Set-2
● Comparison vs. 1:4 fat-tree
● Up to 35% improvement
● At least as good for 2 cases
Part III – Perf. Eval.

27. 27
Further “Killer” Use-cases
Part IV – Use-cases
● HPC workloads are challenging (collectives,
dynamic)
● We are working on integrating and evaluating:
● Data-intensive (Big Data) frameworks (Hadoop)
● Massive VM migration
● Checkpointing
● ...on-going

28. 28
Conclusions
● Hybrid optical/electrical networks are cost-
competitive
● Results show that performance is not degraded
(to say the least)
● Edge engineering burden is not necessarily less
than routing/flow scheduling in electronic fat-tree
● Main Challenges Ahead:
● SDN edge
● Bring Traffic Engineering/Topology Management
closer to the application
● Optical performance in multi-stage optical setups
● More use-cases to increase confidence/persuasion
Part IV – Conclusions

29. 29
Results Publication

Diego Lugones, Konstantinos Christodoulopoulos, Kostas Katrinis, Marco Ruffni, Donal
O'Mahony, and Martin Collier,"Accelerating communication-intensive parallel workloads
using commodity optical switches and a software-configurable control stack”, in
Proceedings of the 2013 International European Conference on Parallel and Distributed
Computing (Euro-Par 2013), Aachen, Germany, August 2013

Kostas Katrinis, Guohui Wang and Laurent Schares, "SDN control for hybrid OCS/electrical
datacenter networks:an enabler or just a convenience?", in Proceedings of the 2013 IEEE
Summer Topicals, IEEE Photonics Society , Hawai, USA, July 2013

Konstantinos Christodoulopoulos, Kostas Katrinis, Marco Ruffini and Donal O’Mahony, "Tailoring
the Network to the Problem: Topology Configuration in Hybrid EPS/OCS
Interconnects", in CONCURRENCY AND COMPUTATION: PRACTICE AND EXPERIENCE Journal,
Wiley Interscience, invited article (in press)

Diego Lugones, Kostas Katrinis, Martin Collier and Georgios Theodoropoulos, "Parallel
Simulation Models for the Evaluation of Future Large-Scale Datacenter Networks", in
Proceedings of the 16th IEEE/ACM International Symposium on Distributed Simulation and Real
Time Applications, Dublin, Ireland, October 2012

Konstantinos Christodoulopoulos, Marco Ruffini, Donal O’Mahony and Kostas Katrinis,
"Topology Configuration in Hybrid EPS/OCS Interconnects", in Proceedings of the 2012
International European Conference on Parallel and Distributed Computing (Euro-Par 2012),
Rhodes Island, Greece, August 2012 (Distinguished Paper Award)

Diego Lugones, Kostas Katrinis and Martin Collier, "A Reconfigurable Optical/Electrical
Interconnect Architecture for Large-scale Clusters and Datacenters", in Proceedings of
the ACM International Conference on Computing Frontiers (CF '12), Cagliari, Italy, May 2012
(Best Paper Award)

30. 30
Dr. Diego Lugones (co-worker)
Dr. Martin Collier (co-author)
Dr. K Christodoulopoulos (co-worker)
Dr. Marco Ruffini (co-author)
Prof. Dr. Donal O'Mahony (co-author)
Trinity College
Dublin
Dublin City
University
Credit

31. 31
THANK YOU!
Q&A