Presentation CC-4153, Verizon Cloud Compute and the SM15000, by Paul Curtis, at the AMD Developer Summit (APU13) November 11-13, 2013.

1. VERIZON CLOUD COMPUTE AND
THE AMD SEAMICRO 15000
Paul Curtis
Chief Architect Verizon Cloud Compute
APU13

2. Collaboration
• Unbelievable amount of support from Seamicro
2

3. Verizon cloud development goals
• Very few different hardware components
• Consistent predictable performance
• Secure
• High performance
• Highly available
• No modification to customer applications
• No special purpose hardware
3

4. Verizon Cloud Differentiation
• Value for Performance
– User defined availability and performance
– User defined resources
• Reserved Performance
– Network, Storage and Compute
• Workload Simplicity
– Seamless integration with other deployments
– Single point of control
• Security
‒ Market leading security capabilities
‒ Embedded into every aspect of platform
• Continuum of Services
‒ Bridging private, public and hybrid clouds
‒ Allow the blending with colocation, managed services, networking
4

5. SM15000 SYSTEM
10 Rack Units, draws 3-3.5 KW
! Compute
– Up to 512 Opteron, Xeon or Atom cores in 10 RU
– 2,048 cores in a rack
– Up to 64GB DRAM/socket = 4 terabytes/system
! Networking
– 10 Gbps half duplex bandwidth to each CPU socket
– 16 x 10GbE Line Rate uplinks to the network
! Storage
– Up to 1,408 disks: HDD or SSD
– Up to 128 Terabytes of internal SSD storage
– Up to 5.3 Petabytes of storage
! Fabric
– 1.28 Tbps Freedom Supercompute Fabric
! Software
– Off the shelf OS, Hypervisors
5

6. Hardware architecture
• There are only three hardware component types. This simplifies
maintenance
– Arista 7508 a 384 port x 10Ge non-blocking L2 switch
– AMD Seamicro SM15000
– SSDs
• Network connections
6

7. Hardware diagram
Juniper
MX960
for
external
connectivity
Arista
7508
4x10Gb
links
from
arista
to
each
chassis
Up
to
90
AMD
Seamicro
SM15000s
7

8. Verizon’s use of seamicro chassis
• 160 GB of external bandwidth (network and storage)
• 54 Server cards for customer loads
• 2 Server cards for Verizon orchestration
• 8 Server cards for storage services
• ~1000000 IOPs
• 96 T usable SSD storage
8

9. Combine Hardware and Verizon software
to get
• A flat layer 2 ethernet switch
– ~12000 port 1 Gb/sec
– ~1500000 vlans
– 8.5M mac address table entries
– 11.5M traffic flows.
– Software configurable
• A storage array
– 90M IOPs
– 8.6 PB of SSD storage
• Scalable router firewall 1Gb- 400Gb/sec
• Scalable load balancers 1Gb-400Gb/sec
• Configurable IO performance
9

10. Network Packet flow
Hypervisor
presents
nic
of
specified
speed
to
VM.
Back
pressure
applied
by
hypervisor
VM
Nic
Queue
Hypervisor
text
Nic
Queue
NPU
VM
Nic
Queue
Hypervisor
Hypervisor
fairly
mixes
flows
from
different
VMs
Limited
to
max
Nic
speed
text
Nic
Queue
NPU
Shaped
to
max
speed
of
receiving
nic
Queue
Policer
Queue
Layer
2
Switching
Shaped
to
max
speed
of
receiving
Nic
Random
packet
drop
back
pressure
form
destinatio
n
queue
Layer
2
Switching
10
G
NIC
10
G
NIC
Queue
Arista
Switch
Prioritized
queue.
10

11. Networking Layer 2
• Hypervisor
– Shapes egress traffic
• NPU
– Provides true layer 2 ethernet switching
– Polices ingress flows
– Shapes egress flows
• Arista 7508
– Lots of bandwidth
• Remote congestion control
– Switch learns speeds of remote flows
– Switch performs remote drop if destination is congested
• Hardware based security
– Each customer network is on its own vlan
• Software configurable
11

12. Data I/O Stack
VM
HV
Paravirt (xvdb)
o
Storage VLAN
AoE Initiator
Ethernet
NPU
Arista
S
S
N
NPU
o
Ethernet
AoE Target
AIO
ZFS
Block
AoE Initiator
Ethernet
Ethernet
AoE Target
Block Device(s)
SCARD
12

13. Storage
• Hypervisor
– Shapes disk traffic (IOPs and Bandwidth)
– Participates in disk replication
• AoE
– Storage over layer 2 ethernet
– Allows storage targets to be any where in world
– Shared volumes
• Replication
• NPU
– Shapes read and write bandwidth
• Storage Service
– Snap shots
– Raid
• Storage card
– AoE target
13

14. Networking Layer 3+
Layer 3 and above network services just work since they are all based
on layer 2 networking.
•
•
•
•
•
Soft routers
Load balancers
Public IP (No Nat)
Tunnels
Wan optimizers
14

15. Inter-data center features
• Single user interface
• Networks can span multiple data centers
• Replicated disks can span multiple data centers
• Taking advantage of being part of a network company
15

16. Availability
• No single point of failure for network traffic
– “Bonded nics”
– “Bonded NPUs”
– Fabric reroutes itself
– Multiple paths through arista switches
• No single point of failure for replicated storage
– Raid 1 on SSDs
– Multiple storage servers
– Option to have replicated volumes span data centers
16

17. Security
• Physical security
• DDOS
• Network security
– Customer traffic on independent VLANs
– Untrusted entities (Hypervisors) firewalled from rest of system
• Storage security
– Each volume on a separate vlan
– Storage vlans firewalled (only AoE traffic, no target to target traffic)
• Management software
– Audit logs
– Security alerts
17

18. Possible Applications
• Move a current three tier app with your choice of soft router/firewall/load
balancer into the cloud
• Bridge a network from your data center to one in the cloud
• Move XEN and VMWare VMs into the cloud without modification
• Write a clustered app using shared storage
• Configure an applications performance so that you know it won’t fall over
when it is 3:00 in the afternoon and the cloud gets busy
• Write and test a new L3 protocol
• Voice
• Storage arrays
• Network devices
18