- eBay runs OpenStack as its private cloud infrastructure, providing on-demand access to computing resources for various eBay brands and services on shared infrastructure. - eBay uses OpenStack APIs and multiple availability zones and regions to provide high availability and scale capacity across its global infrastructure. - eBay has developed tools like StackWatch and StackMetrics to monitor OpenStack performance, simulate workloads, and detect issues in the cloud.

1. OpenStack
Private
Cloud
at
eBay
OpenStack
Design
Summit
–
Nov
2013
1

2. Analyst Conference 2013
John Donahoe- CEO
Marc Carges - CTO
Bob Swan - CFO
Devin Wenig – President eBay NA

3. Prod
Prod
QA
PCI
Secure
QA
DEV
QA
DEV
DEV

4. Frictionless on- Multi-tenant,
demand access isolated
One cloud on
shared infra
billmelater.com
brands4friends.de
corp.ebay.com
ebay-kleinanzeigen.de
ebay.ca
ebay.com
ebay.de
gumtree.com
hunch.com
marketplats.ne
milo.com
5/1/2013
6/1/2013
7/1/2013
8/1/2013
Users
5/1/13
6/1/13
7/1/13
VMs Created
9/1/2013
10/1/2013
Projects
8/1/13
9/1/13
VMs Deleted
10/1/13
5/1/2013
6/1/2013
7/1/2013
8/1/2013
Cores
9/1/2013
10/1/2013
4

5. project
project
VPC
…
project
project
…
VPC
Availability Zone
project
…
project
VPC
VPC
Availability Zone
Region
…
project
…
VPC
AZ
Region
Globally Shared infra
(Compute, Network, Storage)
5

6. OpenStack APIs
Project -> VPC
Network -> VPC
Dashboard
VPC selection
Capacity based
Network selection
Images -> VPC
Dashboard
AZ selection
Class of Service (PCI, DEV, PROD, …)
VPC
Virtual Networks
Block, Object Storage
Flavors classes
Managed images
Large scale networks
Multi tiers storage
Commodity H/W
Win/Linux
6

7. OpenStack
is
not
cloud
hDp://www.subbu.org/blog/2013/07/openstack-­‐is-­‐not-­‐cloud
7

8. Cloud
is
a
service!
Metering
and
chargeback
Monitoring
and
alerRng
High
availability
RemediaRon
Config
management
Log
processing
Infra
onboarding
Scale
out
OpenStack
APIs
Metrics
Cloud
monitoring
Autoscaling
Capacity
planning
CI
Network
design
Upgrades
Builds
Net/info
sec
SLA
AlerRng
Incident
resoluRon
User
experience
Customer
support

9. Monitoring
9

10. nova
boot
-­‐-­‐image
a4a3d552-­‐4f92-­‐4426-­‐82d9-­‐ff2226d72d08
-­‐-­‐flavor
medium
myvm
Is
the
cloud
broken?
User
?
You
10

11. RabbitMQ
Capacity
issues
Quantum
plugin
failures
vm_state=ACTIVE
does
not
really
mean
success
Glance
down
Didn’t
get
DHCP
in
Ame
for
cloud
init
Metadata
Aming
out
11

12. Log
source
logstash
ØMQ
Log
source
logstash
ØMQ
Log
source
logstash
ØMQ
Log
source
logstash
ØMQ
1
Grok
paDerns
ØMQ
logstash
ElasRcSearch
ØMQ
logstash
ElasRcSearch
kibana
ØMQ
logstash
ElasRcSearch
2
Low
latency
3
A
cluster
that
can
scale
horizontally
12

13. Log
source
logstash
ØMQ
Log
source
logstash
ØMQ
Log
source
logstash
ØMQ
Log
source
logstash
ØMQ
ØMQ
logstash
ElasRcSearch
ØMQ
logstash
ElasRcSearch
kibana
ØMQ
logstash
ElasRcSearch
Graphite
StatsD
Zabbix
Metrics
Monitoring
Alerts
13

14. Log
source
logstash
ØMQ
Log
source
logstash
ØMQ
Log
source
logstash
ØMQ
Log
source
logstash
ØMQ
ØMQ
logstash
ElasRcSearch
ØMQ
logstash
ElasRcSearch
kibana
ØMQ
logstash
ElasRcSearch
Graphite
StatsD
OpenStack
nodes,
processes,
network,
storage,
…
Is
the
cloud
broken?
Zabbix
May
be?
14

15. Log
source
logstash
ØMQ
Log
source
logstash
ØMQ
Log
source
logstash
ØMQ
Log
source
logstash
ØMQ
ØMQ
logstash
ElasRcSearch
ØMQ
logstash
ElasRcSearch
kibana
ØMQ
logstash
ElasRcSearch
Graphite
StatsD
StackWatch
Simulate
KPIs
•
•
•
•
•
OpenStack
nodes,
processes,
network,
storage,
…
Zabbix
StackMetrics
Create,
delete
Volumes
Ping
success
Ping
latency
…
hDp://www.subbu.org/blog/2013/10/monitoring-­‐and-­‐alerRng-­‐for-­‐openstack
15

16. Scale
out
16

17. One
Cloud
User
Experience
Availability
for
Business
Availability
1.
Nova
“availability
zones”
2.
Keystone
“regions”
3.
Nova
“cells”
Capacity
17

18. AWS
Region
A
Region
B
Availability
Zone
Availability
Zone
Availability
Zone
Availability
Zone
Availability
Zone
keystone
glance
Availability
Zone
nova API
Hypervisors
cinder API
Block storage
backend
neutron API
network driver +
agents + infra
swift API
storage
18

19. Region A
Dashboard
Nova
Keystone
Quantum
Cinder
Glance
Swift
region
services
az
services
infra
Dashboard behind a
global load balancer VIP
user chosen replication of
selected resources
Region C
Dashboard
Keystone
Glance
Region B
Dashboard
Nova
Keystone
Quantum
Cinder
Glance
Swift
region
services
az
services
infra
Region
≈
A
fully
decoupled
OpenStack
deployment
Nova
Quantum
Cinder
Swift
region
services
az
services
infra
Availability
Zone
≈
Keystone
“region”
Nova
availability
zone?
Nova
cells
–
an
implementaRon
detail
19

20. Build
and
Deployment
20

21. Ubuntu,
apt,
fabric
Ubuntu,
apt,
puppet
and
foreman
Pet
test
clouds
1. Bare-­‐metal
provisioning
for
hypervisors,
controllers,
NVP,
…
2. Virt
env
tarballs,
puppet
and
foreman
3. A
full
working
deployment!
CaDle
test
clouds
Patches
ParRal
upgrades
Rollbacks
1. Change
orchestraRon
2. Foreman
UI
“ClickOps”
21

22. Our
Commitment
to
the
Community
• Puppet
automaRon
• Full
topology
with
virtual
environments
• ProducRon
ready
• Zabbix
templates
• StackWatch
and
StackMetrics
• Contribute
on
VPCs
• Contribute
on
scale-­‐out
paDerns
22

23. JC
MarRn
Subbu
Allamaraju
@jchmarRn
@sallamar
23