Presentation at OpenStack Trove Day 2014 by Mat Lowery, Software Engineer and Auston McReynolds, Software Engineer from eBay

1. OpenStack Trove Day
August 19, 2014
Session Trove in Production Name-at click eBay
to change
Auston McReynolds, Software Engineer, eBay
Mat Lowery, Software Engineer, eBay

2. The eBay Inc. Portfolio
PLATFORM
8/19/14 tesora.com 2

3. Private Cloud at eBay

4. eBay Private Cloud: Like a Public Cloud
• One cloud on shared infra
• Like a public cloud for all eBay Inc properties
• Multi-tenant, isolated
• Different tenants with different needs
• Frictionless, on-demand access
• Reduce time to market
• Enable innovation
8/19/14 tesora.com 4

5. Architecture: Region vs. AZ
Region B
Swift
Keystone Nov
a Neutron Cinder
Glance Dashboard MySQL Misc
Dashboard behind global
load balancer VIP
Region A
Swift
AZ Services Region Svcs
Keystone Nov
a Neutron Cinder
Glance Dashboard MySQL Misc
AZ Services Region Svcs
Region C
Swift
AZ Services Region Svcs
Keystone Nov
a Neutron Cinder
Glance Dashboard MySQL Misc
8/19/14 tesora.com 5

6. Customizations
• VPCs
• Enforce obligations, restrictions, and capabilities
• Associated with Keystone projects
• OpenStack resources mapped to VPC (via metadata)
• Image metadata
• Host aggregate metadata
• Enforcement via custom middleware or scheduler
• Keystone
• LDAP
• Auto-tenancy
8/19/14 tesora.com 6

7. Customizations
• Horizon
• Most use the custom Horizon dashboard vs. API
• Connect To (uses image metadata)
• Copy datastore root password
• Disable UI functionality based on AZ capabilities or
Datastore
• Multi-AZ tenant switcher glues the clouds together
8/19/14 tesora.com 7

8. Packaging and Deployment
• Deployment unit: virtualenv tarball
• virtualenv --no-site-packages –relocatable
• Simplifies rollback
• Not perfect: doesn’t handle OS libs
• Rollout automation by Puppet, Foreman
• Auditing of changes
• Host overrides for testing in isolation
8/19/14 tesora.com 8

9. Monitoring
• Metrics
• Periodic gathering based on service DBs
• Share with end users
• Alerts to engineers
• SLAs
• Ensure being met via sampling; simulates cloud user
• Nova boot
• Swift upload and download
• Capture logs on failures
• Alerts to engineers
• Graphite for graphing and Zabbix for alerting
• Logstash/Elasticsearch/Kibana stack
8/19/14 tesora.com 9

10. Trove at eBay

11. Trove at eBay – The Beginning
• Mid-2013
• 1 developer
• Stack on Folsom
• Not integrated, still Red Dwarf
• MySQL only
• Single region
• No backups
8/19/14 tesora.com 11

12. Trove at eBay – Now
• Team (all ATCs)
• Icehouse
• Multiple Datastores
• MySQL 5.5, 5.6
• MongoDB 2.4
• Cassandra 2.0
• Couchbase 2.2
• Redis 2.8
8/19/14 tesora.com 12

13. Trove at eBay – Now - Continued
• Multi-region + HA, with limited VPC availabilty
• Backups for MySQL, MongoDB, & Couchbase
• Contributions to Trove:
• Couchbase Impl + Backups
• Multi-Datastore Horizon
• Clusters (in-flight)
• + many more (incl. 400+ code-reviews)
8/19/14 tesora.com 13

14. Deployment Details
• HA
• Stable* release cuts
• Puppet + Foreman for Control Plane
• Fabric for Guests
• Shared infra, sans RabbitMQ
• RabbitMQ HA Mirrored
8/19/14 tesora.com 14

15. Deployment Details
• KVM
• Ubuntu
• Network: Bridged or Overlays
• Cinder Volumes
• Venv’d Vanilla Guest Agent
• Diskimage-Builder
• w/ reference & custom elements
• Image per Datastore
8/19/14 tesora.com 15

16. Deployment Details
• Designate (soon)
• Zabbix
• w/o Heat
8/19/14 tesora.com 16

17. Challenges
• With shared infra:
• Nova quotas vs. Trove quotas
• Horizon (compute duplicity, how to connect, etc.)
• True ownership of spawned assets
• Failure; who’s fault?
• Scheduling
8/19/14 tesora.com 17

18. Challenges
• With Trove:
• Upgrades/Rollouts
• RabbitMQ
• Feature support across Datastores
• Multi-region Datastore deployments
8/19/14 tesora.com 18

19. What’s Next
• Clusters!
• Advanced scheduling
• Additional monitoring
• Performance tuning
• More management extensions (UI + API)
• API-driven Datastore differentiation
8/19/14 tesora.com 19

20. eBay Trove Team is Hiring
http://linkd.in/1sSVbHv or vyella@ebay.com

21. Questions