#rackstackatl
Justin Hammond - Developer
Andy Hill - Systems Engineer
Chad Norgan - Systems Engineer
Neutron at Scale
#rackstackatl
Rackspace is early in Neutron implementation
Migrating from older versions of Quantum/Melange used since the...
#rackstackatl
Tens of thousands of compute nodes
Hundreds of thousands of instances
Most instances have two or more ports
...
#rackstackatl
Maintain backwards compatibility with existing products
Neutron will be the ultimate authoritative source fo...
#rackstackatl
Quark Plugin: Open source plugin for Neutron v2 API with IPAM
Custom database migration from Melange/Quantum...
#rackstackatl
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Rackspace’s Neutron Implementation
Neutron-api nodes running
quark pl...
#rackstackatl
Wafflehaus is a middleware for some specific Rackspace requirements
Very simple way to minimize upstream dif...
#rackstackatl
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Wafflehaus - “The API Mullet”
Business logic in the front, party in t...
#rackstackatl
Does the request body contain
particular UUIDs
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Wafflehaus Explained
W...
#rackstackatl
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Wafflehaus Explained
API Request
Wafflehaus middlewares
Quark plugin
...
#rackstackatl
Calls to Keystone
Melange/Quantum Neutron (trunk) Wafflehaus + no-auth
Build 0 5 per port 0
Delete 0 5 per p...
#rackstackatl
Wafflehaus and No-Auth Middleware
Neutron-api with
wafflehaus
PTR for 10.1.2.3?
PTR at compute.trusted.domai...
#rackstackatl
[composite:neutronapi_v2_0]
use = call:neutron.auth:pipeline_factory
noauth = dns_filter request_id catch_er...
#rackstackatl
Call Volume Before & After
#rackstackatl
Call Volume Before & After
#rackstackatl
Nova caches a copy of the instance’s network information (info cache)
Cache is refreshed on instance operati...
#rackstackatl
Happens on nova-compute restart
Also happens every heal_instance_info_cache_interval (default 1m)
Currently ...
#rackstackatl
nova-cells and Info Cache Updates
Child cells periodically sync with parent cells
Migration to Neutron expos...
#rackstackatl
Callback system between nova and neutron
Read-only database slave usage
Cells support
Nova & Neutron: Fewer ...
#rackstackatl
Publicly expose neutron
Security Groups extension support through OVS flows
RACKSPACE® HOSTING | WWW.RACKSPA...
#rackstackatl
Patches, Blueprints
https://review.openstack.org/#/c/88484/ (Neutron, Nova and Cells)
https://blueprints.lau...
#rackstackatl
RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218
US SALES: 1-800-961-2888 | US SUPPORT: 1-800-9...
Upcoming SlideShare
Loading in …5
×

Neutron scale

1,305 views

Published on

In this session, we will discuss the operational issues that Rackspace has encountered during and after implementing Neutron at a large scale. Neutron at scale required a significant amount of development and operations effort, some of which resulted in deviations from upstream code. Finally, our team would like to discuss our solutions and our upstream differences for Neutron and OpenStack that we believe are necessary so that it can be more performant at scale.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,305
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
53
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Neutron scale

  1. 1. #rackstackatl Justin Hammond - Developer Andy Hill - Systems Engineer Chad Norgan - Systems Engineer Neutron at Scale
  2. 2. #rackstackatl Rackspace is early in Neutron implementation Migrating from older versions of Quantum/Melange used since the launch of our public cloud Scope of this talk is primarily Nova ⬄ Neutron interaction and the challenges we faced deploying Neutron at scale Scope of the Talk
  3. 3. #rackstackatl Tens of thousands of compute nodes Hundreds of thousands of instances Most instances have two or more ports RACKSPACE® HOSTING | WWW.RACKSPACE.COM What we mean when we say “at scale”
  4. 4. #rackstackatl Maintain backwards compatibility with existing products Neutron will be the ultimate authoritative source for network state IP Address Management (IPAM) Modular network drivers so Neutron can service heterogeneous port types Enable new products to easily integrate into our public cloud offering RACKSPACE® HOSTING | WWW.RACKSPACE.COM Implementation Requirements
  5. 5. #rackstackatl Quark Plugin: Open source plugin for Neutron v2 API with IPAM Custom database migration from Melange/Quantum->Neutron/Quark Wafflehaus middleware collection RACKSPACE® HOSTING | WWW.RACKSPACE.COM Implementation Details
  6. 6. #rackstackatl RACKSPACE® HOSTING | WWW.RACKSPACE.COM Rackspace’s Neutron Implementation Neutron-api nodes running quark plugin with wafflehaus Active/Passive database with slave Active/Passive Load Balancers
  7. 7. #rackstackatl Wafflehaus is a middleware for some specific Rackspace requirements Very simple way to minimize upstream diffs Upstream efforts better spent on work that benefits the broader community RACKSPACE® HOSTING | WWW.RACKSPACE.COM Wafflehaus Overview
  8. 8. #rackstackatl RACKSPACE® HOSTING | WWW.RACKSPACE.COM Wafflehaus - “The API Mullet” Business logic in the front, party in the back
  9. 9. #rackstackatl Does the request body contain particular UUIDs RACKSPACE® HOSTING | WWW.RACKSPACE.COM Wafflehaus Explained Wafflehaus middlewares Would this request violate policy?Add this tag to the request header Quark plugin Neutron-api API Request
  10. 10. #rackstackatl RACKSPACE® HOSTING | WWW.RACKSPACE.COM Wafflehaus Explained API Request Wafflehaus middlewares Quark plugin Neutron-api
  11. 11. #rackstackatl Calls to Keystone Melange/Quantum Neutron (trunk) Wafflehaus + no-auth Build 0 5 per port 0 Delete 0 5 per port 0 Info Cache Update 0 LOTS 0 TOTAL 0 TOO MANY 0 RACKSPACE® HOSTING | WWW.RACKSPACE.COM
  12. 12. #rackstackatl Wafflehaus and No-Auth Middleware Neutron-api with wafflehaus PTR for 10.1.2.3? PTR at compute.trusted.domain A for compute.trusted.domain? A at 10.1.2.3 DNS Server RACKSPACE® HOSTING | WWW.RACKSPACE.COM API Request x-forwarded-for
  13. 13. #rackstackatl [composite:neutronapi_v2_0] use = call:neutron.auth:pipeline_factory noauth = dns_filter request_id catch_errors extensions neutronapiapp_v2_0 keystone = request_id catch_errors authtoken keystonecontext extensions neutronapiapp_v2_0 [filter:dns_filter] paste.filter_factory = wafflehaus.dns_filter.whitelist:filter_factory whitelist = trusted.domain enabled = true RACKSPACE® HOSTING | WWW.RACKSPACE.COM Wafflehaus Explained
  14. 14. #rackstackatl Call Volume Before & After
  15. 15. #rackstackatl Call Volume Before & After
  16. 16. #rackstackatl Nova caches a copy of the instance’s network information (info cache) Cache is refreshed on instance operations which reach out to Neutron Callback system is needed RACKSPACE® HOSTING | WWW.RACKSPACE.COM On Info Cache Updates
  17. 17. #rackstackatl Happens on nova-compute restart Also happens every heal_instance_info_cache_interval (default 1m) Currently 6 calls to Neutron per port Set heal_instance_info_cache_interval=0 RACKSPACE® HOSTING | WWW.RACKSPACE.COM On Info Cache Updates (continued)
  18. 18. #rackstackatl nova-cells and Info Cache Updates Child cells periodically sync with parent cells Migration to Neutron exposed upstream bug that was corrected in rpc network api, not neutron Cache updates were sent from child cells to global cells faster than global cells could process Delays other messages from being processed
  19. 19. #rackstackatl Callback system between nova and neutron Read-only database slave usage Cells support Nova & Neutron: Fewer calls that do more (e.g., 1 API call, many ports) RACKSPACE® HOSTING | WWW.RACKSPACE.COM What’s needed
  20. 20. #rackstackatl Publicly expose neutron Security Groups extension support through OVS flows RACKSPACE® HOSTING | WWW.RACKSPACE.COM What’s next
  21. 21. #rackstackatl Patches, Blueprints https://review.openstack.org/#/c/88484/ (Neutron, Nova and Cells) https://blueprints.launchpad.net/neutron/+spec/nova-event-callback https://review.openstack.org/#/c/57517/ (noauth python-neutronclient) https://blueprints.launchpad.net/neutron/+spec/ovs-firewall-driver (OVS Firewall Driver) Projects https://github.com/rackerlabs/quark https://github.com/roaet/wafflehaus RACKSPACE® HOSTING | WWW.RACKSPACE.COM Links
  22. 22. #rackstackatl RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218 US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COMRACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM

×