Your SlideShare is downloading. ×
Neutron scale
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Neutron scale

562

Published on

In this session, we will discuss the operational issues that Rackspace has encountered during and after implementing Neutron at a large scale. Neutron at scale required a significant amount of …

In this session, we will discuss the operational issues that Rackspace has encountered during and after implementing Neutron at a large scale. Neutron at scale required a significant amount of development and operations effort, some of which resulted in deviations from upstream code. Finally, our team would like to discuss our solutions and our upstream differences for Neutron and OpenStack that we believe are necessary so that it can be more performant at scale.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
562
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
41
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. #rackstackatl Justin Hammond - Developer Andy Hill - Systems Engineer Chad Norgan - Systems Engineer Neutron at Scale
  • 2. #rackstackatl Rackspace is early in Neutron implementation Migrating from older versions of Quantum/Melange used since the launch of our public cloud Scope of this talk is primarily Nova ⬄ Neutron interaction and the challenges we faced deploying Neutron at scale Scope of the Talk
  • 3. #rackstackatl Tens of thousands of compute nodes Hundreds of thousands of instances Most instances have two or more ports RACKSPACE® HOSTING | WWW.RACKSPACE.COM What we mean when we say “at scale”
  • 4. #rackstackatl Maintain backwards compatibility with existing products Neutron will be the ultimate authoritative source for network state IP Address Management (IPAM) Modular network drivers so Neutron can service heterogeneous port types Enable new products to easily integrate into our public cloud offering RACKSPACE® HOSTING | WWW.RACKSPACE.COM Implementation Requirements
  • 5. #rackstackatl Quark Plugin: Open source plugin for Neutron v2 API with IPAM Custom database migration from Melange/Quantum->Neutron/Quark Wafflehaus middleware collection RACKSPACE® HOSTING | WWW.RACKSPACE.COM Implementation Details
  • 6. #rackstackatl RACKSPACE® HOSTING | WWW.RACKSPACE.COM Rackspace’s Neutron Implementation Neutron-api nodes running quark plugin with wafflehaus Active/Passive database with slave Active/Passive Load Balancers
  • 7. #rackstackatl Wafflehaus is a middleware for some specific Rackspace requirements Very simple way to minimize upstream diffs Upstream efforts better spent on work that benefits the broader community RACKSPACE® HOSTING | WWW.RACKSPACE.COM Wafflehaus Overview
  • 8. #rackstackatl RACKSPACE® HOSTING | WWW.RACKSPACE.COM Wafflehaus - “The API Mullet” Business logic in the front, party in the back
  • 9. #rackstackatl Does the request body contain particular UUIDs RACKSPACE® HOSTING | WWW.RACKSPACE.COM Wafflehaus Explained Wafflehaus middlewares Would this request violate policy?Add this tag to the request header Quark plugin Neutron-api API Request
  • 10. #rackstackatl RACKSPACE® HOSTING | WWW.RACKSPACE.COM Wafflehaus Explained API Request Wafflehaus middlewares Quark plugin Neutron-api
  • 11. #rackstackatl Calls to Keystone Melange/Quantum Neutron (trunk) Wafflehaus + no-auth Build 0 5 per port 0 Delete 0 5 per port 0 Info Cache Update 0 LOTS 0 TOTAL 0 TOO MANY 0 RACKSPACE® HOSTING | WWW.RACKSPACE.COM
  • 12. #rackstackatl Wafflehaus and No-Auth Middleware Neutron-api with wafflehaus PTR for 10.1.2.3? PTR at compute.trusted.domain A for compute.trusted.domain? A at 10.1.2.3 DNS Server RACKSPACE® HOSTING | WWW.RACKSPACE.COM API Request x-forwarded-for
  • 13. #rackstackatl [composite:neutronapi_v2_0] use = call:neutron.auth:pipeline_factory noauth = dns_filter request_id catch_errors extensions neutronapiapp_v2_0 keystone = request_id catch_errors authtoken keystonecontext extensions neutronapiapp_v2_0 [filter:dns_filter] paste.filter_factory = wafflehaus.dns_filter.whitelist:filter_factory whitelist = trusted.domain enabled = true RACKSPACE® HOSTING | WWW.RACKSPACE.COM Wafflehaus Explained
  • 14. #rackstackatl Call Volume Before & After
  • 15. #rackstackatl Call Volume Before & After
  • 16. #rackstackatl Nova caches a copy of the instance’s network information (info cache) Cache is refreshed on instance operations which reach out to Neutron Callback system is needed RACKSPACE® HOSTING | WWW.RACKSPACE.COM On Info Cache Updates
  • 17. #rackstackatl Happens on nova-compute restart Also happens every heal_instance_info_cache_interval (default 1m) Currently 6 calls to Neutron per port Set heal_instance_info_cache_interval=0 RACKSPACE® HOSTING | WWW.RACKSPACE.COM On Info Cache Updates (continued)
  • 18. #rackstackatl nova-cells and Info Cache Updates Child cells periodically sync with parent cells Migration to Neutron exposed upstream bug that was corrected in rpc network api, not neutron Cache updates were sent from child cells to global cells faster than global cells could process Delays other messages from being processed
  • 19. #rackstackatl Callback system between nova and neutron Read-only database slave usage Cells support Nova & Neutron: Fewer calls that do more (e.g., 1 API call, many ports) RACKSPACE® HOSTING | WWW.RACKSPACE.COM What’s needed
  • 20. #rackstackatl Publicly expose neutron Security Groups extension support through OVS flows RACKSPACE® HOSTING | WWW.RACKSPACE.COM What’s next
  • 21. #rackstackatl Patches, Blueprints https://review.openstack.org/#/c/88484/ (Neutron, Nova and Cells) https://blueprints.launchpad.net/neutron/+spec/nova-event-callback https://review.openstack.org/#/c/57517/ (noauth python-neutronclient) https://blueprints.launchpad.net/neutron/+spec/ovs-firewall-driver (OVS Firewall Driver) Projects https://github.com/rackerlabs/quark https://github.com/roaet/wafflehaus RACKSPACE® HOSTING | WWW.RACKSPACE.COM Links
  • 22. #rackstackatl RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218 US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COMRACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM

×