Openstack summit 2015
•
1 like•394 views
With more than 140 million users, KakaoTalk is the most popular mobile messaging platform in South Korea. The team at daumkakao has been using OpenStack with the intention for tranforming the current legacy infrastructure into scale out based cloud to build and offer new services for its users. In this session, we'd like to share our experiences with the OpenStack community, specifically in regards to meeting our needs for networking with Neutron.OpenStack Neutron offers a lot of methods to implement networking for VMs and containers. For production operations, VM migration can be a common activity to manage resources and improve uptime. It's not hard using shared storage like Ceph, but network settings, such as IP addresses need to be preserved. With a shared storage environment, an image can be attached anywhere inside of a data center, but a service IP for a virtual machine is different story. And when you don't use the floating IPs, keeping the same IP across a data center-wide set of VLANs is hard job.To maintain a virtual machine's IP settings and balance IPs between VLANS, we tried several options including overlay, SDN, and NFV technologies. In the end we came to use a route-only network for our virtual machine networks, leveraging technology like Quagga for RIP, OSPF BGP integrated with Neutron.
Recommended
Recommended
More Related Content
What's hot
What's hot (19)
Similar to Openstack summit 2015
Similar to Openstack summit 2015 (20)
Recently uploaded
Recently uploaded (20)
Openstack summit 2015
- 1. L.T.H. SDN, without SDN Andrew Yongjoon Kong
- 3. The Peaceful operation When we’re running out of resources( cpu, memory, disk ), Just add new( or additional ) resources to existing one. System team Network teamCMDB API New servers New servers New servers New servers nkaos (baremetal provisioner) provisioned servers provisioned servers provisioned servers provisioned server Chef server Our Team NSDB Central monitoring tree switches, router, vlans
- 4. The Growth(I) VM creation speed is accelerating
- 5. The Growth(II) Spend more than 45M krane ( $45,000) per month – this also increased. 1 krane = 1 Won ( $0.001) • Using similar pricing with AWSEC2 • Network/Diskusage notincluded
- 6. The Growth(III) Growth is accelerating - No. of Engineer is growing - New Pilot services or experiments are growing. - The resources depletion speed is accelerating à this simply make more work to resource management teams System team Network team New servers New servers New servers New servers NKAOS CMDB API New servers New servers New servers New servers Chef server Our Team NSDB Central monitoring tree New servers New servers New servers New servers New servers New servers New servers New servers
- 7. The Growth(IV) Scale, The only driving force disrupt everything. System team Network teamCMDB API NSDB Central monitoring tree Chef server New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers Chef server New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers Chef server New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers NKAOS Our Team
- 8. Chef server New servers New servers New servers New servers New servers New servers New servers New servers The Growth – Lesson learned Growth doesn’t come alone – Infra growth includes scale-up , scale-out as well – Scale-up includesthese • Add Server, Storage, Switches • Add more power facility to supply juice fluently • This is not that difficult. – Scale-out include these • Add New Datacenters, New Availability Zones • This is nightmare! This leads radical changes over everything – The way of preparing, provisioning – The way of monitoring, logging, developing New servers New servers New servers New servers New servers New servers New servers New servers Chef server New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers Chef server New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers Chef server New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers Chef server New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers Chef server New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers Chef server New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers New servers
- 9. The Growth – Lesson learned, Openstack (2) Resources for Openstack finally comes to be exhausted – CPU, Memory, Storage always experience shortages. – They have skewness. – Sometimes, CPU depleted. Sometimes, Storage depleted. • All resources are able to be re-balanced. • you can migration clients’ VM ( image , volume ) – IP is also Resources. • Very limited than our expectations – No of IP counts is limited. – Location of IP also is limited. • Managing these Resources is getting tougher issue.
- 10. Zone1 (a.k.a Rack) Neutron Network We’ve been using Provider Network (VLAN) – ML2 plugin – From OVS à LinuxBridge. – Network Team plan/setup networks (the VLAN, IP[subnet], Gateways) – Mapping availability zone / Neutron Network to that Physical networks VLAN.1 eth0 eth1 brqxxx eth1.1 tapxxx vm eth0 K V M Hypervisor Zone2 (a.k.a Rack) VLAN.2 Zone3 (a.k.a Rack) VLAN.3
- 11. Zone1 1 CPU 1 storage No IP Left Resource Imbalance After Running multiple Available Zones – Experiencing resource imbalance between zones, naturally – Filter Scheduling won’t helpful. – Migration is a proper solution. ( add extra resource is better ) VLAN.1 Zone2 No CPU No Storage 1 IP VLAN.2 Zone3 VLAN.3 Hey Openstack, Create 1 VM ( 1cpu, 1 IP, 1 Storage) openstack scheduler x
- 12. Resource Imbalance & Remedies Develop Network Count filter – Check Remaining IP count for each zone, treat ip count as resource – Select the zone which have more ip count – but experiencing harder issue • Setup more 2Vlan ( and also trunking ) on same ethernet • leading heterogeneous policy which cause complex configurations • Still, Migration VM through zones with ip unchanged is not possible. Zone1 VLAN.1eth0 eth1 brqxxx eth1.1 tapxxx vm1 eth0 K V M Hypervisor brqYYY eth1.10 vm1 eth0 VLAN trunk VLAN.10
- 13. broadcast domain2 Rationale Rethinking about Connectivity Application TCP IPv4 ethernet driver broadcast domain ARP Table SRC IP mac eth0 Router IP mac eth0 Application TCP IPv4 ethernet driver ARP Table dest IP mac eth0 Router IP mac eth0 broadcast termination A.K.A Router same subnet different subnet client destination
- 14. Rationale Rethinking about Connectivity (Overlay) – it solve remote link layer separation issue. – Still have issue with IP management Application TCP IPv4 ethernet driver broadcast domain ARP Table SRC IP mac eth0 Router IP mac eth0 Application TCP IPv4 ethernet driver ARP Table dest IP mac eth0 Router IP mac eth0 tun nel broadcast domain tun nel
- 15. Remedy , Version 2.0 we need to thinks of those requirement – IP movement inter-rack, inter-zone, inter-dc(?) – IP resource imbalance – Fault Resilience – Dynamically check status of network – Simple IP Resource Planning and Management
- 16. Router We thinks Router as best candidate – it dynamically detect and exchange changes. – it is highly distributed. – it have HA – the issue is that most of time routing is done in ranges
- 17. Finally, Come to route only IP Generally, Known as /32 network. – No L2 (link) consideration needed anymore ( no subnet ) – With Dynamic Routing Protocol, it move every where. – Simple IP planning ( Just think of IP ranges ) – It’s very Atomic Resource, it keeps its IP after migration through zones 10.0.0.1 / 32 or IP 10.0.0.1 netmask 255.255.255.255
- 18. How it setup 1. install nova/neutron agent. 2. create neutron network ( name: freenet, subnet: 10.10.100.0/24) eth1 eth0 Compute node nova-compute neutron- linuxbridge- agent neutron-dhcp- agent dhcp-server process 10.10.100.1
- 19. How it setup 1. install nova/neutron agent. 2. create neutron network ( name: freenet, subnet: 10.10.100.0/24) 3. user create VM eth1 eth0 Compute node nova-compute neutron- linuxbridge- agent neutron-dhcp- agent dhcp-server process 10.10.100.1 linux bridge vm IP:10.10.100.2/32 GW: 10.10.100.1 Controller
- 20. How it work 1. install nova/neutron agent. 2. create neutron network ( name: freenet, subnet: 10.10.100.0/24) 3. user create VM 4. update Routing(with Dynamic routing protocol) eth1 eth0 Compute node nova-compute neutron- linuxbridge- agent neutron-dhcp- agent dhcp-server process 10.10.100.1 linux bridge vm IP:10.10.100.2/32 GW: 10.10.100.1 Controller 192.1.1.201 Routing Table Default GW 192.168.1.1 eth1 Host Route dest 10.10.100.2/32 to 10.10.100.1 Routing Table 1 10.100.10.2/32 via 192.1.1.201 advertising: via Dynamic Routing Protocol 192.1.1.202
- 21. Phase 1 Use RIP and OSP – Heterogeneous setting will be burden – Using Default GW as eth1 even for compute node. Management and service network mixed. eth1 eth0 Compute node nova-compute neutron- linuxbridge- agent neutron-dhcp- agent dhcp-server process 10.10.100.1 linux bridge vm IP:10.10.100.2/32 GW: 10.10.100.1 Controller 192.1.1.201 Routing Table Default GW 192.168.1.1 eth1 Host Route dest 10.10.100.2/32 to 10.10.100.1 Routing Table 1 10.100.10.2/32 via 192.1.1.201 RIP 192.1.1.202 OSPF
- 22. Phase 2 Use BGP and switch namespace – Isolating vm’s traffic using switch namespace. – adoting same dynamic routing scheme to compute node eth1 Compute node nova-compute neutron- linuxbridge- agent neutron-dhcp- agent dhcp-server process 10.10.100.1 linux bridge vm IP:10.10.100.2/32 Controller 192.1.1.201 Routing Table Default GW 192.168.1.1 eth1 Host Route dest 10.10.100.2/32 to 10.10.100.1 Routing Table 1 10.100.10.2/32 via 192.1.1.201 iBGP 192.1.1.202 eBGP Switch Namespace global name space Routing Table Default GW x.x.x.x eth0 eth0
- 23. Phase 2 Use BGP and switch namespace – Isolating vm’s traffic using switch namespace. – adoting same dynamic routing scheme to compute node eth1 Compute node nova-compute neutron- linuxbridge- agent neutron-dhcp- agent dhcp-server process 10.10.100.1 linux bridge vm IP:10.10.100.2/32 Controller 192.1.1.201 Routing Table Default GW 192.168.1.1 eth1 Host Route dest 10.10.100.2/32 to 10.10.100.1 Routing Table 1 10.100.10.2/32 via 192.1.1.201 iBGP 192.1.1.202 eBGP Switch Namespace global name space Routing Table Default GW x.x.x.x eth0 eth0
- 24. What we solved? Compute node2 nova-compute neutron- linuxbridge-agent neutron-dhcp- agent linux bridge Switch Namespace globalname space Compute node1 nova-compute neutron- linuxbridge-agent neutron-dhcp- agent linux bridge Switch Namespace globalname space AZ1 Compute node2 nova-compute neutron- linuxbridge-agent neutron-dhcp- agent linux bridge Switch Namespace globalname space Compute node1 nova-compute neutron- linuxbridge-agent neutron-dhcp- agent linux bridge Switch Namespace globalname space AZ2 Compute node2 nova-compute neutron- linuxbridge-agent neutron-dhcp- agent linux bridge Switch Namespace globalname space Compute node1 nova-compute neutron- linuxbridge-agent neutron-dhcp- agent linux bridge Switch Namespace globalname space AZ3 tor1 tor2 tor3 vm10.10.100.2/32 Routing Table 1 10.100.10.2/32 via tor1 rt1 rt2 Routing Table 1 10.100.10.2/32 via RT1 rt3 rt4 rt5 rt6 Routing Table 1 10.100.10.2/32 via RT3 Routing Table 1 10.100.10.2/32 via tor2
- 25. What we solve? Simple IP planning – only IP ranges matter. (no more VLAN, IP subnet, Router planning) Resource imbalancing – No chance of IP imbalancing. Fault Resilience – If one router gone, it propagated by Dynamic routing protocol to other router Distributed – deciding routing path is very distributed. No single point of failure. – scale out nature.
- 26. What we still have to solve? Still many issue – Apply this to physical server – Making Router setup by API ( REST, RPC) using seed BGP( only advertising) – ACL propagation using API ( e.g. Flowspec) – Shared storage base service
- 27. One More Thing
- 28. KField Network
- 29. CI