Integrating OpenStack to  Existing Infrastructure         Cheng, Hui      freedomhui@gmail.com        Twitter:@freedomhui ...
AgendaBackground●   Who We Are●   Infrastructure & Platform●   ChallengesIntegration Challenges●   Network Deployment●   S...
Who Are We                                      Sina.com                                      • Largest infotainment web p...
Infrastructure & PlatformPhysical ServersTraditional OperationVirtualization Platform(IaaS)●VM Management System(VMMS) → S...
Sina App Engine• No. 1 Public PaaS Platform inChina launched in Nov 2009• PHP, Python, Java and RubySupport• Numbers160,00...
ChallengesSAE meets the majority of business needs, but does not coverall, especially for web gamesCustomers require full ...
Why Choose OpenStack  100% Python & Open Source
OpenStack Deployment                                    Rabbit                                    MySQL   dashboard       ...
Nova NetworkNetworking is the biggest challenges for IaaSNetwork Topology:•   VLAN•   FlatDHCP•   FlatDHCP & Multihost
Network Topology --- VLANCapability:• Accessibility of VMs within one tenant• Isolation of VMs from different tenants• VM ...
Network Topology(Flat)Capability:• Accessibility of all VMs in the fixed IP range• VM is able to access public network• VM...
Network Topology(Flat &                  Multihost)Capability:• Accessibility of all VMs in the fixed IP range• VM is able...
Security in OpenStackSecurity Group --- Layer 3 Filter          Static filters --- Layer 2 FilterRole-based firewall      ...
Security EnhancementSWS FilterPrevent Intranet Penetration• Intranet is the internal network outside of  OpenStackEgress f...
Security EnhancementSecurity Group VS SWS Filter                                  17
Load BalancerGoalsLoad Balance• Dispatch request                                  DNS Acceleration Design• Support multipl...
Load BalancerLayer 7 Load BalancerConsideration:1. dispatch request by Host header2. nginx module                         ...
Load BalancerLayer 4 Load BalancerConsideration:1. dispatch request by TCP port2. lvs + haproxy                           ...
Swift Evaluation   Extremely Durable and Highly Available   Superior Scalability   Linear Growth of Performance   Symm...
Swift Evaluation                                                       • 1 Zone = 1 Physical Server with 12x2T disk       ...
Swift Evaluation   Swift packages    Proxy Server   Account Server  Container Server    Object Server            Physical ...
Swift EvaluationPerformance issueCPU utilization rate up to 100% even without requestTesting environment:                 ...
RPC●   Biling & Monitoring                        Database                                                Client    Comput...
●   Kanyun: Monitoring system     Compute                     Worker      Network                            RDBMS        ...
RPC●   Dough:Billing system                                              Database                                         ...
Q&A      28
Upcoming SlideShare
Loading in...5
×

Integrating OpenStack To Existing Infrastructure

5,257

Published on

1. How to integrate OpenStack environment to our existing infrastructure.
2. How to efficiently interconnect the SAE & SWS, while preserving security properties and seamless connection.
3. The challenges we are facing when building & providing OpenStack-based public cloud service and how we solved it.

http://openstackconferencespring2012.sched.org/event/370f9d74a4e9e938a7f6f1e2af0958fe?iframe=yes&w=990&sidebar=no&bg=no#?iframe=yes&w=990&sidebar=no&bg=no#sched-body-outer

Published in: Technology
1 Comment
18 Likes
Statistics
Notes
No Downloads
Views
Total Views
5,257
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
380
Comments
1
Likes
18
Embeds 0
No embeds

No notes for slide

Integrating OpenStack To Existing Infrastructure

  1. 1. Integrating OpenStack to Existing Infrastructure Cheng, Hui freedomhui@gmail.com Twitter:@freedomhui 1 2012-04-19
  2. 2. AgendaBackground● Who We Are● Infrastructure & Platform● ChallengesIntegration Challenges● Network Deployment● Security Consideration● Load Balancer● Swift EvaluationOur Contributions● Billing● Monitoring
  3. 3. Who Are We Sina.com • Largest infotainment web portal in China • Provides various on-line services, like news, Finance, video, email, blog hosting, etc. • Operates first PaaS cloud computing platformSina Weibo• twitter-like microblog service• over 300m users• huge influence on Chinas society We are building a reliable, scalable and secure infrastructure and platform to support our business.
  4. 4. Infrastructure & PlatformPhysical ServersTraditional OperationVirtualization Platform(IaaS)●VM Management System(VMMS) → Sina WebService(SWS)●VMMS is private solution developed in-house●SWS is based on OpenStackApplication Platform(PaaS)●Virtual Host → Sina App Engine(SAE)●SAE provides both Public and Private Service.
  5. 5. Sina App Engine• No. 1 Public PaaS Platform inChina launched in Nov 2009• PHP, Python, Java and RubySupport• Numbers160,000+ developers200,000+ apps on SAE800 million page views per day20+ Services• SAE Cloud Storage Service is replaced by Swift• Deploy SAE on OpenStack
  6. 6. ChallengesSAE meets the majority of business needs, but does not coverall, especially for web gamesCustomers require full stack of cloud computingWe Choose OpenStack as our IaaS solution
  7. 7. Why Choose OpenStack 100% Python & Open Source
  8. 8. OpenStack Deployment Rabbit MySQL dashboard schedule nova-api nova-compute nova-compute nova-network nova-network keystone glanceSina SSO Swift
  9. 9. Nova NetworkNetworking is the biggest challenges for IaaSNetwork Topology:• VLAN• FlatDHCP• FlatDHCP & Multihost
  10. 10. Network Topology --- VLANCapability:• Accessibility of VMs within one tenant• Isolation of VMs from different tenants• VM is able to access public network• VM can be accessible from public network• Isolation between virtual network and internal network Drawback: • Pre-allocate network for future projects • Traffic bottleneck in the NAT gateway 12
  11. 11. Network Topology(Flat)Capability:• Accessibility of all VMs in the fixed IP range• VM is able to access public network• VM can be accessible from public network• Full isolation between virtual network and internal networkDrawback:Tenant isolation lessensTraffic bottleneck in the NAT gateway 13
  12. 12. Network Topology(Flat & Multihost)Capability:• Accessibility of all VMs in the fixed IP range• VM is able to access public network• VM can be accessible from public networkBonus:• Totally distributed architecture avoid single-point failure.• Multiple gateway eliminates NAT bottleneck• High throughout between OS regionsDrawback:• Tenant isolation lessens• Need security facility(SWS-filter) to protect intranet If security problems were solved, this would be our best choice! 14
  13. 13. Security in OpenStackSecurity Group --- Layer 3 Filter Static filters --- Layer 2 FilterRole-based firewall MAC, IP, and ARP spoofing protection One security group is a Role  Not configurableIngress filtering  Defined in /etc/libvirt/nwfilter/*.xml Target is the instance Implemented by ebtables Source can be CIDR or another group  ebtables -t nat --listImplemented by iptables See details: iptables -t filter -n -L Whitelist mechanism(ACCEPT rules) 15
  14. 14. Security EnhancementSWS FilterPrevent Intranet Penetration• Intranet is the internal network outside of OpenStackEgress filtering• Target is internal network• Source is instances in OpenStackImplementation• Whitelist mechanism(ACCEPT rules)• On the top of nova-filter-top Forward ChainRational• SWS filter is managed by cloud manager• Only explicit authorized packets can reach Internal network C• Packet should be controlled within Compute Node 16
  15. 15. Security EnhancementSecurity Group VS SWS Filter 17
  16. 16. Load BalancerGoalsLoad Balance• Dispatch request DNS Acceleration Design• Support multiple routing algorithm• Health check Smart DNSAcceleration• Reality: narrow bandwidth between ISPs• Building fiber channels from ISPs to pivot Public Network• Given the same endpoint within user’s ISP Telecom Unicom Mobile Others ISPIPv4 Shortage• Reality: dozens of public IPs support hundreds of VMs High speed fiber channel• IPv4 has been exhausted• IPv6 is not realistic yet in China Pivot 18
  17. 17. Load BalancerLayer 7 Load BalancerConsideration:1. dispatch request by Host header2. nginx module 19
  18. 18. Load BalancerLayer 4 Load BalancerConsideration:1. dispatch request by TCP port2. lvs + haproxy 20
  19. 19. Swift Evaluation Extremely Durable and Highly Available Superior Scalability Linear Growth of Performance Symmetric Architecture No Single-failure Simple & Reliable 21
  20. 20. Swift Evaluation • 1 Zone = 1 Physical Server with 12x2T disk GET abc.png • Write/Read applies quorum protocol PUT abc.png Load Balancer Zone1 Zone2 Zone3 Zone4 Zone5 Proxy Server Proxy Server Proxy Server Proxy Server Proxy ServerObject Server Object Server Object Server Object Server Object ServerContainer Server Container Server Container Server Container Server Container ServerAccount Server Account Server Account Server Account Server Account Server 22
  21. 21. Swift Evaluation Swift packages Proxy Server Account Server Container Server Object Server Physical Deployment Storage Nodes OS installation sda sdb sdc sdd sdk raid 1 ……disk1 disk2 disk3 disk4 disk5 disk12 23
  22. 22. Swift EvaluationPerformance issueCPU utilization rate up to 100% even without requestTesting environment: Audit:Nodes: 5 x Dell R510 swift-account-auditor : 1.5mCPU: Intel® Xeon® E5360 swift-account-replicator: 9.5mMemory: 12GBReplica: 3 swift-container-auditor: 8.4m swift-container-replicator: 9.3mNo. of Objects: 150,000,000 swift-container-updater: 19.0mNo. of Accounts: 120,000No. of Containers: 160,000 swift-object-updater: 0.1 s swift-object-replicator: 10.5 hours swift-object-auditor: 48.3 hoursResult:Periodic scanning all partitions, calculating checksum and synchronization 24
  23. 23. RPC● Biling & Monitoring Database Client Compute Network RDBMS Dashboard Storage Monitoring Billing (Metering) 25 NoSQL
  24. 24. ● Kanyun: Monitoring system Compute Worker Network RDBMS Dashboard Storage Worker Retrieve usage info API daemon Billing Aggregator Responds to client Calculates/stores request metrics http://github.com/lzyeval/kanyun 26 NoSQL
  25. 25. RPC● Dough:Billing system Database Client Compute Network RDBMS Dashboard Storage Collector Monitoring Farmer API daemon (Metering) Dispatch jobs Subscribe or Collector unsubscribe products / Check status / Query info Retrieve usage / http://github.com/lzyeval/dough 27 Create purchases
  26. 26. Q&A 28
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×