Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Integrating OpenStack to Existing infrastructure


Published on

Hui Cheng's Thurs, 1 pm session

Published in: Technology
  • Be the first to comment

Integrating OpenStack to Existing infrastructure

  1. 1. Integrating OpenStack to Existing Infrastructure Cheng, Hui 1 2012-04-19
  2. 2. AgendaBackground● Who We Are● Infrastructure & Platform● ChallengesIntegration Challenges● Network Deployment● Security Consideration● Load Balancer● Swift EvaluationOur Contributions● Billing● Monitoring 2
  3. 3. Who Are We • Largest infotainment web portal in China • Provides various on-line services, like news, Finance, video, email, blog hosting, etc. • Operates first PaaS cloud computing platformSina Weibo• twitter-like microblog service• over 300m users• huge influence on Chinas society We are building a reliable, scalable and secure infrastructure and platform to support our business. 3
  4. 4. Infrastructure & PlatformPhysical ServersTraditional OperationVirtualization Platform(IaaS)●VM Management System(VMMS) → Sina WebService(SWS)●VMMS is private solution developed in-house●SWS is based on OpenStackApplication Platform(PaaS)●Virtual Host → Sina App Engine(SAE)●SAE provides both Public and Private Service.●Proved to be Efficient and Robust 4
  5. 5. Sina App Engine• No. 1 Public PaaS Platform inChina launched in Nov 2009• PHP, Python, Java and RubySupport• Numbers160,000+ developers200,000+ apps on SAE800 million page views per day20+ Services• SAE Cloud Storage Service is replaced by Swift• Deploy SAE on OpenStack 5
  6. 6. ChallengesSAE meets the majority of business needs, but does not coverall, especially for web gamesCustomers require full stack of cloud computingWe Choose OpenStack as our IaaS solution 8
  7. 7. Why Choose OpenStack 100% Python & Open Source 9
  8. 8. OpenStack Deployment Rabbit MySQL dashboard schedule nova-api nova-compute nova-compute nova-network nova-network keystone glanceSina SSO Swift 10
  9. 9. Nova NetworkNetworking is the biggest challenges for IaaSNetwork Topology:• VLAN• FlatDHCP• FlatDHCP & Multihost 11
  10. 10. Network Topology --- VLANCapability:• Accessibility of VMs within one tenant• Isolation of VMs from different tenants• VM is able to access public network• VM can be accessible from public network• Isolation between virtual network and internal network Drawback: • Pre-allocate network for future projects • Traffic bottleneck in the NAT gateway 12
  11. 11. Network Topology(Flat)Capability:• Accessibility of all VMs in the fixed IP range• VM is able to access public network• VM can be accessible from public network• Full isolation between virtual network and internal networkDrawback:Tenant isolation lessensTraffic bottleneck in the NAT gateway 13
  12. 12. Network Topology(Flat & Multihost)Capability:• Accessibility of all VMs in the fixed IP range• VM is able to access public network• VM can be accessible from public networkBonus:• Totally distributed architecture avoid single-point failure.• Multiple gateway eliminates NAT bottleneck• High throughout between OS regionsDrawback:• Tenant isolation lessens• Need security facility(SWS-filter) to protect intranet If security problems were solved, this would be our best choice! 14
  13. 13. Security in OpenStackSecurity Group --- Layer 3 Filter Static filters --- Layer 2 FilterRole-based firewall MAC, IP, and ARP spoofing protection One security group is a Role  Not configurableIngress filtering  Defined in /etc/libvirt/nwfilter/*.xml Target is the instance Implemented by ebtables Source can be CIDR or another group  ebtables -t nat --listImplemented by iptables See details: iptables -t filter -n -L Whitelist mechanism(ACCEPT rules) 15
  14. 14. Security EnhancementSWS FilterPrevent Intranet Penetration• Intranet is the internal network outside of OpenStackEgress filtering• Target is internal network• Source is instances in OpenStackImplementation• Whitelist mechanism(ACCEPT rules)• On the top of nova-filter-top Forward ChainRational• SWS filter is managed by cloud manager• Only explicit authorized packets can reach Internal network C• Packet should be controlled within Compute Node 16
  15. 15. Security EnhancementSecurity Group VS SWS Filter 17
  16. 16. Load BalancerDesignLoad Balance• Dispatch request DNS Acceleration Design• Support multiple routing algorithm• Health check Smart DNSAcceleration• Reality: narrow bandwidth between ISPs• Building fiber channels from ISPs to pivot Public Network• Given the same endpoint within user’s ISPIPv4 Shortage Telecom Unicom Mobile Others ISP• Reality: dozens of public IPs support hundreds of VMs High speed fiber channel• IPv4 has been exhausted• IPv6 is not realistic yet in China Pivot 18
  17. 17. Load BalancerLayer 7 Load BalancerConsideration:1. dispatch request by Host header2. nginx module 19
  18. 18. Load BalancerLayer 4 Load BalancerConsideration:1. dispatch request by TCP port2. lvs + haproxy 20
  19. 19. Swift Evaluation Extremely Durable and Highly Available Superior Scalability Linear Growth of Performance Symmetric Architecture No Single-failure Simple & Reliable 21
  20. 20. Swift Evaluation • 1 Zone = 1 Physical Server with 12x2T disk GET abc.png • Write/Read applies quorum protocol PUT abc.png Load Balancer Zone1 Zone2 Zone3 Zone4 Zone5 Proxy Server Proxy Server Proxy Server Proxy Server Proxy ServerObject Server Object Server Object Server Object Server Object ServerContainer Server Container Server Container Server Container Server Container ServerAccount Server Account Server Account Server Account Server Account Server 22
  21. 21. Swift Evaluation Swift packages Proxy Server Account Server Container Server Object Server Physical Deployment Storage Nodes OS installation sda sdb sdc sdd sdk raid 1 ……disk1 disk2 disk3 disk4 disk5 disk12 23
  22. 22. Swift EvaluationPerformance issueCPU utilization rate up to 100% even without requestTesting environment: Audit:Nodes: 5 x Dell R510 swift-account-auditor : 1.5mCPU: Intel® Xeon® E5360 swift-account-replicator: 9.5mMemory: 12GBReplica: 3 swift-container-auditor: 8.4m swift-container-replicator: 9.3mNo. of Objects: 150,000,000 swift-container-updater: 19.0mNo. of Accounts: 120,000No. of Containers: 160,000 swift-object-updater: 0.1 s swift-object-replicator: 10.5 hours swift-object-auditor: 48.3 hoursResult:Periodic scanning all partitions, calculating checksum and synchronization 24
  23. 23. RPC● Biling & Monitoring Database Client Compute Network RDBMS Dashboard Storage Monitoring Billing (Metering) 25 NoSQL
  24. 24. ● Kanyun: Monitoring system Compute Worker Network RDBMS Dashboard Storage Worker Retrieve usage info API daemon Billing Aggregator Responds to client Calculates/stores request metrics 26 NoSQL
  25. 25. RPC● Dough:Billing system Database Client Compute Network RDBMS Dashboard Storage Collector Monitoring Farmer API daemon (Metering) Dispatch jobs Subscribe or Collector unsubscribe products / Check status / Query info Retrieve usage / 27 Create purchases
  26. 26. Q&A 28