Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Openstack days taiwan 2016 0712

592 views

Published on

GMO Internet Inc., public cloud arch for ConoHa cloud, z.com cloud, GMO Apps Cloud

Published in: Internet
  • Be the first to comment

Openstack days taiwan 2016 0712

  1. 1. 1 ~ Architecture of our public clouds ~ OpenStack Days Taiwan Jul 12, 2016 Naoto Gohko (@naoto_gohko) GMO Internet, Inc. How is GMO Ineternet using OpenStack for Public Cloud Slide URL http://www.slideshare.net/chroum/openstack-days-taiwan-2016-0712-public-cloud-arch ConoHa public cloud (lang zh) https://www.conoha.jp/zh/ ConoHa public cloud (lang en) https://www.conoha.jp/en/
  2. 2. 2 Public Clouds We are offering multiple public cloud services.
  3. 3. 3 Physical Servers Running VMPhysical Server 1508 25294 Created VM Running Infrastructure (2015/10) 137223
  4. 4. 4 OpenStack service development team
  5. 5. 5 Cloud service development team: (abount 30 people) – OpenStack Neutron team: 4 people • Neutron driver / modification / engineering – Cloud API development team: 5 people • Public API validation program • OpenStack modification / scaduler programing / keystone – Cloud Infra. development team: 11 people • Security engineering / glance driver / cinder driver / nova additional extensions / construction of OpenStack infra. – Applicatoin cloud service development team: 5 people • Billing engineering / staff tools / GMO AppsCloud web GUI Additional engineering team: many people (30 ~) – QA Team / Server Engineering Team / GUI development Team – Network Engineering Team / SaaS development Team – CRM backend and billing Team Cloud service development team: Now(2016)
  6. 6. 6 Cloud service development team: Office(2016) #1 Neutron Team And Cloud API Team Cloud Infra. Team And AppsCloud Team
  7. 7. 7 Cloud service development team: Office(2016) #2 Neutron Team And Cloud API Team Cloud Infra. Team And AppsCloud Team
  8. 8. 8 Limied number of people. But, we have to run a lot of OpenStack service clusters.
  9. 9. 9 Service developmemt history by OpenStack
  10. 10. 10 Swift cluster GMO Internet, Inc.: VPS and Cloud services Onamae.com VPS (2012/03) : http://www.onamae-server.com/ Forcus: global IPs; provided by simple "nova-network" tenten VPS (2012/12) http://www.tenten.vn/ Share of OSS by Group companies in Vietnam ConoHa VPS (2013/07) : http://www.conoha.jp/ Forcus: Quantam(Neutron) overlay tenant network GMO AppsCloud (2014/04) : http://cloud.gmo.jp/ OpenStack Havana based 1st region Enterprise grade IaaS with block storage, object storage, LBaaS and baremetal compute was provided Onamae.com Cloud (2014/11) http://www.onamae-cloud.com/ Forcus: Low price VM instances, baremetal compute and object storage ConoHa Cloud (2015/05/18) http://www.conoha.jp/ Forcus: ML2 vxlan overlay, LBaaS, block storage, DNSaaS(Designate) and original services by keystone auth OpenStack Diablo on CentOS 6.x Nova Keystone Glance Nova network Shared codes Quantam OpenStack Glizzly on Ubuntu 12.04 Nova Keystone Glance OpenStack Havana on CentOS 6.x Keystone Glance Cinder Swift Swift Shared cluster Shared codes KeystoneGlance Neutron Nova Swift Baremetal compute Nova Ceilometer Baremetal compute Neutron LBaaS ovs + gre tunnel overlay Ceilometer Designate SwiftOpenStack Juno on CentOS 7.x NovaKeystone Glance Cinder Ceilometer Neutron LBaaS GMO AppsCloud (2015/09/27) : http://cloud.gmo.jp/ 2nd region by OpenStack Juno based Enterprise grade IaaS with High IOPS Ironic Compute and Neutron LBaaS Upgrade Juno GSLB Swift Keystone Glance CinderCeilometer Nova Neutron Ironic LBaaS
  11. 11. 11
  12. 12. 12 OpenStack Swift: shared cluster
  13. 13. 13 Swift Hardware: Object nodes • Boot: SSD x2 • HDD: 4TB x12 • E3-1230 v3 @ 3.30GHz • Memory 16GB • 10GbE x2 (SFP+) (Intel NIC) ASUSTeK COMPUTER INC. RS300-H8-PS12
  14. 14. 14 Hardware: LVS-DSR and reverse-proxy(Layer7) nodes • Boot: SSD x2 • E3-1230 v3 @ 3.30GHz • Memory 16GB • 10GbE NIC x1 (Intel NIC) Supermicro microblade 8 blade nodes type
  15. 15. 15 Hardware: swift-proxy nodes • Boot: HDD x6 (1.7TB) – Ceilometer Log disk – (Swift all request billing data) • E5620 @ 2.40GHz x2 CPU • Memory 64GB • NIC: 10GbE SFP+ x2(Intel NIC) System x3550 M3 (old IBM) Hardware: account/container-server nodes • Boot: HDD x2 • Account/Container storage: SSD x2 • E5620 @ 2.40GHz x2 CPU • Memory 64GB • NIC: 10GbE SFP+ x2(Intel NIC) System x3550 M3 (old IBM)
  16. 16. 16 Swift cluster (Havana to Juno upgrade) SSD storage: container/account server at every zone
  17. 17. 18 swift proxy keystone OpenStack Swift cluster (5 zones, 3 copy) swift proxy keystone LVS-DSRLVS-DSR HAProxy(SSL)HAProxy(SSL) Xeon E3-1230 3.3GHz Memory 16GB Xeon E3-1230 3.3GHz Memory 16GB Xeon E5620 2.4GHz x 2CPU Memory 64GB swift objects swift objects Xeon E3-1230 3.3GHz swift account swift container Xeon E5620 2.4GHz x 2CPU Memory 64GB, SSD x 2 swift objects swift objects Xeon E3-1230 3.3GHz swift account swift container Xeon E5620 2.4GHz x 2CPU Memory 64GB, SSD x 2 swift objects swift objects Xeon E3-1230 3.3GHz swift account swift container Xeon E5620 2.4GHz x 2CPU Memory 64GB, SSD x 2 swift objects swift objects Xeon E3-1230 3.3GHz swift account swift container Xeon E5620 2.4GHz x 2CPU Memory 64GB, SSD x 2 swift objects swift objects Xeon E3-1230 3.3GHz swift account swift container Xeon E5620 2.4GHz x 2CPU Memory 64GB, SSD x 2
  18. 18. 19 swift objects swift objects swift objects swift objects swift objects swift objects swift objects swift objects swift objects swift objects swift proxy keystone Havana AppsCloud swift proxy keystone Grizzly ConoHa Havana To Juno swift account swift container swift account swift container swift account swift container swift account swift container swift account swift container swift proxy keystone Juno ConoHa swift proxy keystone Juno AppsCloud Swift cluster: multi-auth and multi-endpoint swift proxy keystone Juno Z.com
  19. 19. 20 Swift shared cluster: ex) Swift-Proxy [filter:keystone] reseller_prefix = nc_ [filter:ceilometer] reseller_prefix = zc_ [filter:keystone] reseller_prefix = gac_ [filter:keystone] reseller_prefix = [filter:keystone] ## reseller_prefix = [filter:keystone] reseller_prefix = zc_ [filter:ceilometer] reseller_prefix = zc_ Swift-Proxy [filter:keystone] reseller_prefix = nc_ [filter:ceilometer] reseller_prefix = zc_ [filter:keystone] reseller_prefix = gac_ [filter:keystone] reseller_prefix = [filter:keystone] ## reseller_prefix = [filter:keystone] reseller_prefix = zc_ [filter:ceilometer] reseller_prefix = zc_ << account-server << container-server << object-server ( 2 x 5 nodes ) reverse-proxyreverse-proxyLVS-DSR LVS-DSR keystone ConoHa VPS cluster (Grizzly) https://swift-url/<tenant-id>/ keystone AppsCloud cluster (Havana) https://swift-url/AUTH_<tenant-id>/ keystone AppsCloud cluster (Juno) https://gac-swift-url/gac_<tenant-id>/ keystone ConoHa Cloud cluster (Juno) https://nc-swift-url/nc_<tenant-id>/ keystone Z.com Cloud (Juno) https://zc-swift-url/zc_<tenant-id>/ the Internet
  20. 20. 21 OpenStack history of computing environment
  21. 21. 22 Oname.com VPS(Diablo) • Service XaaS model: – VPS (KVM, libvirt) • Network: – 1Gbps • Network model: – Flat-VLAN (Nova Network), without floting IP(no L3) – IPv4 only • Public API – None (only web-panel) • Glance – Public image only. OpenStack service: Onamae.com VPS(Diablo)
  22. 22. 23 ConoHa(Grizzly) • Service XaaS model: – VPS + Private networks (KVM + ovs) • Network model: – Flat-VLAN + Quantam ovs-GRE overlay – IPv6/IPv4 dualstack • Network: – 10GE wired(10GBase-T) • Public API: None (only web) • Glance – Only Public image • Cinder: None • ObjectStorage – Swift (After Havana) OpenStack service: ConoHa(Grizzly, 2013/07)
  23. 23. 24 Grizzly • Quantam Network: – It was using the initial version of the Open vSwitch full mesh GRE-vlan overlay network with LinuxBridge Hybrid  But When the scale becomes large, Localization occurs to a specific node of the communication of the GRE-mesh-tunnel (with under cloud network(L2) problems) (Broadcast storm?) OpenStack service: ConoHa(Grizzly)
  24. 24. 25 • Service XaaS model: – KVM compute + Private VLAN networks + Cinder + Swift • Network: – 10Gbps wired(10GBase SFP+) • Network model: – IPv4 Flat-VLAN + Neutron LinuxBridge(not ML2) + Cisco Nexsu L2 sw/port driver – Brocade ADX L4-LBaaS original driver • Public API – Provided the public API • Ceilometer (Billing) • Glance : Provided(GlusterFS) • Cinder : HP 3PAR(Active-Active Multipath original) + NetApp • ObjectStorage : Swift cluster • Bare-Metal Compute – Modifiyed cobbler bare-metal deploy driver – Cisco Nexsus switch bare-metal networking driver (L2 tenant NW) OpenStack service: GMO AppsCloud(Havana)
  25. 25. 26 OpenStack service: GMO AppsCloud model compute vm NIC Vlan network bridge NIC vlan tap vNIC Vlan network vNIC bridge vlan tap compute NIC bridge NIC vlan bridge vlan public network Neutron LinuxBridge model(very Fast, simple is Best)  this cloud is optimized services for the GAME server.
  26. 26. 27 Cisco Nexsus L2 sw/Port manage driver(self made) • L2 resource is limited / SW CPU – MAC ADDRESS – VLAN per Network – VLAN per Port Allowed VLAN to trunked port is allowed only VLAN to be used in LinuxBridge in VM/Baremetal Compute node. – Baremetal : link aggregation port – Port discovery using by lldp • Cisco Nexsus NX-OS – Server: LACP : port-Channel Active-Active link aggreration Fully redundant server (Act-Act link aggreration) Nexus 5k’s (VPC) Nexus 2k: FEX’s (dual homed) Compute node Baremetal Compute node Switch/Port API server Cisco Nexsus Fabric SW Manage NW OpenStack Manage NW
  27. 27. 29 Nova-baremetal(havana)/Ironic(juno) ansible Baremetal networking • Bonding NIC + lldp discovery • Taged VLAN • allowd VLAN + dhcp native VLAN
  28. 28. 30 GMO AppsCloud(Havana/Juno)
  29. 29. 31 Public API security and load balance: • LVS-DSR • L7 reverse-proxy • API validation wrapper
  30. 30. 32 public API Web panel(httpd, php) API wrapper proxy (httpd, php Framework: fuel php) Nova API Customer sys API Neutron API Glance API OpenStack API for input validation Customer DB Keystone API OpenStack API Cinder APICeilometer API Endpoint L7:reverse proxy Swift Proxy
  31. 31. 33 Public API global network LVS-DSR (act-stby) the Cloud (Internet) HAProxy LVS heatbeat api-reverse-proxy01 api-reverse-proxy02elvs01 elvs02 VMx2 LVS heatbeat VMx2 HAProxy ext-api-wrapper01 php + httpd - keystone - nova - cinder - neutron - glance - account ext-api-wrapper02 php + httpd - keystone - nova - cinder - neutron - glance - account control-nodes01 - keystone API - nova API - cinder API - neutron API - glance API control-nodes02 - keystone API - nova API - cinder API - neutron API - glance API OpenStack Management network step 1) step 2) step 3) step 4) public API: step 1, step 2) step 1) LVS-DSR (L4) is received https(tcp/443) packet, then forward api-reverse-proxy real IP’s. step 2) HAProxy has valid API ACL and backend server configurations. IF HAProxy allowed POST “/v2.0/tokens”, then the request call to ext-api-wrapper0[12].
  32. 32. 34 Public API global network LVS-DSR (act-stby) the Cloud (Internet) HAProxy LVS heatbeat api-reverse-proxy01 api-reverse-proxy02elvs01 elvs02 VMx2 LVS heatbeat VMx2 HAProxy ext-api-wrapper01 php + httpd - keystone - nova - cinder - neutron - glance - account ext-api-wrapper02 php + httpd - keystone - nova - cinder - neutron - glance - account control-nodes01 - keystone API - nova API - cinder API - neutron API - glance API control-nodes02 - keystone API - nova API - cinder API - neutron API - glance API OpenStack Management network step 1) step 2) step 3) step 4) public API: step 3), step 4) step 3) ext-api-wrapper0 [12], it is a php program. request URI and header, and the input value of json of the body was confirmed by php, and then call the real OpenStack API as the next processing. step 4) OpenStack API that is checked the input value will be run.
  33. 33. 35 OpenStack Juno cluster: • ConoHa (Juno) and Z.com cloud • AppsCloud (Juno)
  34. 34. 36 Tokyo Singapore Sanjose # ConoHa has data centers in 3 Locations
  35. 35. 37 Tokyo Singapole User/tenant User/tenant API Management Keystone API API Management Keystone APIAPI Management Keystone API Token Token Tokyo SanJoseSingapore API Management Keystone API API Management Keystone API READ/WRIT E READ READ TokenToken Token Do not create/delete users Do not create/delete users Our Customer base User administration # User-registration is possible in Japan only DB Replication DB Replication User/tenant User/tenantUser/tenant R/W R/W
  36. 36. 38 OpenStack Juno: 2 service cluster, released Mikumo ConoHa Mikumo Anzu Mikumo = 美雲 = Beautiful cloud New Juno region released: 10/26/2015
  37. 37. 39 • Service model: Public cloud by KVM • Network: 10Gbps wired(10GBase SFP+) • Network model: – Flat-VLAN + Neutron ML2 ovs-VXLAN overlay + ML2 LinuxBridge(SaaS only) – IPv6/IPv4 dualstack • LBaaS: LVS-DSR(original) • Public API – Provided the public API (v2 Domain) • Compute node: ALL SSD for booting OS – Without Cinder boot • Glance: provided • Cinder: SSD NexentaStore zfs (SDS) • Swift (shared Juno cluster) • Cobbler deply on under-cloud – Ansible configuration • SaaS original service with keystone auth – Email, web, CPanel and WordPress OpenStack Juno: 2 service cluster, released • Service model: Public cloud by KVM • Network: 10Gbps wired(10GBase SFP+) • Network model: – L4-LB-Nat + Neutron ML2 LinuxBridge VLAN – IPv4 only • LBaaS: Brocade ADX L4-NAT-LB(original) • Public API – Provided the public API • Compute node: Flash cached or SSD • Glance: provided (NetApp offload) • Cinder: NetApp storage • Swift (shared Juno cluster) • Ironic on under-cloud – Compute server deploy with Ansible config • Ironic baremetal compute – Nexsus Cisco for Tagged VLAN module – ioMemory configuration
  38. 38. 40 OpenStack Cinder Block storage: ConoHa: NexentaStor(SDS) AppsCloud: NetApp
  39. 39. 41 NexentaStor zfs cinder: ConoHa cloud(Juno) Compute
  40. 40. 42 NetApp storage: GMO AppsCloud(Havana/Juno) If you are using the same Cluster onTAP NetApp a Glance and Cinder storage, it is possible to offload a copy of the inter-service of OpenStack as the processing of NetApp side. • Create volume from glance image ((glance the image is converted (ex: qcow2 to raw) required that does not cause the condition) • Volume QoS limit: Important function of multi- tenant storage • Uppper IOPS-limit by volume
  41. 41. 43 OpenStack Ironic: Only AppsCloud: • Undercloud Ironic deploy • Multi-tenant Ironic deploy
  42. 42. 44 Ironic with undercloud: GMO AppsCloud(Juno) For Compute server deployment. Kilo Ironic and All-in-one • Compute server: 10G boot • Clout-init: network • Compute setup: Ansible Under-cloud Ironic(Kilo): It will use a different network and Ironic Baremetal dhcp for Service baremetal compute Ironic(Kilo). (OOO seed server) Trunk allowed vlan, LACP
  43. 43. 45 Ironic(Kilo) baremetal: GMO AppsCloud(Juno) Boot baremetal instance • baremetal server (with Fusion ioMemory SanDisk) • 1G x4 bonding + Tagged allowed VLAN • Clout-init: network + lldp • Network: Nexsus Cisco Allowd VLAN security Ironic Kilo + Juno: Fine • Ironic Python driver • Whole Image write • Windows: OK
  44. 44. 46 • Service model: Public cloud by KVM • Network: 10Gbps wired(10GBase SFP+) • Network model: – Flat-VLAN + Neutron ML2 ovs-VXLAN overlay + ML2 LinuxBridge(SaaS only) – IPv6/IPv4 dualstack • LBaaS: LVS-DSR(original) • Public API – Provided the public API (v2 Domain) • Compute node: ALL SSD for booting OS – Without Cinder boot • Glance: provided • Cinder: SSD NexentaStore zfs (SDS) • Swift (shared Juno cluster) • Cobbler deply on under-cloud – Ansible configuration • SaaS original service with keystone auth – Email, web, CPanel and WordPress OpenStack Juno: 2 service cluster, released • Service model: Public cloud by KVM • Network: 10Gbps wired(10GBase SFP+) • Network model: – L4-LB-Nat + Neutron ML2 LinuxBridge VLAN – IPv4 only • LBaaS: Brocade ADX L4-NAT-LB(original) • Public API – Provided the public API • Compute node: Flash cached or SSD • Glance: provided (NetApp offload) • Cinder: NetApp storage • Swift (shared Juno cluster) • Ironic on under-cloud – Compute server deploy with Ansible config • Ironic baremetal compute – Nexsus Cisco for Tagged VLAN module – ioMemory configuration
  45. 45. 47 Fin.

×