Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Openstack Ops Meetup Palo Alto LT


Published on

LIghtning talk at Ops mid-cycle meetup at Palo Alto on August 18, 2015. I explain the way how NTT & NTT Resonant applied OpenStack to build web service infrastructure.

Published in: Software
  • Be the first to comment

Openstack Ops Meetup Palo Alto LT

  1. 1. Copyright©2015 NTT corp. All Rights Reserved. OpenStack Use Case and related Project in NTT Group Toshikazu Ichikawa August/18/2015 Cloud SE Project NTT Software Innovation Center, NTT Corporation Copyright(c)2015 NTT Corp. All Rights Reserved.
  2. 2. 2Copyright©2015 NTT corp. All Rights Reserved. • NTT Group Affiliates • Use case as a web service infrastructure • Components and Deployment • Networking and Customization • Process Layout • Future plans • Masakari: VMHA for OpenStack Compute • VMHA for “Pets” service model • Architecture and Data Flow Index
  3. 3. 3Copyright©2015 NTT corp. All Rights Reserved. NTT Group Affiliates Citation:
  4. 4. 4Copyright©2015 NTT corp. All Rights Reserved. • Web Service Infrastructure • Web search engine • Web portal site • Smartphone application • Online store • Enterprise customer services • And so on • Using OpenStack as Private Cloud • In production since 2014 October • 80+ services • 1 Billion Page View per month • 400 hypervisors • 4,800 physical cores • 1,700+ virtual servers Use Case of Web service infrastructure
  5. 5. 5Copyright©2015 NTT corp. All Rights Reserved. • OpenStack Components • Nova • Swift • Glance • Keystone • Horizon • Neutron • Deployment • Distribution: RDO with CentOS6 • Icehouse version • Single Datacenter • 2 Nova cells (200 hypervisors x 2 cells) • Puppet for Configuration Management • Zabbix for Monitoring Components and Deployment of Web Infra.
  6. 6. 6Copyright©2015 NTT corp. All Rights Reserved. • Networking with OpenStack • Using ML2, Linuxbridge agent, vlan-type provider network • An administrator prepares networks and subnets per tenant • A tenant is not allowed to create/delete his/her network • No DHCP agent, No L3 agent • Guest OS gets IP address by cloud-init and config-drive • Networking without OpenStack • Router, NAT • Load Balancer, Firewall • Customization • Users access only Horizon, not API • Horizon is modified • Enforcement naming convention of server name • No access to network creation/deletion • No access to security group function • and so on Networking and Customization of Web Infra.
  7. 7. 7Copyright©2015 NTT corp. All Rights Reserved. • HA strategy • MariaDB Galera Cluster (multi-master MySQL) • RabbitMQ (mirrored queue) • Haproxy of load balancing to state-less services • API • Horizon • DB (with single writer configuration) • MQ • Pacemaker (active-passive) for state-full services • Virtual IP (for haproxy) • nova-consoleauth • Active-active deployment of state-less internal services • nova-*, swift-* • No redundancy at Compute Node • Nova-compute and neutron-linuxbridge-agent HA strategy of Web Infra.
  8. 8. 8Copyright©2015 NTT corp. All Rights Reserved. Process Layout Web Infra. Top cell controller node pacemaker(Active-Passive) •Virtual-IP(endpoint) •nova-consoleauth HAProxy(balancer) nova-api glance-registry keystone-api neutron-server nova-novncproxy horizon nova-cells Apache (Horizon) Rabbitmq(mirrored queue) Compute node nova-compute neutron-linuxbridge-agent DB Node Pacemaker Haproxy glance-api swift-proxy swift-objectstore … MySQL (MariaDB galera cluster) pacemaker(Active-Passive) •Virtual-IP (for rabbimq) HAProxy(balancer) nova-conductor nova-scheduler nova-cells Rabbitmq(mirrored queue) Children cell controller node Cell 1 Cell 2 Swift
  9. 9. 9Copyright©2015 NTT corp. All Rights Reserved. • Future Plans • Upgrade • to Juno, Kilo and furthermore • 400 hypervisors • HostOS: CentOS6 -> CentOS7 (RDO requirement) • Running: nova-compute and neutron-linuxbridge-agent • with config-drive • How to minimize customer impact (downtime)? • VM instances migration (migrate or shelve) and OS rolling upgrade? • Expand • More hypervisors • at second datacenter • with different external L3 connectivity • How to make them seamless? • Independent regions, regions with single Keystone or any other choice? Future plans of Web Infra.
  10. 10. 10Copyright©2015 NTT corp. All Rights Reserved. • Service Model: Pets vs Cattle • Cloud-native application handles High Availability at its own layer. (Cattle service model) • Sometimes, customers and/or applications still prefer “Pets” service model. • Masakari • Provides VMHA for “Pets” service model • Automatic VM instance recovery in case of VM or hypervisor failure to minimize the downtime • Requires Nova and Shared file system such as NFS • Needs No modification of OpenStack components • Uses Pacemaker to detect a host failure • Is now at • Author: Kentaro TANAKA <> • Is mostly written in Python • Is open source under Apache license Masakari: VMHA for OpenStack Compute
  11. 11. 11Copyright©2015 NTT corp. All Rights Reserved. Masakari: Architecture and Data Flow OpenStack API ComputeNodesControllerNodes &BackendNodes