NTTドコモ様導入事例 OpenStack Summit 2016 Barcelona 講演「Expanding and Deepening NTT DOCOMO's Private Cloud」 - OpenStack最新情報セミナー(2016年12月)

Copyright©2016 NTT DOCOMO, INC. All rights reserved.
Expanding and Deepening
NTT DOCOMO’s private cloud
NTT DOCOMO Inc. Jun Ishii
Kojiro Amano
VirtualTech Japan Hiromichi Ito

DOCOMO, INC All Rights Reserved
Jun Ishii
o Research Engineer, NTT DOCOMO
o Developer, operator and technical consultant in NTT
DOCOMO private cloud
Hiromichi Ito
o CTO, VirtualTech Japan
o One of the first members of proposing OpenStack Bare
Metal Provisioning (currently called "Ironic")
Kojiro Amano
o Research Engineer, NTT DOCOMO
o Security consultant in NTT DOCOMO private cloud
About us

Expanding strategy of
our private cloud
Scale-up
strategy

o One year after launched our private cloud,
it goes larger and larger!
Jun. 2015 Oct. 2016 Mar. 2017
Number of DCs 1 2 4
Number of HWs 50 300 900
Cores 1500 10000 Over 35000

o Why could we expand our cloud so fast ?
o Main Strategy : Forest and Tree
Make a forest
Fill in trees

o Three important details
 Decided to migrate a large scale in-house system
 Obtain budget for making a forest
 Fast deployment methods by normalization
 Enable to create the forest quickly
 Novel challenges for various users
 Enrich the forest to plant various trees
• L2GW
• GPU instance
• Reference model
• Security update

o Decided to migrate a large scale in-house system
 Update whole system due to HW EOL
o OpenStack-based cloud has many strengths.
 Three years TCO is superior to an on-premises
 Reduce 22% TCO, CAPEX & OPEX
 Distributed architecture is compatible with cloud.
 REST interfaces are suitable for maintaining systems.
 Feasibility of migration/replication between long distance
 L2GW, details are mentioned in later.

o Fast deployment methods by normalization
 DRY : Don't repeat yourself
o How to
 deploy
 set up [compute node, swift storage node] more
 deal with hardware trouble
 ansible, KB (See our Tokyo summit presentation)
o Only take one month to deploy new DC
 From after racking and cabling ends till finish first QA test
 Over 300 nodes, HW configuration settings & OpenStack install are just
finished in 10 days by 5 operators.

o Novel challenges… to satisfy various users' will, there are many
difficulty and many know-how.
 Add functions
 L2GW
 GPU instance
 Reduce time to construct and manage users' systems
 Reference model
 Security update

o Three important details
 Decided to migrate a large scale in-house system
 Obtain budget for making a forest
 Fast deployment methods by normalization
 Enable to create the forest quickly
 Novel challenges for various users
 Enrich the forest to plant various trees
• L2GW
• GPU instance
• Reference model
• Security update
These reasons enable our private cloud so FaT !!

How deepening our private cloud
Enrich
for users

L2 Gateway
for
connecting existing large-scale networks
and inter-cloud networking.
I'm good at
connecting.

○ Overview
 Our user has large scale existing network and proprietary computer
systems.
– This network system has the great ability that provides Layer-2 connectivity
to nationwide.
– This proprietary computer system side does not have enough flexibility.
 REST API
 Service mobility
 They decided to migrate to OpenStack on this renewal timing.
 Network system side migration must be minimal.
 Our user requested new two network services.
– Connect the tenant network between the two datacenters
– Connect instance and existing equipment with the layer-2

○ Before
DC 1
RTT 20-40ms
WAN
(Nationwide)
DC 2
dedicated
equipment
dedicated
equipment
dedicated
equipment
dedicated
equipment
proprietary
computer
system
proprietary
computer
system

○ After
DC 1
RTT 20-40ms
WAN
(Nationwide)
DC 2
dedicated
equipment
dedicated
equipment
dedicated
equipment
dedicated
equipment
L2
GW
RT
L2
GW
RT

○ Our user's requirements
 High Availability
– Do not share control services between data center.
– Avoid Single-Point-Of-Failure (SPOF)
 Technical limitations
– Do not change IP addressing and routing architecture.
– Do not use Network Address Translation(NAT).
– Must connect instance and existing equipment by Layer 2.
 Performance
– Total throughput requirement is several dozen Gbps.
– Average packet size is smaller than general private clouds.
– Several hundred VLAN must connect.

– Avoid Single-Point-Of-Failure (SPOF).
 Performance
We choose "Region" zoning model.
In "Region" model all service is separated correctly.

 Performance
Our base OpenStack deployment model avoid SPOF already.

 Performance
Existing system's IP addressing and routing architecture can
deploy on the overlay network.

 Performance
NAT is must technique for floating IP and connecting the external network.
But, This system does not request the floating IP address function.

 Performance
Our base OpenStack deploy model is using L3 ECMP fabric and VXLAN.
So, We choose VXLAN Layer 2 Gateway(L2 Gateway).

 Performance
Software based VXLAN L2 Gateway does not match short packet workload.
So, We choose using hardware based VXLAN L2 Gateway.

○ Equipment selection
 Hardware VTEP
– Modern L3 switch chipset has Hardware VTEP gateway functions.
 Intel FM6000, Broadcom Trident II, Trident II+, Tomahawk
– We tried to examine both Intel FM6000 and Broadcom Trident II based L3
network switch.
– Finally, we compared three vendors' L3 switches. (A, D, J)
 Comparison result
– Vendor A's L3 switch can support VXLAN within a Multi-Chassis LAG
(MLAG) deployment. Other vendors can not. (as of June 2016)
– All vendors' L3 switches cleared performance criteria.
– All vendors OVSDB protocol support has some issues.
We choose vendor A's L3 switch. Because they support MLAG.

○ Software test and proof-of-concept
 Test target
– Neutron networking-l2gw
 API's and implementations to support L2 Gateways in Neutron.
– Networking-l2gw provides "L2GW Service Plugin" and "L2GW Agent".
 "L2GW Service Plugin" provides L2GW API services.
 "L2GW agent" controls L3 switch by OVSDB protocol.
 Test results
– Several minor bugs (Already fixed by the community.)
– Missing of features that is required for the production environment.
 SSL support (Already implemented by the community.)
 Handling Mcast_Macs_Remote table (We created modified patch for
vendor A based on community patch, not merged yet.)

○ Controller Node
Neutron Server
L2GW Service
Plugin
API
ML2
L3
Nova
Keystone
Glance
Cinder
Horizon
ML2 L2POP
Compute Node
ML2 OVS
Agent
Open vSwitch
VTEP
Virtual
Switch
OVSDB
Server
ML2 L2POP
A’s Management Virtual appliance
Network Node
L2GW
Agent
ML2 OVS Agent
Open vSwitch
VTEP
L3
Agent
A’s
Hardware VTEP
WAN
Hardware
VTEP
Virtual Router
VLAN VLAN
OVSDB
Server
OVSDB
protocol
OVSDB
Server
Control

○ Result of pilot tests with as scale as production environment
 Hardware L2GW side
– OVSDB Server crash issues
 When inserting a large number of record at one time, OVSDB server has
crashed. (This issue already fixed by the vendor.)
 Networking-L2GW side
– We encountered several critical bugs.
 But It is hard to reproduce.
 When hit these bugs, L2GW agent stopped.
– L2GW agent recovery from a crash state is terrible.
 L2 gateway agent always syncs state between neutron database and
OVSDB.
 Unfortunately, when L2GW agent crashed or stalled, these two databases
sometimes lost sync.
 So, We must re-register L2GW connections manually when met these bugs.

o Network trouble occurred without missing a week.
 The L2GW agent is unstable.
 The l2gw agent does not work correctly after few days when users run a long
test.
• That test includes continuous instance creating and deleting.
• That test includes continuous CRUD testing for neutron virtual network
port.
 The instance could not communicate another region instance and existing
equipment.

o We decided to not use Networking-l2gw in the production
environment for the time being.
 We could not reproduce connection troubles between OVSDB and the
L2GW agent that occurred a weekly.
 We could not fix all critical bugs that we encountered.
 In our environment, a port status issue occurred.
 That issue will cause L2GW agent problem.
 We would not like to use the l2population.
 The l2population does not have enough scalability yet.
Keep to a delivery date.
 Delivery date delay. aka. Our project death.

o Our solution
 We created a manual procedure to manage L2GW.
 Manage an OVSDB by CLI
• "vtep-ctl" command
 Manage an OVS flow table by CLI
• "ovs-ofctl" command
 We created an automation system based on the manual procedure.
 The upper management system calls the system.
This system is working correctly now.

o Results
 We provide stable L2GW which is connecting the two regions instances
and existing equipment.
 We passed all test criteria provided from the client.
 Gets excellent service flexibility by the OpenStack.
 The existing network configuration was kept that our user requested.
o Next challenges of our L2GW project
 Fix known issues of networking-l2gw.
 We would like to provide OpenStack API for managing L2GW.
 Fix scalability issue of l2population.
 Investigate EVPN for expanding L2GW services.

GPU instance
Machine learning
on OpenStack

Nvidia Tesla M40
for CUDA/cuDNN

o Deploy
 To deploy GPU nodes, not only PCI pass-through functions but also
IOMMU functions must be enabled.
 Enable PCI pass-through
 Whitelist, alias, flavor
 See OpenStack wiki*
 Enable IOMMU
 Add grub settings to grub.conf
 GRUB_CMDLINE_LINUX_DEFAULT=“$GRUB_CMDLINE_LINUX_DEFAUL
T intel_iommu=on”
 dmseg | grep –e DMAR –e IOMMU
* https://wiki.openstack.org/wiki/Pci_passthrough

o Operate : Take care of flavor memory size
 As a result of our verification, IOMMU allocates all memory resources
when instances are launched.
 If you set flavor memory size large enough and launch maximum number
of instances, OOM-killer might kill the qemu process.
 Swapping doesn't work well because IOMMU allocate memory too fast.
Normal compute node
Mem space
IOMMU-enabled
compute node
Mem space
HostOS
HostOS
Instance
A
Instance B
Instance A Instance B
Allocated
arter
ballooning

o A workarounds to this problem are to take enough margin for
host OS.
 Reduce flavor memory size
 Sometimes too uncomfortable for GPU users
 Set reserved_host_memory_mb in nova.cfg to large size
 Also affect other flavors
 Decrease maximum number of instances on per host
→Any other solutions? Help us!

o How should we offer GPU flavor to in-house users?
o GPU with OpenStack, pros/cons
As a point of Pros. Cons.
Virtualization More stable than container Some GPU card trouble needs host
reboot
Immutableness Fast deploy, fast PDCA cycle Difficulty fair sharing GPU resources
Preparation
before
run machine
learning
Can provide device driver and
CUDA pre-installed image file
Need to follow new version, new
combination of driver, guest OS, CUDA
ver, library ver…
Cooperate with GPU instance users is important for private cloud providers

Reference Model

Aim to migrate some of in-house APP to our cloud
Problem
 Strict security policies：over 100 guidelines for system architecture
when users reconstruct APP on our cloud
 A lot of efforts are required to meet the policies.
 Nice if predefined models are provided:)
Monitoring
Security
vulnerabilityCertificate
Remote Access
Identification and
authentication IDS/IPS
Log
Encryption
Server Network Storage Operation
Firewall
Redundancy
Backup
Configuration
management
… … … …

“Reference Model” on our cloud
 System architecture based on many of security policies
 Sets of OSS stacks that have been heavily tested on our project
Our cloud
Web three-tier
model
Heat template
• Web basic model
• Web three-tier model
Input Template file into heat
Virtual router
Virtual network
Jump server
Proxy server
Web basic
model
Virtual router
Virtual network
LB/Web server
Jump server
Proxy Server LB/Web server
DB ServerWEB/LB
Jump
…...
AP server
Images
Mechanism

System architecture of Reference Model
WEB/LB WEB/LB
DB DB
Storage Storage
Internet
AP AP
Backup
Public-NW 192.168.10.0/24
Private-NW 192.168.20.0/24
Management-NW 192.168.30.0/24
SSL termination SSL termination
LB
VI
P
DB
VI
P
VPN
SSL -VPNProxy/NT
P
Monitoring
Storage
LB-HA-NW
DB-HA-NW
DB-repl-NW
End User
HTTPS
Operator
VPN

WEB（Apache）/LB（LVS w/ Ultramonkey） server
 HTTPS, dummy certificates installed by default
 WAF for IDS/ IPS
 The key point is to not only install, but also complete the
default setting about security.
 Why didn’t we use LBaaS_v1 ?
 LBaaS_v1(juno) doesn’t satisfy with use cases of our users.
 Required to
• set security group to LB(LBaaS_v2:not yet)
• terminate SSL at LB(LBaaS_v2：done)
• provide sorry page (LBaaS_v2：not yet)

 VPN（OpenVPN） server
 SSL-VPN for secure remote access
 Tools for operation of SSL-VPN, such as create and revoke certificates
 Why didn’t we use VPNaaS_v1 ?
 The algorithm for authentication in IKE phase1 accepted sha1, which
will be encryption losing safety assurance.
 VPNaaS in recent version “Newton” accepted sha256.

The covered areas by “Reference Model”
 60% of security policies
Future
 Update this model by adding the missing parts about security policies
 Aims to cover 100%
Monitoring
Security
vulnerabilityCertificate
Remote Access
Identification and
authentication IDS/IPS
Log
Encryption
Server Network Storage Operation
Firewall
Redundancy
Backup
Configuration
management
… … … …

Security Update

 Current daily operation about vulnerabilities
 Most of the operation is manual.
 Check vulnerabilities
 Risk assessment of vulnerabilities
 Management TODO list
 Update our cloud
 Caused
 Human error
• Forget to check vulnerabilities
 Time：1hours/day
 More important operation of security as our cloud expands
 Nice if these operations can be automated:)

 Current Operation
 We proposed how to be automatic through processes.
 Testing to enable us to reduce human error
Check
vulnerabilities
Risk
assessment
Management
TODO list
Update
our cloud
Semi-
automatic
Operation
By the script checking
package-version related
with vulnerabilities
By checking
vendor site
By Excel
Semi-
automatic
By making ansible
playbook

 Check vulnerabilities
 CVE & CVSS
 CVE: attached ID to vulnerabilities
 CVSS: score to vulnerabilities
 API “CVE-search” is used for check
 Github: https://github.com/cve-search/cve-search

 Risk Assessment
 Key point
 CVSS risk assessment is not always match with our environment.
 Important
 The usage and version of package（→script）
 Whether the host can be internal NW or not.
 Vulnerability that guest OS can invade host OS
 Need to re-evaluate the CVSS score for each host regarding its
environment

 Management TODO list
 Do not forget vulnerabilities which have high risk until the patch of the
vulnerability is applied.
 Important
 Even if the CVSS score is low, it will sometimes become high score
in our environment.
 Need to check the same vulnerabilities continuously

 Update Our cloud
 Semi-automatic Procedure
 Manual interventions are required only for check points
 Consider the influence on users’ instance
 Future
 Apply our proposed way in the test environment at first
 Extend our tools for user’s self check
Live Migration
users’ instance
Security
Update
Return back
user’s instance
Check point①
The normality
of User’s APP
Check point②
The normality of
Openstack function
Check point③
The normality of
User’s APP

Thank you for listening!

NTTドコモ様 導入事例 OpenStack Summit 2016 Barcelona 講演「Expanding and Deepening NTT DOCOMO's Private Cloud」 - OpenStack最新情報セミナー(2016年12月)

More Related Content

What's hot

Similar to NTTドコモ様 導入事例 OpenStack Summit 2016 Barcelona 講演「Expanding and Deepening NTT DOCOMO's Private Cloud」 - OpenStack最新情報セミナー(2016年12月)

More from VirtualTech Japan Inc.

Recently uploaded