How we Upgraded Public Cloud From Juno to Queens with Minimal Downtime? | Nguyễn Duy Bình

2. Content overview
BizFly Cloud - before upgrade
Problems and Upgrade Challenges
Big Upgrade from Juno to Queens
with minimal downtime
BizFly Cloud - after upgrade

3. VCCORP Joint stock company
Electronic Media - ChannelVN
E-commerce - Zamba
VOD, Video Streaming, Game - Soha
Advertisement - Admicro
IaaS, PaaS and SaaS – BizFly Cloud

4. THE DEVELOPMENT PRODUCTS BY VCCORP

5. Build Private Cloud since 2012
Release Public Cloud since 6/2014
Release CDN since 2015 and have distributed system at almost big Datacenter in VietNam
• Serving projects at VCCorp
• Size
~ 5000 cloud server
~ 4000 Physical CPU Core
~ 800 TB RAM
~ 5000 TB DATA
CLOUD AT VCCORP

6. BizFly Cloud - Before Upgrade
Openstack release: Juno
Keystone with LDAP
Simple dashboard, less feature
Provide only 2 services: Cloud
server and CDN

7. BizFly Cloud - Before Upgrade

8. BizFly Cloud - Before Upgrade

9. Problems
DNS, proxy Cache, CDN, DDoS Filter
Juno is a very old Openstack release at that time
New keystone version can’t CRUD with LDAP
Hard to develop new services based on
Openstack such as: Loadbalancer, VPN, DBaaS,
Auto Scale…..
Lack many new features of core services of new
Openstack version
Hard to develop new dashboard for users with
new features

10. Upgrade Challenges
DNS, proxy Cache, CDN, DDoS Filter
Queens - Juno = 6 Openstack versions
Convert from LDAP to Mysql
Minimal downtime for ~1200 VM on Public Cloud
Data integrity, network, IP, metadata of VM
Dashboad, billing... after upgrade

11. Big Upgrade from Juno to Queens with minimal
downtime
3/2018
4/2018 6/2018
Tháng
7/2018
Upgrade and finish
convert VM from Juno
to Queens
Define Problems,
Challenges and Start
Research
Build up full Plan for
Upgrade Build Queens Cloud
Develop New
Dashboard for Cloud
Convert LDAP to
MySQL
Acceptance Test Full
System
Automation upgrade
process

12. Convert from LDAP to MySQL
User
Login to
Dashboard and
change password
Update to LDAP
Keystone Create
Project/User and write to
Mysql
LDAP MySQL

13. Network, IP, Metadata of VM and data integrity
Get all information of an instance:
ID, volume ID, Snapshot of volume,
name, network, keypair, metadata
Juno Openstack
Cloud
Python Script
Create the same VM,
volume, snapshot
with all the
information get from
Juno Cloud
Queens Openstack
Cloud
Ceph Cluster
Stop Queens Cloud VM and update
database to mapping VM with
volume on Ceph backend
VM running on Ceph
Cluster

14. Minimal downtime with ~1200 VM
Results: downtime ~5 minutes per VM
Juno Openstack
Cloud
Queens
Openstack
Cloud
Stop VM on Juno Cloud and
Start corresponding VM on
Queens Cloud

15. BizFly Cloud - After upgrade
Openstack release: Queens
Keystone with MySQL
New Dashboard with many new features
Develop new Openstack Cloud based services
New hardware with more powerful:
● Compute node with new and high
performance CPU
● Upgrade network from 20GB to 40GB,
dedicate network for VM and storage
● More high availablity in all system

16. BizFly Cloud - After upgrade
More and more Openstack Cloud Based services released

17. BizFly Cloud - After upgrade

18. BizFly Cloud - After upgrade

19. BizFly Cloud - After upgrade

20. THANK YOU