This slide is to declare how to rolling upgrade in OpenStack with five main projects

1. A f O S
?
Licensed under Creative Commons Attribution license.
Get more templates at slidescarnival.com

2. What will we discuss today?
● Rolling upgrade overview
● Sensitive points
● Rolling upgrade in some OpenStack projects
2
Image source: https://getsilo.com/rsa2016/

3. O/
Dai, Dang Van
OpenStack Active Contributor
Linux, OpenStack, Docker and Ansible
@daikk115(Twitter) - @daidv(IRC)
daidv@vn.fujitsu.com
F f !
3

4. Rolling upgrade overview
● What is rolling upgrade?
● Rolling upgrade in OpenStack
1
4

5. What is rolling upgrade?
◇ Allow distributed system running well while upgrade
parts of system one by one.
Image source:
http://www.dba-oracle.com/t_rolling_upgrades.htm
https://pubs.vmware.com/vfabric5/index.jsp?topic=/com.vmware.vfabric.gemfire.6.6/getting_started/rolling_upgrade.html
https://www.quora.com/What-is-meant-by-a-rolling-upgrade-in-software-development
5

6. Rolling upgrade in OpenStack
◇ Upgrade with minimal downtime on control plane
◇ Data plane is fine during upgrade process
◇ Assert:supports-rolling-upgrade tag
Read more: Rolling Upgrade [Part1]
https://blog.vietstack.vn/rolling-upgrade-part-1/
6

7. Sensitive points
Which points should we concern during OpenStack rolling upgrade processes?
2
7

8. API & Message Queue
Refer:
https://specs.openstack.org/openstack/openstack-user-stories/user-stories/proposed/rolling-upgrades.html#gaps
API & RPC version change
>> API micro-versioning &
Multi-version Interoperability
Cheng Tong Yan: http://tgod.cn
8
Backend
Supported backend/version change
Database
Schema change
>> Online schema migration
Icon source: Clipart Library

9. nova-scheduler
HTTP
Nova-api
2.42
nova-api
2.42
Neutron
Old novaclient
nova-api
2.53
Ocata
Pike
Support MIN -> MAX versions
(2.1 -> 2.53)
9
API micro-versioning
Backward compatible with old API call

10. Multi-version Interoperability
nova-api nova-scheduler
RabbitMQ
nova-conductor
1.16
nova-conductor
1.16
nova-api nova-scheduler
nova-conductor
1.17
Ocata
Pike
Check version and
make compatible
10
Enabling communication between different versions of the same OpenStack Service

11. Online schema migration
11
◇ Database change during upgrade.
◇ Allow both of new and old versions working together with the same database.
Image source:
http://dbperfection.com/migration-from-sql-server-to-oracle-using-sql-developer/

12. Online schema migration
Read more:
Rolling upgrade [part2]
12
Trigger-base Trigger-less
Use trigger Versioned object model
Stick on database Implement in codebase
It’s hard for testing Easy to test
Deleable column/table from
N -> N+1
Not deleable column/table from
N-> N+1
Using by KeyStone, Glance, etc. Using by Nova, Neutron, Cinder, etc.

13. Rolling upgrade in some
OpenStack projects
● KeyStone
● Glance
● Nova
● Neutron
● Cinder
3
Detail commands:
https://github.com/daikk115/openstack_upgrade_test/tree/master/upgrade_script
13

14. Upgrade
14

15. 15
Source: https://github.com/daikk115/openstack_upgrade_test
HAProxy

16. Keystone, Glance
Controller1 Controller2 Controller3
Refer:
https://docs.openstack.org/keystone/ocata/upgrading.html
https://specs.openstack.org/openstack/glance-specs/specs/ocata/approved/glance/rolling-upgrades.html
https://specs.openstack.org/openstack/glance-specs/specs/ocata/approved/glance/database-strategy-fo
r-rolling-upgrades.html
16

17. Keystone, Glance
Controller1 Controller2 Controller3
Set drain mode in haproxy
17

18. Keystone, Glance
Controller1 Controller2 Controller3
1. Stop service(s)
2. Install new source code
3. Run expand phase
4. Run migration phase
5. Start service(s)
18

19. Keystone, Glance
Controller1 Controller2 Controller3
Unset drain mode in haproxy
19

20. Keystone, Glance
Controller1 Controller2 Controller3
Set drain mode in haproxy
20

21. Keystone, Glance
Controller1 Controller2 Controller3
1. Stop service(s)
2. Install new source code
3. Start service(s)
21

22. Keystone, Glance
Controller1 Controller2 Controller3
Unset drain mode in haproxy
22

23. Keystone, Glance
Controller1 Controller2 Controller3
Set drain mode in haproxy
23

24. Keystone, Glance
Controller1 Controller2 Controller3
1. Stop service(s)
2. Install new source code
3. Start service(s)
4. Run contract phase
24

25. Keystone, Glance
Controller1 Controller2 Controller3
Unset drain mode in haproxy
25

26. Nova
Pre-upgrade
◇ Create nova_cell0 database
◇ Make an DevStack Ocata ENV
◇ Upgrade Nova database from the ENV
◇ Add support multi nova-compute versions (controllers/computes)
￭ [upgrade_levels]compute=auto in nova.conf
Source: https://docs.openstack.org/nova/ocata/upgrade.html
26

27. Nova
Upgrade services except nova-compute
Stop all Nova
services on
Controllers
Upgrade
source code
Start all
Nova
services on
Controllers
Notes: Start nova-conductor first and nova-api last
27

28. Nova
◇ Upgrade nova-compute one by one
◇ Complete all online data migrations using:
nova-manage db online_data_migrations --limit <number>
28

29. Neutron
Notes
◇ Upgrade neutron-server as we did with Keystone/Glance
◇ After that, node by node:
￭ upgrade L2 agents
￭ Then, upgrade all other agents(L3, DHCP, metadata...)
Source: https://docs.openstack.org/neutron/ocata/devref/upgrade.html
29

30. Neutron
Controller
Network
Compute
30Source: https://github.com/daikk115/openstack_upgrade_test/blob/master/upgrade_script/upgrade/neutron.sh

31. Neutron
Controller1 Controller2 Controller3
Set drain mode in haproxy
31

32. Neutron
Controller1 Controller2 Controller3
1. Stop neutron-server
2. Install new source code
3. Run expand phase
4. Run migration phase
5. Start neutron-server
32

33. Neutron
Controller1 Controller2 Controller3
Unset drain mode in haproxy
33

34. Neutron
Controller1 Controller2 Controller3
Set drain mode in haproxy
34

35. Neutron
Controller1 Controller2 Controller3
1. Stop neutron-server
2. Install new source code
3. Start neutron-server
35

36. Neutron
Controller1 Controller2 Controller3
Unset drain mode in haproxy
36

37. Neutron
Controller1 Controller2 Controller3
Set drain mode in haproxy
37

38. Neutron
Controller1 Controller2 Controller3
1. Stop neutron-server
2. Install new source code
3. Start neutron-server
4. Run contract phase
38

39. Neutron
Controller1 Controller2 Controller3
Unset drain mode in haproxy
39

40. Cinder
Notes
◇ Our public cloud is using A/A with cinder-volumes but the community’s
recommendation is A/P.
◇ We didn’t use Pacemaker, we manually stop/start cinder-volume for
upgrade testing.
◇ Be careful with backend compatible.
Source: https://docs.openstack.org/ha-guide/storage-ha-block.html
40

41. Cinder
Pre-upgrade
◇ Make an DevStack Ocata ENV
◇ Upgrade Cinder database from the ENV
Source: https://docs.openstack.org/cinder/ocata/upgrade.html
41

42. Cinder
Upgrade
cinder-api
node-by-node
Upgrade
cinder-scheduler
node-by-node
Upgrade
cinder-volume
services
42

43. Cinder
Controller1 Controller2 Controller3
Upgrade cinder-volumeUpgrade cinder-volume
Upgrade cinder-volume services
43

44. Cinder
Controller1 Controller2 Controller3
Start cinder-volumeStop cinder-volume
44

45. Cinder
Controller1 Controller2 Controller3
Upgrade cinder-volume
45

46. To Be Continued...
46
Fast
forward
upgrade
(FFU)

47. “
“If you can't do great things, do small
things in a great way.”
- Martin L King
47

48. Thanks!
Image source: http://www.softicons.com/web-icons/emotions-2-icons-by-kirozeng/question-icon
48

49. References
◇ https://docs.openstack.org/ha-guide/
◇ https://docs.openstack.org/oslo.versionedobjects/latest/
◇ https://blog.vietstack.vn/rolling-upgrade-part-1/
◇ https://blog.vietstack.vn/rolling-upgrade-part-2/
◇ https://specs.openstack.org/openstack/openstack-user-stories/user-stories/proposed/rolling-upgrades.html#g
aps
◇ https://github.com/daikk115/openstack_upgrade_test
◇ https://docs.openstack.org/keystone/ocata/upgrading.html
◇ https://specs.openstack.org/openstack/glance-specs/specs/ocata/approved/glance/
◇ https://docs.openstack.org/nova/ocata/upgrade.html
◇ https://docs.openstack.org/neutron/ocata/devref/upgrade.html
◇ https://docs.openstack.org/cinder/ocata/upgrade.html
◇ https://www.openstack.org/videos/sydney-2017/making-fast-forward-upgrade-openstack-possible-with-configuratio
n-mapping-tool
49