Audience: Advanced About: Real world lessons and war stories about Catalyst IT’s experience in rolling out an OpenStack based public cloud in New Zealand. This presentation will provide tips and advice that may save you a lot of time, money and nights of sleep if you are planning to run OpenStack in the future. It may also bring some insights to people that are already running OpenStack in production. Topics covered will include: selection of hardware for optimal costs, techniques that drive quality and service levels up, common deployment mistakes, in place upgrades, how to identify the maturity level of each project and decide what is ready for production, and much more! Speaker Bio: Bruno Lago – Entrepreneur, Catalyst IT Limited Bruno Lago is a solutions architect that has been involved with the Catalyst Cloud (New Zealand’s first public cloud based on OpenStack) from its inception. He is passionate about open source software, cloud computing and disruptive technologies. OpenStack Australia Day - Sydney 2016 https://events.aptira.com/openstack-australia-day-sydney-2016/

1. Presented by Bruno Lago // 05 May 2016
Things you MUST know before
you deploy OpenStack

4. WARNING!
I AM NOT HERE TO SELL YOU A PRODUCT
So...
I don’t have to make it look good

5. How much will it cost?
~ USD $150k one off
For a production cluster and pre-prod environment
+ 2 to 3 people per month to run it
OR
A service provider to manage it remotely for you (~ USD $10k / month)

6. Selecting your hardware

7. Network hardware
2x 10Gbps switches per rack
2x 40Gbps switches for the spine
2x 1Gbps switches for the management network
1x 1Gbps switch for the pre-prod cluster
Features required: VLAN, VXLAN, MLAG, L3 routing using BGP ECMP
Forget Cisco, Juniper, Arista. Use open source switches!
Avoid using vendor specifc neutron providers and go for Open vSwitch.

8. Network hardware
2x 10Gbps switches per rack
2x 40Gbps switches for the spine
2x 1Gbps switches for the management network
1x 1Gbps switch for the pre-prod cluster
Features required: VLAN, VXLAN, MLAG, L3 routing using BGP ECMP
Forget Cisco, Juniper, Arista. Use open source switches!
Avoid using vendor specifc neutron providers and go for Open vSwitch.
Not required on
day one!

9. Server specs

10. Compute nodes

11. ALL THE HYPERVISORS!
(Yeah, Right!)
KVM is by far the most widely adopted and best supported hypervisor.
Open source hypervisors is where the numbers stack up!
AND where you get most support from the community.
That said: OpenStack does work with most hypervisors on the industry
and there are successful deployments running Xen or even VMware.

12. Node segmentation (for financial reasons)
●
Specialised object storage nodes allow optimisation for low cost, high
capacity
●
Block storage nodes can be optmised independently for performance
(IO operations completed under 30ms or 10ms)
●
Compute optimised for high CPU and memory density (and maybe
GPUs)

13. Techniques to drive quality and service
levels up

14. Node segmentation (service levels)
Potential issues with hyper convergence:
●
Kernel bug high memory
●
OVS / kernel bug affecting network namespaces
Segment at least controll plane, compute and storage. If possible
segment network nodes.

15. Useful techniques
●
Run CI / automated tests in your own cloud (and ensure you can run
it on someone’s else cloud too if you have only one region)

16. Useful techniques
●
Run CI / automated tests in your own cloud (and ensure you can run
it on someone’s else cloud too if you have only one region)
●
Run tempest scenario tests as a CI gateway and monitoring check

17. Useful techniques
●
Run CI / automated tests in your own cloud (and ensure you can run
it on someone’s else cloud too if you have only one region)
●
Run tempest scenario tests as a CI gateway and monitoring check
●
Have a decent pre-production environment (YES, you need one)

18. Useful techniques
●
Run CI / automated tests in your own cloud (and ensure you can run
it on someone’s else cloud too if you have only one region)
●
Run tempest scenario tests as a CI gateway and monitoring check
●
Have a decent pre-production environment (YES, you need one)
●
Think about communication channels with customers and prepare
communication tools ahead of time

19. Useful techniques
●
Run CI / automated tests in your own cloud (and ensure you can run
it on someone’s else cloud too if you have only one region)
●
Run tempest scenario tests as a CI gateway and monitoring check
●
Have a decent pre-production environment (YES, you need one)
●
Think about communication channels with customers and prepare
communication tools ahead of time
●
Monitoring that picks up automatically every service / component
deployed

20. In place upgrades
(Yes Sergey, they are possible!)
●
No big bang. One service at a time. Most services have backward
compatible API.
●
Test every change in CI with automated tests
●
Reherse every move in pre-prod
●
Bullet proof live migration (Mitaka, QEMU guest agent)
●
Have scripts to migrate routers and DHCP agents with minimum
downtime

21. Common deployment mistakes

22. GUI driven OpenStack

23. Carrying your own patches
●
As a rule of thumb, never run code in production that has not been
merged upstream
●
Every patch that is not commited upstream creates a recurring
overhead on the team with every release of OpenStack!
●
DON’T do it, unless it is absolutely necessary
●
Trusted me - people have wasted millions with this mistake!
●
Be prepared to fix bugs and introduce new features upstream. If you
are not, then ask for a service provider to do it for you

24. Cloud != Hypervisor
●
A cloud is a complex distributed system with many moving parts
●
It touches every part of your data centre
●
Your team needs to be prepared to dive deep in each area to
troubleshoot incidents and problems

25. Keystone != IDP
●
Back Keystone with OpenLDAP, Active Directory or a SAML based
IdP
●
Think about how people will create / terminate accounts, reset
passwords

26. All projects are production ready
“A project exists, therefore I can do it in production”
How to identify projects ready?
●
Understand your requirements
●
Validate functional and non-functional requirements in real life
●
Try HA procedures in real life
●
Try upgrade procedures in real life
●
Validate security standards
●
Consider doing a code inspection yourself

27. Do the numbers stack up?

28. Can OpenStack beat the prices of “massive
sacale” global cloud providers?
AWS Sydney m3.large /
month
USD $136.16

29. Can OpenStack beat the prices of “massive
sacale” global cloud providers?
AWS Sydney m3.large /
month
USD $136.16
AWS USA m3.large /
month
USD $97.36

30. Can OpenStack beat the prices of “massive
sacale” global cloud providers?
AWS Sydney m3.large /
month
USD $136.16
AWS USA m3.large /
month
USD $97.36
AWS USA m3.large
reserved 3Y upfront
USD $38.14

31. Can OpenStack beat the prices of “massive
sacale” global cloud providers?
AWS Sydney m3.large /
month
USD $136.16
AWS USA m3.large /
month
USD $97.36
AWS USA m3.large
reserved 3Y upfront
USD $38.14
OpenStack Cloud USD $15.13
Price difference USD -$23.01
Price difference (%) 152%