All Things Open 2014 - Day 2 Thursday, October 23rd, 2014 Mark Voelker Technical Leader with Cisco Cloud/OpenStack Considerations for Operating an OpenStack Cloud

1. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1
Mark T. Voelker, Technical Leader @ Cisco
OpenStack ATC/StackForge Puppet Core/Foundation Member #54
All Things Open 2014

2. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
@marktvoelker
• Tech Lead at Cisco, StackForge Puppet core developer, OS Foundation
Member #54
• Fact: can be bribed with doughnuts
• Currently works in Cisco’s Cloud & Virtualization Group
• In copious (hah!) spare time: OpenStack solutions, Big Data, Massively
Scalable Data Centers, Devops, making sawdust with extreme prejudice

3. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
• Tech lead, manager, software developer, architect
• Started in OpenStack in 2011 at the Diablo Design Summit

4. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
The great thing about my job is that I get to have fun exploring a lot
of new things…

5. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
….and I get to help build a LOT of clouds.

6. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
Today’s talk won’t be overly formal….

7. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
…because I tend to get excited by this stuff.

8. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8

9. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9

10. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
……then you know how to get to Day 1.
Now let’s talk about getting to Day 30…

11. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
• Architecture
• Components
• High Availability
• Bare metal bring-up
• Config management
• CI/CD
• Packaging
• Automated test
• Monitoring
• Up/down alerting
• Trending data
• Logging and log search

12. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
High
Availability?
Sounds
great--I’ll
take two!

13. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
• Consider whether you want active/active or active/passive
• Setup and tooling differs a bit, but I generally like active/active
• Note that docs.openstack.org has an HA Guide
• A bit dated…patches welcome!
• Prioritize HA for the control plane
• That also means thinking about your database, network, and RPC bus
• Instance-level HA: there be dragons
• But yes, it’s being looked at
• Pets vs cattle
• Note: HA == more hardware
• Some components need at least 3 nodes

14. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
• Stuff OpenStack needs to run: message brokers
• Check out RabbitMQ clustering and mirrored queues
• Check out Galera for MySQL/MariaDB
• I usually see Percona XtraDB
• Frontend with an HAProxy/Keepalived pair

15. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
• Don’t do rabbit clustering
over a WAN
• Be aware of the SELECT…
FOR UPDATE issue

16. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
• Long story short: Neutron and some parts of Nova invoke an SQL
pattern known as “SELECT…FOR UPDATE” which Galera
doesn’t support due to issues with cross-node locking.
• Can cause deadlocks symptoms.
• Neutron/nova code being refactored to remove, but will likely not
be done until at least Kilo.
• Meanwhile: use HAProxy to send writes to a single Galera node
and you should be fine
• With the obvious scalability bottleneck
• More info here.
• Thank Jay Pipes & Peter Boros for
the find!

17. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
• Use Swift, Ceph, or other highly available storage to back Glance
• Pick a highly available storage backend for Cinder too
• Use Keepalived/HAProxy to front-end multiple API servers
• Or another load balancer technology of your choice
• Can be deployed as dedicated nodes for scale, or cohabitate
• Network: DVR vs Provider Network Extensions
• Distributed Virtual Routers are a new experimental feature in Juno (not yet
ready for production)
• Please go test it and report/fix bugs!
• Provider networks essentially punt the availability issue to your physical
network
• Allows you to use standard tools like virtual port channels and VRRP
• Also highly performant

18. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
• Architecture
• Components
• High Availability
• Bare metal bring-up
• Config management
• CI/CD
• Packaging
• Automated test
• Monitoring
• Up/down alerting
• Trending data
• Logging and log search

19. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
We start with bare metal.

20. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
• For a cloud of any real size, you don’t want to be installing
operating systems by hand
• Remember that baremetal bringup actually isn’t something that
just happens once…often recurs for upgrades, capacity
expansion, etc.
• Baremetal bringup tools can also have other uses, like inventory
or bootstrapping configuration management agents.

21. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
• A simple (~15k lines of Python code) tool for managing baremetal
deployments
• Flexible usage (API, CLI, GUI)
• Allows you to define systems (actual machines) and profiles (what
you want to do with them)
• Provides hooks for Puppet so you can then do further automation
once the OS is up and running
• Provides control for power (via IPMI or other means), DHCP/PXE
(for netbooting machines), and more.

22. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22

23. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23

24. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24
• Razor
• Developed by EMC, managed by Puppet Labs (occasionally used with Chef
too)
• Initial release in 2012
• Uses a “microkernel” loaded onto the machine to gather facts before
provisioning
• Tag + Policy model
• Crowbar
• Originally written by Dell, now a community project
• Originally designed to deploy OpenStack on all the way from baremetal
• Now deploys other stuff too (namely, Hadoop)
• Uses Chef to handle everything after the OS install
• Foreman
• Used by Red Hat among others
• Does baremetal bringup and serves as a Puppet ENC

25. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25
• Architecture
• Components
• High Availability
• Bare metal bring-up
• Config management
• CI/CD
• Packaging
• Automated test
• Monitoring
• Up/down alerting
• Trending data
• Logging and log search

26. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26

27. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
“Cloud isn’t just an infrastructure technology….it’s a new
operations model. And with OpenStack in particular, it’s
one that’s very well suited to a DevOps style of
management. Many companies aren’t just adopting
cloud, they’re changing how they operate.”
“Besides, logging into servers to mess with config files
makes me sad.”
--That ranty guy in Raleigh again

28. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
• Remember, OpenStack is a set of interoperating distributed
systems
• That means you’re going to have a lot of software to configure on
a lot of machines
• You’re probably going to want to make changes over time
• You’re probably going to have more than one person touching
your cloud
• CM tools help you treat configuration as code, so you can
collaborate more easily

29. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
Pile of
Bash
Scripts

30. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30

31. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
• An increasingly common pattern:
• Puppet or Chef for configuration management, PLUS
• Ansible or Salt for cross-node orchestration
• Recommendation: use the tools that work for you!
• But remember: you don’t have to do it alone.
• Several CM tools have thriving collaborators in the OpenStack community
• Links for later:
• Puppet for OpenStack
• Chef for OpenStack
• Ansible for OpenStack
• SaltStack for OpenStack
• Pile of bash scripts for OpenStack

32. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32
• Unit tests for your deployment code are a good idea
• ServerSpec tests to make sure your config management system
did what it was supposed to are great

33. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33
• Architecture
• Components
• High Availability
• Bare metal bring-up
• Config management
• CI/CD
• Packaging
• Automated test
• Monitoring
• Up/down alerting
• Trending data
• Logging and log search

34. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34
…well, haven’t you always wanted a butler?

35. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35
• DevOps: actually pretty handy
• OpenStack change velocity (community’s and yours)
• Anecdote: the majority of deployments I work with have some
customizations or backports from future releases
• It’s not just OpenStack, it’s all the underpinning components and
your CM code too!

36. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36
• OpenStack itself uses CI/CD tools in it’s development
process…you should consider using them in your cloud buildout
too!
• The OpenStack Infra team has created some awesome tools: JJB, Zuul, etc
• They’re all open source and you can even see how OpenStack’s own CI is
set up (check out Elizabeth Joseph’s slides from yesterday for more!).
• The basics:
• An integration server (Jenkins, Go, Travis, etc)
• A code review and repository tool (Gerrit, Cgit, GitHub, etc)
• A battery of automated tests (lint checks, rspec-puppet, Tempest, Rally, etc)
• Some form of packaging (rpmbuild/mock, sbuilder/pbuilder, etc)
• An artifact repository (Artifactory, yum/apt repos, etc)
• Optionally, some deployment jobs (usually powered by your CM tool)

37. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37
• …you never intend to change the code yourself
• …building your own packages would violate a support contract
with your distribution
• …you’ve never used a CI/CD pipeline before (but really: you
should start learning)
• …you have a static environment that absolutely will not change,
need to add capacity, etc.

38. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38
• Architecture
• Components
• High Availability
• Bare metal bring-up
• Config management
• CI/CD
• Packaging
• Automated test
• Monitoring
• Up/down alerting
• Trending data
• Logging and log search

39. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39
• Now that you have a cloud, you’ll probably want to know that all
it’s parts stay in good working order.

40. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40

41. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41

42. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 42
• I’ve worked on a lot of OpenStack clouds and almost everyone
has their own preferred monitoring toolset.
• One possible exception: almost everybody seems to love Graphite.
• The golden rule is: use the tools that work for you!
• Very often this will be whatever you’re using in the rest of your
infrastructure.
• Break it down into at least two buckets:
• Up/down and alerting (ex: Nagios or it’s derivatives…yes, there are
OpenStack plugins out there on NagiosExchange)
• Trending data collection/plotting (ex: collectd/statsd feeding graphite)
• Also: use your peers!
• Check out Tong Li’s Monitoring as a Service talk later today!
• Operators often willing to share, so ask on the openstack-operators list.

43. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43
• Architecture
• Components
• High Availability
• Bare metal bring-up
• Config management
• CI/CD
• Packaging
• Automated test
• Monitoring
• Up/down alerting
• Trending data
• Logging and log search

44. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 44

45. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 45
• Distributed systems generate logs…..all over the place.
• Finding the root of problems may mean correlating logs from different
machines…but which?
• OpenStack in particular *can* be pretty verbose
• You may also be dealing with logs from other distributed tools in
your cloud (RabbitMQ, databases, etc)
• Generally you want to get logs together, be able to search them,
and be able to visualize them.

46. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 46
Unlike monitoring tools, there seems to be pretty broad consensus
on good tools here in deployments I’ve worked with….

47. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 47
http://www.elasticsearch.org/blog/openstack-elastic-recheck-powered-elk-stack/
(visualization)
(collection)
(search/analytics)

48. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 48
Questions?
@marktvoelker
http://openstack.org/
http://cisco.com/go/openstack/
(yes, we’re hiring!)