Nothing clears out a conference room faster than a discussion around information security. Securing complex computer systems, such as OpenStack clouds, is extremely difficult. To make matters worse, attackers can make many mistakes without consequences. A defender’s single mistake could lead to a breach. Don't let fear rule the discussion around security. Operators need a simple and scalable method for securing OpenStack clouds. That starts with grouping components into compartments and then looking at how those compartments interact with each other. Those interactions form the backbone of security policies and technical controls. In this vendor-neutral talk, Major Hayden, principal architect at Rackspace, will break down the complexity of securing OpenStack clouds using real-world scenarios. Attendees will learn how to: Divide OpenStack deployments into compartments Analyze the interactions between each component Develop security policies and apply technical controls

1. Holistic Security for
OpenStack Clouds
Major Hayden
Principal Architect, Rackspace
@majorhayden
Photo credit: bastiend (Flickr)

2. Image credit: Wikipedia

3. Security feels like this
Image credit: Wikipedia

4. Securing complex
systems creates
more challenges

5. Securing OpenStack can feel like
taking a trip to the Upside Down.

6. It doesn’t have to be that way
(even with something as complex as OpenStack)
Image credit: Pixabay

7. The key is
taking the right approach
to secure a complex system.

8. Major Hayden
Principal Architect
● At Rackspace since 2006
● Working on OpenStack since 2012
● Focused on information security for
Rackspace Private Cloud
● Fedora Linux contributor; Fedora Security
Team and Server Working Group member
● Has a terrible domain name purchase habit
(please, no ideas for domain names today)

9. Holistic
characterized by comprehension of the
parts of something as intimately
interconnected and explicable only by
reference to the whole
-- Oxford English Dictionary

10. The holistic approach for
humans considers a person
to be made of a body, a
mind, and a spirit.
Image credit: Pixabay

11. The holistic approach for
OpenStack considers
a cloud to be made of
servers, software, and a
business goal.

12. A holistic approach to security
involves people, processes,
and technologies working in
tandem.

13. “The whole is greater
than the sum of its parts,
especially in the case of OpenStack.”
-- (partially) Aristotle
Image credit: Wikipedia

14. How does this apply to
securing an OpenStack
cloud?
Let’s do a quick security
refresher.

15. Assume that attackers
will get inside eventually.
Image credit: Pixabay

16. Attackers are on offense.
They can be wrong many times.
Defenders can only be wrong
once for a breach to occur.

17. Securing only the outer perimeter
is not sufficient.

18. We must secure our OpenStack cloud.
We need to go deeper.

19. We just bought an expensive firewall for
the perimeter. Isn’t that enough?

20. (no caption necessary)

21. Build small security improvements
at multiple layers.*
* This is the cornerstone of defense-in-depth.

22. Individually, these changes may
not seem to have much value.
All of these changes create a
strong, valuable security strategy
when they are added together.

23. Let’s get to the good stuff.
Image credit: Pexels

24. Work from the outside in
(just like you would at a fancy dinner)
Image credit: Wikipedia

25. Four layers
Outer perimeter
Control and data planes
Control plane deep dive:
OpenStack services and backend services
OpenStack services deep dive
Image credit: imageme (Flickr)

26. The outer perimeter
Image credit: Pixabay

27. OUTER PERIMETER SECURITY GOAL:
Convince your attackers that
it’s easier to attack someone
else’s cloud

28. Key concepts
Make it expensive for attackers to
breach your perimeter defense
When they do make it through,
ensure that you know about it
immediately
Perimeters usually have openings
on the outside and inside --
secure both of them

29. Tactical
objectives
Require a VPN for access from
external networks
Segregate internal networks using
a firewall or an internally-facing
VPN
Monitor all logins (successful and
unsuccessful) for unusual activity
Track bandwidth usage trends
using netflow data

30. Secure the perimeter
VPN
Internet Corporate network
Firewall
Log collector Alert system
Netflow collector
Auth system

31. Control and data planes
Image credit: Pixabay

32. Control and data plane
Control plane
keystone, nova, glance,
cinder, neutron, horizon,
rabbitmq, mysql,
memcached
Data plane
Hypervisors and
tenant-built items (VMs,
containers, networks,
storage)

33. CONTROL/DATA PLANES SECURITY GOAL:
Keep the inner workings
of your OpenStack cloud
separated from
tenant infrastructure

34. Key concepts
Tenant infrastructure should have
extremely limited access to the
control plane, and vice versa
A misconfigured tenant VM could
open a wide hole in your secure
network
Protect your cloud from VM exit
exploits that allow attackers to
gain hypervisor access

35. Tactical
objectives
Separate control plane,
hypervisors and tenant
infrastructure with VLANs and
strict firewall rules (and monitor
dropped packets)
Use SELinux or AppArmor on
hypervisors to reduce the impact
of VM and container exit exploits

36. Hypervisor
Linux Security Module refresher
Three popular implementations:
SELinux, AppArmor, and TOMOYO
sVirt (in libvirt) ensures that all
processes are labeled properly
(SELinux) or have profiles configured
(AppArmor)
VM exit exploits are confined in most
situations
Tenant VM
Storage Network
Linux Security Module

37. Do not disable
SELinux or AppArmor
on your hypervisors.
(Seriously. Leave it enabled.)

38. Control plane deep dive:
OpenStack and backend services
Image credit: Wikipedia

39. CONTROL PLANE SECURITY GOAL:
Heavily restrict lateral
movement and restrict access
to the “crown jewels”
“crown jewels” are the databases and message queues
in your OpenStack cloud

40. Control plane deep dive
OpenStack services
keystone, nova, glance,
cinder, neutron, horizon
Backend services
mysql, rabbitmq,
memcached, syslog
The “crown jewels” are here
The map to the “crown
jewels” is here

41. Key concepts
Allow the least amount of access
possible from the OpenStack
services to backend services
Further restrict access to specific
ports, sources, and destinations
Deploy services into containers to
apply fine-tuned network and
process restrictions

42. Tactical
objectives
Use a load balancer or firewall to
create a “choke point” between
OpenStack and backend services
Monitor messaging and database
performance closely to look for
anomalies or unauthorized access
Use unique credentials for each
MySQL database and RabbitMQ
virtual host

43. OpenStack services deep dive
Image credit: Wikipedia

44. OPENSTACK SERVICES SECURITY GOAL:
Know what valid communication
looks like and alert on
everything else

45. OpenStack has many (predictable) interactions

46. Key concepts
OpenStack services are heavily
interconnected, but the
connections are predictable
Limit access between OpenStack
services and monitor any invalid
questions

47. Tactical
objectives
Use iptables rules to limit access
between OpenStack services; alert
on any invalid connections
Give each service a different
keystone service account (with
different credentials)
Monitor closely for high
bandwidth usage and high
connection counts

48. Let’s wrap up

49. Analyze.
Isolate.
Monitor.
Repeat.

50. These small security changes
add up to a strong defense
Image credit: Wikipedia

51. Try OpenStack-Ansible
OpenStack-Ansible deploys
enterprise-grade OpenStack clouds
using Ansible.
Security and reliability are two of the
core priorities for the project. Most of
the security changes in this talk are
already implemented.
Learn more:
http://bit.ly/openstack-ansible

52. RACKSPACE PRIVATE CLOUD
POWERED BY OPENSTACK®
Learn more about our
proven operational expertise,
industry-leading reliability,
and OpenStack Everywhere.
Join us at the Rackspace booth (A22)
in the OpenStack Marketplace.
RACKSPACE INVENTED
OPENSTACK® – NOW WE'RE
PERFECTING IT

53. Thank you!
Major Hayden
@majorhayden
major.hayden@rackspace.com
Photo credit: bastiend (Flickr)