Slides from our Q2 MeetUp

1. Q2 MeetUp
May 30th 2017

2. Agenda
17:45-18:25 - Check-in, food, drinks and networking
18:25-18:40 - Intro, Summit Recap and Reminders
18:40-19:20 - Operational War Stories
19:20-19:35 - Break
19:35-20:15 - Let’s talk about Ceph!

3. Let’s Say Thanks To Our Sponsors

4. Introductions

5. Stacy Véronneau - EGO Slide
● Director of OpenStack Solutions and Lead OpenStack
Architect at CloudOps.
● Using public cloud resources since 2007
○ When AWS only had 3 options :)
● Started ‘exploring’ OpenStack at Folsom
● OpenStack MeetUp organizer
○ Montreal, Ottawa, Edmonton and Toronto (Co-Org)
● Speaker and Mentor at OpenStack Summit
○ Austin, Barcelona, Boston
● Lazy Man repo
○ github.com/sveronneau

6. Now Let’s Talk OpenStack!

7. Summit Recap and Reminders
Stacy Véronneau - CloudOps

8. Summit Recap
● 5000+ Attendees
● 1014 Companies Represented
● 63 Countries Represented
● 750+ Sessions
● Average Day 1 Keynotes
○ Except the following:
■ U.S. Army Cyber School
● Going from 3 borrowed servers connected to users via CAT6
cables running through a drop ceiling to a 2000-core cluster
backed by a 4PB Ceph array that is 100% code-driven.

9. Summit Recap
● SuperUser award goes to
○ Paddy Power Betfair (Online gambling)
■ Built a true dev/0ps and continuous delivery model for developers using OpenStack
as the middleware. They’ve also migrated 25 percent of production applications onto
OpenStack in a year, for over 100 applications total. They grew from 500
deployments a week to over 1,000 deployments a day using the OpenStack APIs to
increase time-to-market.
(https://betsandbits.com/2016/10/25/openstack-reference-architecture/)
○ UKCloud (Government)
■ The leading infrastructure-as-a-service (IaaS) provider to the United Kingdom
public sector with a 38 percent share on G-Cloud–the framework set up by the UK
government for IT procurement. From online tax returns with HMRC, complex data
analytics at Genomics England and integrated vehicle and driver records at the
Driver and Vehicle Licensing Agency (all hosted on UKCloud), pooling central
government resources and moving them to the cloud has resulted in £600 million in
savings, leading to the UK being recognized by the United Nations as the most
digitally advanced government in the world.

10. Summit Recap
● Day 2 Keynotes and demos saved the day!
○ Great demos (some working, some not)
○ Interop challenge
■ A big ‘Woot!’ to MO from Vexxhost
○ Edward Snowden live
■ Privacy in an always connected world

11. Summit Recap
● Day 2 is still a challenge
for many OpenStack
deployments.
● Almost more monitoring
vendors than storage one at
this summit.

12. Summit Recap
● What application tools run on your OpenStack???
○ K8s-45% ; OpenShift-18% ; CloudFoundry-18% ; Built Our Own-17% ;
Mesos-14% ; Docker Swarm 14% ; Other 17%
● PCaaS is getting more and more traction
○ Hosted private or remotely managed private clouds
● All the cool videos from the Summit can be found at:
○ https://www.openstack.org/videos/summits/boston-2017

13. Summit Recap
● OpenStack Canada
Slack Channel
Gathering and Party
Crashing :)

14. Summit Recap
● Best T-Shirt???

15. Reminders
● Next MeetUp will be in
September
○ We’ll have an OpenContrail talk
○ But we need more!
● Submit your talk proposal via
MeetUp page
● Possible OpenStack Days Canada
- Ottawa in October/November

16. Reminders
● Link to today’s presentations will be
shared on the Meetup page.
● Join your fellow Stackers on
○ http://openstack-canada-slack-invite.herokuapp.com
○ Pages section on the MeetUp site

17. New tools addition for the community
● GitHub Repo
○ https://github.com/openstack-canada-repo
● Etherpad
○ https://etherpad.openstack.org/p/openstack-canada

18. Stacker Talk
Operational War Stories
Mohammed Naser - VEXXHOST

19. ● Mohammed Naser
● Deploying and contributing since 2011
● CEO @ VEXXHOST, Inc.
● OpenStack Corporate Sponsor & Infrastructure Donor
● Architected many cloud deployments, migrations
● Find me on Twitter: @_mnaser
● Always accessible on Freenode IRC: mnaser (or the Canadian users Slack!)
So, who are you again?

20. No clouds were harmed in the
production of this presentation

21. Upgrades
● Production installs absolutely need 3 controllers
● Load balancers are your life line
● Automation is an absolute must (I personally recommend Ansible)!
● A single controller should have enough capacity to run a single OpenStack service
● The magic mix:
○ Backups, backups, backups!
○ Shutdown a specific OpenStack service from all nodes except a single one
○ Upgrade service on that single node with automation (you tested this before, right?)
■ If upgrade is success, complete process on other controllers, re-add to load balancers
■ If upgrade is a fail, collect all logs, shutdown upgrade controller, restore database, startup old
controllers, leave upgrade controller in place for further inspection
○ Go out and celebrate, only to be paged in that something stopped working.

22. Testing & Monitoring
● Upgrade went great, or so you thought.
● OpenStack is a huge system, a simple smoke test of “hey I can create a VM” will likely not cover
the most common use cases (or it could in a private cloud, but context of a public cloud differs).
● Take advantage of Tempest
● Tempest is the most advanced set of OpenStack tests covering many different tools. Run that
before and after upgrades and check on any failures.
● Take it a step further and run it on a nightly basis to see what sort of things might have changed
in your environment
● Really have some spare time while running OpenStack (maybe you should speak about how you
do that next time?), run Rally against your cloud to benchmark it or run smoke tests.
● Now, you think you can get some peace as everything is monitored, but management wants
centralized storage...

23. Centralized Storage
● There are many great open source centralized storage systems, I Ceph
● You might have the luxury of redeploying your cloud. We can’t really do that.
● With local storage, instances are stored as qcow2 files in /var/lib/nova/instances/<uuid>/..
● With Ceph, instances are stored in <pool_name>/<uuid>_disk
● Quick guide to changing the entire architecture of your OpenStack cloud (bonne chance!):
○ Deploy a Ceph cluster to host all your infrastructure
○ Configure new compute nodes which use that Ceph cluster for storage
○ Disable all the compute nodes not using Ceph
○ Convert all your images from QCOW2 to RAW in Glance
○ WARNING: close your eyes and breath heavily now
■ Shut down VMs, use qemu-img to convert qcow2 disks directly to Ceph matching the correct
name, edit Nova database to point to a new compute node that fits, start up the VM again.
● Now you have centralized storage but now you have a bunch of machines with unused drives.

24. Hyperconverged infrastructure
● It is possible! We run it on our public cloud and all of our customer private clouds.
● Hyperconverged Ceph + Compute for OpenStack
● Few things to keep in mind:
○ We use SSD storage only, don’t think this would make sense for non-SSD storage.
○ SSD specific note: Don’t cheap out on drives. Seriously. Intel SSDs work best in our experience.
○ Keep your OSD per machine count small. Don’t run 24 OSDs and 400 VMs. Be a bit realistic.
○ CPU pinning sounds good on paper, until your cluster sustains high load, pinned CPUs are waiting for IO
and the OSD can’t do anything.
○ Use `cgroups` instead to give OSDs fully dedicated cores on the machine.
● This architecture minimizes the hardware you have to setup and makes the cloud a lot easier to
scale.
● Everything is going great, except your Linux distro is being a PITA?

25. Switching operating systems
● We originally built our cloud off of the Ubuntu Cloud Archives
● David Moreau Simard presented RDO which is (IMHO) the best way to package OpenStack
● Exciting. We can fix bugs now!
● So now you got a hyperconverged infrastructure with open source technology and live migration?
○ Deploy a few CentOS based nodes using RDO packaging with Ceph installed (side note: use the SIG pkgs!)
○ The RPC layer of OpenStack communicates with no problems, it doesn’t care about the operating system.
○ Run a live migration and VMs will be populated on the new operating system and everything will be great.
○ Just kidding, you didn’t think it was easy?
■ It’s a mess. AppArmor is security model on Ubuntu, SELinux is the model on CentOS, many patches
later and questionable decisions were done to make live migrations successful! I’d recommend cold
migrations instead (but downtime, sigh!)
● Now you want to migrate your control plane?

27. Migrating OpenStack control plane
● Many reasons you want to do this: change deployment tool, upgrades, hardware replacement
● For most of the OpenStack services which are stateless, you just run the binaries and update your
load balancers
● The “fun” part comes with the underlying stateful infrastructure of OpenStack such as RabbitMQ
and Galera
● Fun stories (aka: “that one time at band camp”):
○ That time some clients interface order were reversed after we migrated our Galera cluster
○ Orchestrating a RabbitMQ cluster cutover with the least downtime possible (chicken and egg problem)
○ RabbitMQ melt down featuring Neutron self destruction
● Closing thought: Remove your old services. Seriously. Customer once started up a
decommissioned controller node so Nova compute and Neutron agents were getting fed different
information from two different databases with a race condition.

28. Everything works in
OpenStack, except
for when it doesn’t.

29. Thanks!
Reach out:
Email: mnaser@vexxhost.com
IRC: mnaser @ freenode
Twitter: @_mnaser
LinkedIn: linkedin.com/in/mdnaser

30. Break Time!

31. Stacker Talk
Let’s talk about Ceph!
Marcos Garcia - Red Hat

32. Let’s talk about Ceph
Marcos Garcia
marcos.garcia@redhat.com
Senior Cloud Solutions Architect, Red Hat
OpenStack Montreal meetup
May 30th, 2017

33. Free e-Book!
www.redhat.com/en/resources/openstack-storage-dummies
https://goo.gl/XsFIuS

34. Agenda
● Ceph 101 refresh: Object, Block and File Storage
● Hyper Converged OpenStack deployments
● Containerized Ceph
● Ceph for Big Data
● What’s coming in 2017 www.redhat.com/en/resources/openstack-storage-dummies
https://goo.gl/XsFIuS

35. Ceph 101 refresh

36. Unified Storage
OBJECT STORE
RESTful Interface
S3- and Swift-compliant APIs
S3-style subdomains
Unified S3/Swift namespace
User management
Usage tracking
Striped objects
Cloud solution integration
Multi-site deployment
Disaster recovery
BLOCK DEVICE
Thin-provisioned
Images up to 16 exabytes
Configurable striping
In-memory caching
Snapshots
Copy-on-write cloning
Kernel driver support
KVM/libvirt support
Back-end for cloud
solutions
Incremental backup
Disaster recovery
FILESYSTEM
POSIX-compliant
semantics
Separates metadata from
data
Dynamic rebalancing
Subdirectory snapshots
Configurable striping
Kernel driver support
FUSE support
NFS/CIFS deployable
Use with Hadoop
(replace HDFS)
http://docs.ceph.com/docs/master/

37. Architecture

38. Architecture
OBJECT STORE
BLOCK DEVICE
FILESYSTEM

39. Monitor nodes

40. Cluster Topology: CRUSH

41. Data Storage and Replication
Ceph OSD Daemons store all data as objects in a
flat namespace (e.g., no hierarchy of directories). An
object has an identifier, binary data, and metadata
consisting of a set of name/value pairs.

42. Erasure Coding or not?

43. Use Case: Web Storage

44. Use Case: Cold Storage

45. Use Case: OpenStack Storage

46. Hyper
Converged
OpenStack

47. Why HCI?
● Colocate Nova+KVM with Ceph OSDs in the same node
● HCI lets us deploy a smaller footprint
○ HCI: 6 nodes for HA (+ director)
■ 3 controllers/monitors + 3 computes/OSDs
○ Non-HCI: 9 nodes for HA (+ director)
■ 3 controllers/monitors + 3 computes + 3 OSDs
● Further standardization of hardware
○ Hardware vendors may offer a discount for more of the same type of server
○ Fewer server types simplify operations
○ May enable more efficient use of hardware resources

48. To HCI or not HCI
Compute VM VM
Storage OSD OSD
Compute VM VM
Storage OSD OSD
Compute VM VM
Storage OSD OSD
HCI VM OSD
HCI VM OSD
HCI VM OSD
HCI VM OSD
HCI VM OSD
HCI VM OSD

49. More info on HCI
Red Hat OpenStack Platform 10
(Newton) with Red Hat Ceph
Storage 2 (Jewel)
https://access.redhat.com/documentation
/en-us/reference_architectures/2017/html
/hyper-converged_red_hat_openstack_p
latform_10_and_red_hat_ceph_storage_
2/
A proper network design is very
important

50. Containerized
Ceph
Deployment

51. Containerized Ceph - Why?
DATA ANALYTICS JOBS
Run storage services alongside with other application containers
Resources are:
precisely allocated
orchestrated
monitored
restrained
ie: NFV components run during the day while other data analytics jobs using
Ceph containers would run in the night.

52. Containerized Ceph - Why?
ON-DEMAND STORAGE CLUSTER
Create storage instances that meet specific QoS requirements for different
tenants
Hyper-converged model with multiple Ceph clusters on the same machines
Deploy Ceph development environments (PaaS)
Integrate with in-house products: OpenShift and Atomic Host/Platform

53. Containerized Ceph - Setup

54. Containerized Ceph - For more info
https://github.com/ceph/ceph-docker/tree/centos
Special thanks to
Sébastien Han <seb@redhat.com> http://www.sebastien-han.fr/
Deepthi Dharwar <ddharwar@redhat.com>
Narendra Narang <nnarang@redhat.com>

55. Big Data

56. Big Data is a top CIO priority
BI & Analytics will see most increase in 2017 spend
http://bit.ly/2jrMK8V
“Analytics is one the top three disruptive
forces of software in the enterprise market”

57. Big Data Evolution: 2nd Gen data lakes
● Structured Data
● Low Scale
● Retrospective reporting
● Strong Governance
● 1st Gen Big Data era
● Initial Hadoop installs
● Team-centric deployment
● Ad-hoc discoverability & governance
● 2nd Gen Big Data era
● Business-centric deployment
● Centralized discovery &
Governance
Data
Warehouse
Data
Swamp
Data
Lake
1980-2005 2005 onwards Now

58. Current Gen 1 Analytics: Monolithic
Analytics
+
Infrastructure
Hadoop vendors do
analytics software …and single-purpose
infrastructure

59. Hadoop + HDFS: Batch Analytics
BATCH
Companies experience difficulty expanding the benefits of Hadoop.
to other
warehouses
and
repositories
INGEST EXIT

60. Result: Hadoop cluster silos
BATCH
SILO
STREAMING
SILO
INTERACTIVE
SILO
Specialized analytics engines built in multiple silos

61. Gen 2 Analytics: Split infrastructure
Analytics
Infrastructure
OpenStack or OpenShift Provisioned Compute Pool
Ceph Common Object Store
They do
analytics
software
Red Hat does
infrastructure
software

62. Better handling of Data Gravity
Since large data sets are expensive to move, analytics engines orbit the data
For data generated by internal systems and devices,
this can inform the decision whether to store
on-premise or expatriate to cloud providers

63. New pattern
Disaggregating compute resources from an object storage solution enables the most flexibility
INGEST from multiple sources
using Ceph’s S3 API
ANALYTICS operate directly on
common data lake without
duplicating datasets onto
multiple special-purpose clusters
CLUSTERS provisioned
dynamically optimized for batch,
interactive, or query engines
EXPLORATORY analysis support
by ephemeral clusters

64. Ceph 2017
Luminous
Full presentation and slides from Sage
Well in OpenStack Summit Boston 2017
https://www.openstack.org/videos/boston-
2017/ceph-project-update
https://www.slideshare.net/sageweil1/com
munity-update-at-openstack-summit-bost
on

65. Ceph Releases
Red Hat
Storage 2

66. Bluestore: stable and default
New OSD backend
consumes raw block device(s)
no more XFS
Fast on both HDDs (~2x) and SSDs (~1.5x)
Smaller journals
happily uses fast SSD partition(s) for internal metadata, or NVRAM for journal
Inline compression (zlib, snappy)
policy driven by global or per-pool config, and/or client hints
Stable and default

67. Bluestore: performance gains

68. Ceph-Mgr: Management improvements
Ceph-mgr
new management daemon to reduce burden on ceph-mon (monitor)
easier integration point for python management logic
integrated metrics
make ceph-mon scalable again
offload pg stats from mon to mgr
push to 10K OSDs (planned “big bang 3” test @ CERN)
new REST API
Pecan, based on previous Calamari API
built-in web dashboard
webby equivalent of 'ceph -s'

69. Ceph-Mgr: Management improvements

70. Improvements
AsyncMessenger
new network Messenger implementation, event driven, more performance
RDMA backend (ibverbs), built by default, limited testing
DPDK backend - prototype!
Perfectly Balanced OSDs
CRUSH improvements: alternate weight sets, flexibility to optimize weights
Pg upmap: simple optimizer, explicitly mapping PGs to specific devices
Other RADOS improvements

71. RadosGW Improvements
RGW Metadata search via Elastic Search
Search across Multi-site deployments, shared containers
NFS gateway
NFSv4 and v3
full object access (not general purpose!)
dynamic bucket index sharding (automatically)
inline compression
Encryption, following S3 encryption APIs

72. CephFS
multiple active MDS daemons (finally!)
subtree pinning to specific daemon
directory fragmentation on by default
(snapshots still off by default)
so many tests
so many bugs fixed
kernel client improvements
CephFS is the backend of choice for OpenStack Manila

73. ARM-based intelligent disks for Ceph
(For Mimic release)
http://ceph.com/community/500-osd-ceph-cluster/

74. Day
June 20th - Sheraton Montreal
http://www.corpium.ca/meeting-reg/redhat/FY18/RHDmontreal/
https://goo.gl/IrhWlN

75. Thanks!

76. Thanks Everyone! --- See you in Q3