Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
BUILDING AND RUNNING OPENSTACK ON
POWER8
LANCE ALBERTSON
Oregon State University Open Source Lab
@ramereth
0
SUMMARY
POWER8 Overview
POWER at OSUOSL
Building a RHEL-based P8 platform with Openstack
Architecture porting issues
Probl...
DISCLAIMER
POWER8 OVERVIEW
DESIGN
Designed to be a massively multithreaded chip
Designs are available for licensing under the OpenPOW...
OPENPOWER ABSTRACTION LAYER (OPAL)
OPAL is the new Open Source firmware for POWER8
Acts as an on-system HMC
Enables the ma...
POWER AT OSUOSL
HISTORY
Providing PPC64 compute resources since 2005
Close collaboration with IBM LTC
POWER5, POWER7 and n...
POWER8 AT OSUOSL
Goal is to provide on-demand PPC64/PPC64LE compute resources to FOSS
projects
Assist with ppc64/ppc64le p...
PROJECTS RUNNING ON OUR P8 CLUSTER
CloudFoundry, Docker, CentOS, CouchDB
Haskell, Glibc, JXcore, LLVM, NodeJS
OpenJDK, GoL...
BUILDING A RHEL-BASED P8 PLATFORM WITH OPENSTACK
SUPPORTED OS PLATFORMS
PowerKVM
Ubuntu
RHEL
DECISION TO USE RHEL
Little community support at the time and opportunity to help the community
We use CentOS internally a...
OPENSTACK ARCHITECTURE (OLD)
Started in 2014
Icehouse
Controller node
Runs all public API services, dashboard
DB hosted on...
OPENSTACK ARCHITECTURE (NEW)
Deployed 2016 (deployed last week)
Mitaka
Controller node
Runs all public API services, dashb...
COMPUTE NODES
Did initial development on Fedora 19
Fedora 20 PPC64 base system (old)
Fedora 21 versions of a few packages
...
ARCHITECTURE PORTING ISSUES
CHEF
No PPC64/PPC64LE Chef client
Needed to build our own chef-client
Omnibus
Bootstrap build ...
PACKAGE SUPPORT
Support for P8 was bleeding edge and new features were added weekly
Built versions of latest packages from...
GUEST OS IMAGES
Few OS supported ppc64/ppc64le or provided guest images pre-built
Variety of tools which are platform spec...
PACKER -- MULTI PLATFORM SUPPORT
We needed Go to use Packer
GoLang support was literally in the works
Finally built our ow...
ARCHITECTURE ISSUES
OPAL firmware bugs
pre-P8 machines were very buggy
IPMI console would sometimes stop working
Random lo...
PROBLEMS WE HAVE ENCOUNTERED WITH OPENSTACK
LEARNING AND UNDERSTANDING OPENSTACK
Lots of moving pieces
Neutron networking ...
BUGS AND "FEATURES"
Interaction between libvirt and nova-compute was buggy at times
Some bugs were just Icehouse itself, o...
STABILITY
Rabbitmq would constantly need to be restarted
nova-compute services would randomly stop working
Running Fedora ...
CENTOS 7 PPC64LE ON OPENSTACK
RHEL / CENTOS SUPPORT
Introduced in 7.1 and fully supported in 7.2
CentOS community was stil...
RDO
Community for deploying Openstack on CentOS, Fedora and RHEL
Repositories built against each Platform
Each release of ...
RHEV (RED HAT ENTERPRISE VIRTUALIZATION)
Updated KVM packaging
Part of the Virt SIG of CentOS
Used SRPMs to build ppc64le ...
OPENSTACK DEPLOYMENT WITH CHEF
WHY CHEF?
Primary CM tool used at the OSL
Provides a lot of testing capability on deploymen...
CHEF OPENSTACK
Set of cookbooks that will deploy the various services of Openstack
Part of the OpenStack umbrella
Communit...
OSL OPENSTACK
Created a wrapper cookbook (osl-openstack)
OSL site specific configuration
Split recipes out by upstream coo...
OSL WRAPPER COOKBOOK
RECIPES/DEFAULT.RB
Default configuration for cluster
Include local yum repos
Include command clients
...
RECIPES/IDENTITY.RB
Just includes recipes
Some wrapper, some upstream
Allows us to test just Keystone by itself
recipes/id...
RECIPES/CONTROLLER.RB
Pulls in all wrapper recipes needed to build a controller
Allows for us to split things out eventual...
TESTING AND DEVELOPMENT
Unit Testing
ChefSpec
RSpec
Integration Testing
Test Kitchen
ServerSpec
Chef Provisioning
Deploy V...
UNIT TESTING
Ensure the Chef code is doing what it’s supposed to do
Easily test Architecture-specific logic
Verify configu...
UNIT TESTING (OUTPUT)
$ rspec spec/default_spec.rb
osl­openstack::default
  includes cookbook base::ifconfig
  includes co...
TEST KITCHEN & SERVERSPEC
Test Kitchen
Test CLI tool which allows you to execute the configured code on one or
more platfo...
TEST KITCHEN (LIST)
$ kitchen list
Instance                            Driver     Provisioner  Verifier  Transport  Last A...
TEST KITCHEN (TEST)
$ kitchen test default
­­­­­> Starting Kitchen (v1.8.0)
­­­­­> Cleaning up any prior instances of <def...
NEXT STEPS
INFRASTRUCTURE NEXT STEPS
Add Nagios checks (DONE!)
Continue to fix bugs and other issues as they come up
Rebui...
PROJECT EXPERIENCE
Improve and streamline on boarding process
Expand cluster’s disk storage capacity
Improve stability of ...
QUESTIONS?
Lance Albertson
lance@osuosl.org
@ramereth
–http://osuosl.org http://lancealbertson.com
Links:
http://github.co...
Upcoming SlideShare
Loading in …5
×

Building and Running OpenStack on POWER8

266 views

Published on

Over the past two years the OSL has been building a POWER8 based OpenStack environment working in conjuction with IBM. The purpose of this environment is to provide a stable yet flexible infrastructure for FOSS projects to port and test their code on the new PowerPC 64bit Little Endian (ppc64le) architecture.

This session will cover various aspects of the path we took to build and continue to run the environment. Some topics will include some of the initial challenges we faced and how we solved them. In addition, we’ll cover some of the specific porting issues we ran into with ppc64le. We’ll also cover some of the major issues we ran into with OpenStack specifically on ppc64le and how we solved them. And finally, we’ll discuss the future of the cluster and the work we’ve put into designing it.

Attendees to this session should have some foundational knowledge of the POWER and OpenStack ecosystem. If you’re interested in architecture porting issues and/or interested in OpenStack deployment war stories, this session might be for you.

Published in: Internet
  • Be the first to comment

  • Be the first to like this

Building and Running OpenStack on POWER8

  1. 1. BUILDING AND RUNNING OPENSTACK ON POWER8 LANCE ALBERTSON Oregon State University Open Source Lab @ramereth 0
  2. 2. SUMMARY POWER8 Overview POWER at OSUOSL Building a RHEL-based P8 platform with Openstack Architecture porting issues Problems we have encountered with OpenStack Openstack deployment with Chef OSL Wrapper cookbook Next Steps 0
  3. 3. DISCLAIMER
  4. 4. POWER8 OVERVIEW DESIGN Designed to be a massively multithreaded chip Designs are available for licensing under the OpenPOWER Foundation Little-Endian & Big-Endian Several non-IBM companies building P8 hardware Tyan, Rackspace (OpenCompute-based) & Google
  5. 5. OPENPOWER ABSTRACTION LAYER (OPAL) OPAL is the new Open Source firmware for POWER8 Acts as an on-system HMC Enables the machine to boot similar to PC servers Linux Kernel and loads the boot loader Petitboot Petitboot provides a shell environment for debugging and setup Petitboot will use kexec and boot into the system kernel
  6. 6. POWER AT OSUOSL HISTORY Providing PPC64 compute resources since 2005 Close collaboration with IBM LTC POWER5, POWER7 and now POWER8 OSL managed LPAR deployment to make it easier on projects Pre-P8 Projects: Debian, Gentoo, Fedora, PostgreSQL Linux Foundation, Haskell, GoLang Mozilla, OpenSUSE, LLVM, GCC
  7. 7. POWER8 AT OSUOSL Goal is to provide on-demand PPC64/PPC64LE compute resources to FOSS projects Assist with ppc64/ppc64le porting & testing Expose OSU students to OpenStack and POWER8 Collaboration with IBM engineers on architecture issues Create a vanilla Openstack cluster for FOSS projects
  8. 8. PROJECTS RUNNING ON OUR P8 CLUSTER CloudFoundry, Docker, CentOS, CouchDB Haskell, Glibc, JXcore, LLVM, NodeJS OpenJDK, GoLang, oVirt, libjpeg-turbo BLCR, Gentoo
  9. 9. BUILDING A RHEL-BASED P8 PLATFORM WITH OPENSTACK SUPPORTED OS PLATFORMS PowerKVM Ubuntu RHEL
  10. 10. DECISION TO USE RHEL Little community support at the time and opportunity to help the community We use CentOS internally as our primary OS & more familiar with the RHEL eco-system RHEL has the RDO OpenStack distribution that is well supported Chef support with OpenStack needed some help I love challenges!
  11. 11. OPENSTACK ARCHITECTURE (OLD) Started in 2014 Icehouse Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 6 VM running on Ganeti+KVM Compute node(s) Nova compute and networking Flat networking PPC64 Fedora
  12. 12. OPENSTACK ARCHITECTURE (NEW) Deployed 2016 (deployed last week) Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running on Ganeti+KVM Compute node(s) Nova Compute Neutron Networking Linuxbridge Provider and Tenant networking using VXLAN PPC64LE CentOS 7.2
  13. 13. COMPUTE NODES Did initial development on Fedora 19 Fedora 20 PPC64 base system (old) Fedora 21 versions of a few packages CentOS 7.2 PPC64LE base system (new)
  14. 14. ARCHITECTURE PORTING ISSUES CHEF No PPC64/PPC64LE Chef client Needed to build our own chef-client Omnibus Bootstrap build env Build dependency issues Architecture configuration issues in Omnibus Chef has stable ppc64/ppc64le builds today
  15. 15. PACKAGE SUPPORT Support for P8 was bleeding edge and new features were added weekly Built versions of latest packages from Fedora rawhide packages: qemu libvirt kernel Internal repo for these custom packages: Kernel required a few custom options to be enabled Runtime setup: Disable SMT http:// p.osuosl.org/pub/osl/repos/yum/openpower/centos-7/ppc64le/
  16. 16. GUEST OS IMAGES Few OS supported ppc64/ppc64le or provided guest images pre-built Variety of tools which are platform specific Missing support for cloud-init Initially started creating images manually with qemu directly
  17. 17. PACKER -- MULTI PLATFORM SUPPORT We needed Go to use Packer GoLang support was literally in the works Finally built our own packer binary last Nov! WIP Packer Templates: http:// p.osuosl.org/pub/osl/openpower/rpms/ https://github.com/osuosl/bento/tree/ramereth/ppc64
  18. 18. ARCHITECTURE ISSUES OPAL firmware bugs pre-P8 machines were very buggy IPMI console would sometimes stop working Random lockups Included HW RAID, but no cached write-back support
  19. 19. PROBLEMS WE HAVE ENCOUNTERED WITH OPENSTACK LEARNING AND UNDERSTANDING OPENSTACK Lots of moving pieces Neutron networking is complex and a moving target Deciding on the proper design architecture for our use case
  20. 20. BUGS AND "FEATURES" Interaction between libvirt and nova-compute was buggy at times Some bugs were just Icehouse itself, others were architecture specific Learning how to deploy Openstack and making (gasp) mistakes! Iptables issues between Chef and Openstack Provider networks configures dnsmasq as an open resolver SSL API endpoints
  21. 21. STABILITY Rabbitmq would constantly need to be restarted nova-compute services would randomly stop working Running Fedora on compute and CentOS on controller made things ... interesting
  22. 22. CENTOS 7 PPC64LE ON OPENSTACK RHEL / CENTOS SUPPORT Introduced in 7.1 and fully supported in 7.2 CentOS community was still bootstrapping and testing We built our own pre-release CentOS 7.2 for testing Using ppc64le on compute nodes
  23. 23. RDO Community for deploying Openstack on CentOS, Fedora and RHEL Repositories built against each Platform Each release of OpenStack separated
  24. 24. RHEV (RED HAT ENTERPRISE VIRTUALIZATION) Updated KVM packaging Part of the Virt SIG of CentOS Used SRPMs to build ppc64le versions in a location repo One patch needed to work around bug
  25. 25. OPENSTACK DEPLOYMENT WITH CHEF WHY CHEF? Primary CM tool used at the OSL Provides a lot of testing capability on deployment Can use the full power of the Ruby language for configuring the cluster
  26. 26. CHEF OPENSTACK Set of cookbooks that will deploy the various services of Openstack Part of the OpenStack umbrella Community driven Did a major refactor of the code for Mitaka release
  27. 27. OSL OPENSTACK Created a wrapper cookbook (osl-openstack) OSL site specific configuration Split recipes out by upstream cookbook name Contains ppc64le specific changes Currently only tested on CentOS 7 https://github.com/osuosl-cookbooks/osl-openstack
  28. 28. OSL WRAPPER COOKBOOK RECIPES/DEFAULT.RB Default configuration for cluster Include local yum repos Include command clients Logic around endpoints recipes/default.rb
  29. 29. RECIPES/IDENTITY.RB Just includes recipes Some wrapper, some upstream Allows us to test just Keystone by itself recipes/identity.rb
  30. 30. RECIPES/CONTROLLER.RB Pulls in all wrapper recipes needed to build a controller Allows for us to split things out eventually if we want to recipes/controller.rb
  31. 31. TESTING AND DEVELOPMENT Unit Testing ChefSpec RSpec Integration Testing Test Kitchen ServerSpec Chef Provisioning Deploy VMs as controller/compute Deploy on bare-metal for a test cluster
  32. 32. UNIT TESTING Ensure the Chef code is doing what it’s supposed to do Easily test Architecture-specific logic Verify configuration files contain proper settings Examples: spec/default_spec.rb spec/compute_controller.rb spec/linuxbridge_spec.rb
  33. 33. UNIT TESTING (OUTPUT) $ rspec spec/default_spec.rb osl­openstack::default   includes cookbook base::ifconfig   includes cookbook selinux::permissive   includes cookbook yum­qemu­ev   includes cookbook openstack­common   includes cookbook openstack­common::logging   includes cookbook openstack­common::sysctl   includes cookbook openstack­identity::openrc   includes cookbook openstack­common::client   includes cookbook openstack­telemetry::client   setting arch to x86_64     does not add OSL­Openpower repository on x86_64   setting arch to ppc64     add OSL­openpower­openstack repository on ppc64   setting arch to ppc64le
  34. 34. TEST KITCHEN & SERVERSPEC Test Kitchen Test CLI tool which allows you to execute the configured code on one or more platforms Integrates with testing frameworks Must have tool for Chef users Configured via ServerSpec RSpec tests for configured servers Integration tests Ensures things actually happen on the system Example: .kitchen.yml test/integration/default/serverspec/default_spec.rb
  35. 35. TEST KITCHEN (LIST) $ kitchen list Instance                            Driver     Provisioner  Verifier  Transport  Last Action default­centos­72                   Openstack  ChefZero     Busser    Rsync      Not Created mon­centos­72                       Openstack  ChefSolo     Busser    Rsync      Not Created mon­controller­centos­72            Openstack  ChefSolo     Busser    Rsync      Not Created ops­messaging­centos­72             Openstack  ChefZero     Busser    Rsync      Not Created identity­centos­72                  Openstack  ChefZero     Busser    Rsync      Not Created image­centos­72                     Openstack  ChefZero     Busser    Rsync      Not Created network­centos­72                   Openstack  ChefZero     Busser    Rsync      Not Created linuxbridge­centos­72               Openstack  ChefZero     Busser    Rsync      Not Created compute­controller­centos­72        Openstack  ChefZero     Busser    Rsync      Not Created compute­centos­72                   Openstack  ChefZero     Busser    Rsync      Not Created dashboard­centos­72                 Openstack  ChefZero     Busser    Rsync      Not Created block­storage­centos­72             Openstack  ChefZero     Busser    Rsync      Not Created block­storage­controller­centos­72  Openstack  ChefZero     Busser    Rsync      Not Created telemetry­centos­72                 Openstack  ChefZero     Busser    Rsync      Not Created controller­centos­72                Openstack  ChefZero     Busser    Rsync      Not Created
  36. 36. TEST KITCHEN (TEST) $ kitchen test default ­­­­­> Starting Kitchen (v1.8.0) ­­­­­> Cleaning up any prior instances of <default­centos­72> ­­­­­> Destroying <default­centos­72>...        Finished destroying <default­centos­72> (0m0.00s). ­­­­­> Testing <default­centos­72> ­­­­­> Creating <default­centos­72>...        OpenStack instance with ID of <a25fa410­5caf­4f96­bddb­1e6daddd06d9> is ready. ... Chef Client finished, 115/198 resources updated in 03 minutes 12 seconds Finished converging <default­centos­72> (3m41.31s). ­­­­­> Setting up <default­centos­72>... Finished setting up <default­centos­72> (0m0.00s). ­­­­­> Verifying <default­centos­72>... Preparing files for transfer ...
  37. 37. NEXT STEPS INFRASTRUCTURE NEXT STEPS Add Nagios checks (DONE!) Continue to fix bugs and other issues as they come up Rebuild old Icehouse cluster as Mitaka (no upgrade) Add support for object storage Update documentation Add support for non-live migration Mellanox networking
  38. 38. PROJECT EXPERIENCE Improve and streamline on boarding process Expand cluster’s disk storage capacity Improve stability of the cluster Add more projects! Submit your request: http://osuosl.org/services/powerdev/request_hosting
  39. 39. QUESTIONS? Lance Albertson lance@osuosl.org @ramereth –http://osuosl.org http://lancealbertson.com Links: http://github.com/ramereth/presentation-openstack-power8 https://github.com/osuosl-cookbooks/osl-openstack http://osuosl.org/services/powerdev/request_hosting http:// p.osuosl.org/pub/osl/repos/yum/openpower/centos-7/ppc64le/ http:// p.osuosl.org/pub/osl/openpower/ Attribution-ShareAlike CC BY-SA ©2016

×