Openstack In Real Life

20,402 views

Published on

Presentation on a real live production Openstack deployment done by Cloudscaling. Presented to the SF LSPE group on April 19, 2012.

Published in: Technology, Education
0 Comments
23 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
20,402
On SlideShare
0
From Embeds
0
Number of Embeds
6,914
Actions
Shares
0
Downloads
410
Comments
0
Likes
23
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • SKIP after “no”\n
  • SKIP after “no”\n
  • SKIP after “no”\n
  • SKIP after “no”\n
  • SKIP after “no”\n
  • \n
  • \n
  • SKIP\n
  • SKIP\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Openstack In Real Life

    1. 1. Openstack Clouds IRL Paul Guth CloudscalingJuly 26th, 2012 1LA #openstack meetup
    2. 2. Me • @pgutheb • wordpress.com/constructolution • slideshare.net/pgutheb • paul | at | cloudscaling - dot - comJuly 26th, 2012LA #openstack meetup 2
    3. 3. Stuff I will say today • Context setting • What do you do before you build your Openstack cloud • What do you do while you’re building your Openstack cloud • What do you do after you build your Openstack cloudJuly 26th, 2012LA #openstack meetup 3
    4. 4. What is Openstack? • Open source cloud management system (IaaS) • Five components: • Nova - VMs • Swift - Object Storage • Glance - Images • Horizon - Dashboard/UI • Keystone - Authentication • Great Openstack architecture overview from Ken PeppleJuly 26th, 2012LA #openstack meetup 4
    5. 5. What is a private cloud?July 26th, 2012LA #openstack meetup 5
    6. 6. What is a private cloud? • BlahblahblahblahblahblahblahJuly 26th, 2012LA #openstack meetup 5
    7. 7. What is a private cloud? • Blahblahblahblahblahblahblah • “I am not a dictionary, I am a free man!” - Number 6July 26th, 2012LA #openstack meetup 5
    8. 8. What Have We Done?July 26th, 2012LA #openstack meetup 6
    9. 9. What Have We Done? • Private cloud deployment for BigCo (not a Web co)July 26th, 2012LA #openstack meetup 6
    10. 10. What Have We Done? • Private cloud deployment for BigCo (not a Web co) • ~10 distinct internal groups using • So exercises multi-tenant features • Being used for new cloud-y appsJuly 26th, 2012LA #openstack meetup 6
    11. 11. What Have We Done? • Private cloud deployment for BigCo (not a Web co) • ~10 distinct internal groups using • So exercises multi-tenant features • Being used for new cloud-y apps • Multiple sites set up as independent regions (1 AZ/ region)July 26th, 2012LA #openstack meetup 6
    12. 12. What Have We Done? • Private cloud deployment for BigCo (not a Web co) • ~10 distinct internal groups using • So exercises multi-tenant features • Being used for new cloud-y apps • Multiple sites set up as independent regions (1 AZ/ region) • “Several dozen” compute nodes per site (VM capacity of “a few thousand”)July 26th, 2012LA #openstack meetup 6
    13. 13. What Have We Done? • Private cloud deployment for BigCo (not a Web co) • ~10 distinct internal groups using • So exercises multi-tenant features • Being used for new cloud-y apps • Multiple sites set up as independent regions (1 AZ/ region) • “Several dozen” compute nodes per site (VM capacity of “a few thousand”) • One site containing SwiftJuly 26th, 2012LA #openstack meetup 6
    14. 14. What Have We Done? • Private cloud deployment for BigCo (not a Web co) • ~10 distinct internal groups using • So exercises multi-tenant features • Being used for new cloud-y apps • Multiple sites set up as independent regions (1 AZ/ region) • “Several dozen” compute nodes per site (VM capacity of “a few thousand”) • One site containing Swift • Integrated into existing IT Ops processesJuly 26th, 2012LA #openstack meetup 6
    15. 15. Plan Your Plan • Which version of Openstack? • What hardware do you use? • What Operating System?July 26th, 2012LA #openstack meetup 7
    16. 16. What’s in a version? • Openstack versions: • Bexar (Feb 2011) • Cactus (Apr 2011) • Diablo-release (Sept 2011) • Diablo-stable (Oct 2011) (-final?) • Essex (Apr 2012) (you are here) • Folsom (Sept 2012)July 26th, 2012LA #openstack meetup 8
    17. 17. What’s in a version? • Openstack versions: • Bexar (Feb 2011) Danger • Cactus (Apr 2011) Will Robinson! • Diablo-release (Sept 2011) • Diablo-stable (Oct 2011) (-final?) • Essex (Apr 2012) (you are here) • Folsom (Sept 2012)July 26th, 2012LA #openstack meetup 8
    18. 18. What do you put it on? • “There are more x86 server platforms in heaven and earth than are dreamt of in your philosophy” - Shakespeare • At scale, you’ll have a huge number of compute nodes, a reasonable number of storage nodes, and a small number of other nodes - so compute node platform generally dominates the decisionJuly 26th, 2012LA #openstack meetup 9
    19. 19. Server Hardware • Compute nodes • Lots of RAM (64G+) • Enough CPU (cost/benefit) • Enough disk (more spindles = better) • Model the workloads you want to offer to find the right hardware • Storage nodes (object servers, not proxies) • Lots of spindles • Enough space • Server memory/CPU not as criticalJuly 26th, 2012LA #openstack meetup 10
    20. 20. One Operating System to Rule Them All • We use Ubuntu • The vast majority of Openstack development is done on Ubuntu • Canonical are big-time bought into Openstack • Other OSes probably work, YMMVJuly 26th, 2012LA #openstack meetup 11
    21. 21. The Great Hypervisor DebateJuly 26th, 2012LA #openstack meetup 12
    22. 22. The Great Hypervisor Debate • We use KVM • “And that’s all I have to say about that.” - Forrest GumpJuly 26th, 2012LA #openstack meetup 12
    23. 23. Building your Build • Bootstrapping • Network configuration • Openstack configurationJuly 26th, 2012LA #openstack meetup 13
    24. 24. BootstrappingJuly 26th, 2012LA #openstack meetup 14
    25. 25. Bootstrapping • Openstack does need a running OS to be installed on • Fairy dust not on roadmap until 2014July 26th, 2012LA #openstack meetup 14
    26. 26. Bootstrapping • Openstack does need a running OS to be installed on • Fairy dust not on roadmap until 2014 • Options • Bootstrap all your nodes off an ISO yourself • Bootstrap all your nodes off a PXEboot server yourself • Cobbler, Juju, MAAS (Canonical) • Crowbar (Dell) • Open Cloud OS (Cloudscaling) • PentOS (Piston)July 26th, 2012LA #openstack meetup 14
    27. 27. Bootstrapping • Openstack does need a running OS to be installed on • Fairy dust not on roadmap until 2014 • Options • Bootstrap all your nodes off an ISO yourself • Bootstrap all your nodes off a PXEboot server yourself • Cobbler, Juju, MAAS (Canonical) • Crowbar (Dell) • Open Cloud OS (Cloudscaling) • PentOS (Piston) • (don’t forget to burnin your hardware!)July 26th, 2012LA #openstack meetup 14
    28. 28. The NetworkJuly 26th, 2012LA #openstack meetup 15
    29. 29. The Network • Typical setups use 3-4 distinct networks • Hardware management - IPMI, bare-metal mgmt • Node management - PXE, installation, configuration • External - Ingress/Egress traffic • Internal - Admin, Compute->Storage, Compute->Compute • Internal/External can be high bandwidth • Hardware/node management are low-bandwidthJuly 26th, 2012LA #openstack meetup 15
    30. 30. The Network • Typical setups use 3-4 distinct networks • Hardware management - IPMI, bare-metal mgmt • Node management - PXE, installation, configuration • External - Ingress/Egress traffic • Internal - Admin, Compute->Storage, Compute->Compute • Internal/External can be high bandwidth • Hardware/node management are low-bandwidth • Nova inserts a weird network traffic flow model into your pretty network architecture diagrams (central node providing NAT and DHCP for all VMs) • Creates SPOF leading to ugly failover implementations (for varying definitions of ugly) • Creates aggregation point for trafficJuly 26th, 2012LA #openstack meetup 15
    31. 31. Tomato, Tomato • You have choices wrt configuration when you deploy Openstack • APIs • S3 vs Swift • EC2 vs Nova • Queuing models (RabbitMQ vs 0MQ) • Network models • nova-volume (EBS) vs DASJuly 26th, 2012LA #openstack meetup 16
    32. 32. Tomato, Tomato • You have choices wrt configuration when you deploy Openstack • APIs • S3 vs Swift • EC2 vs Nova • Queuing models (RabbitMQ vs 0MQ) • Network models • nova-volume (EBS) vs DASJuly 26th, 2012LA #openstack meetup 16
    33. 33. Have Cloud, Will Operate • Now that you’ve got it, it’s got to runJuly 26th, 2012LA #openstack meetup 17
    34. 34. You Bought It, You Break ItJuly 26th, 2012LA #openstack meetup 18
    35. 35. You Bought It, You Break It • Configuration Management • Capacity Management • User Management • Remote Access • High Availability • Upgrading • Troubleshooting • Monitoring • AuditingJuly 26th, 2012LA #openstack meetup 18
    36. 36. You Bought It, You Break It • Configuration Management • Capacity Management • User Management • Remote Access • High Availability • Upgrading • Troubleshooting • Monitoring • Auditing • Yes, you need to do these, and no, Openstack doesn’t do themJuly 26th, 2012LA #openstack meetup 18
    37. 37. What We Do • Configuration Management, User Management = chef • Capacity Management = metrics via collectd • Remote Access = dedicated/hardened SSH server, VPN • Troubleshooting = rsyslog centralized logging, collectd • Monitoring = Nagios, Zabbix • Auditing = tcpspy, auditdJuly 26th, 2012LA #openstack meetup 19
    38. 38. High Availability • For Nova, several services can be redundant just by running more than one instance, like the scheduler • nova-compute and swift-object-server redundancy at the node level (any node can fail without it being an emergency) - we use that for the services we build where possible • Replace RabbitMQ with 0MQ, nova single mySQL DB with mySQL cluster behind UCARP failover (similar for glance) • Some services need TCP loadbalancing, e.g. the nova-api and swift-proxy services • We do this with Quagga (ECMP OSPF) + Pound (also SSL) • nova-network is tricky - we are taking it out of the critical path of NAT and DHCP, replacing it with our own plug-in networking driver for OpenstackJuly 26th, 2012LA #openstack meetup 20
    39. 39. Patches/UpgradesJuly 26th, 2012LA #openstack meetup 21
    40. 40. Patches/Upgrades • just run apt-get upgradeJuly 26th, 2012LA #openstack meetup 21
    41. 41. Patches/Upgrades • just run apt-get upgrade • JUST KIDDINGJuly 26th, 2012LA #openstack meetup 21
    42. 42. Patches/Upgrades • just run apt-get upgrade • JUST KIDDING • Need to consider persistent data - mySQL DB and queues • Can use live-ish migration to take load off of compute servers and upgrade individually • As long as you have 3+ zones, you can just take swift nodes offline to upgrade individually • Central services require careful planning and testing • Hope for tolerance of multiple versions at once • So far we’ve been able to do everything without impacting VMs or objectsJuly 26th, 2012LA #openstack meetup 21
    43. 43. Patches/Upgrades • just run apt-get upgrade • JUST KIDDING • Need to consider persistent data - mySQL DB and queues • Can use live-ish migration to take load off of compute servers and upgrade individually • As long as you have 3+ zones, you can just take swift nodes offline to upgrade individually • Central services require careful planning and testing • Hope for tolerance of multiple versions at once • So far we’ve been able to do everything without impacting VMs or objects • ^^ That is not a guarantee!July 26th, 2012LA #openstack meetup 21
    44. 44. Who Ops your Ops? • You probably have an existing IT Ops org and processes that will interface with your cloud - if not run it directly • Cloud is a different paradigm, requiring different Ops perspectives and potentially skillsets • Recommendation • Build a dedicated Cloud Ops team • Build bridges from that team to existing teams from day oneJuly 26th, 2012LA #openstack meetup 22
    45. 45. SummaryJuly 26th, 2012LA #openstack meetup 23
    46. 46. Summary • You can build and run a private cloud on Openstack todayJuly 26th, 2012LA #openstack meetup 23
    47. 47. Summary • You can build and run a private cloud on Openstack today • It does require a fair bit of clue and experienceJuly 26th, 2012LA #openstack meetup 23
    48. 48. Summary • You can build and run a private cloud on Openstack today • It does require a fair bit of clue and experience • To be reliable and scalable, you have to add lots of stuff around the edges and configure it all the right wayJuly 26th, 2012LA #openstack meetup 23
    49. 49. Summary • You can build and run a private cloud on Openstack today • It does require a fair bit of clue and experience • To be reliable and scalable, you have to add lots of stuff around the edges and configure it all the right way Thank you! @pgutheb / paul at cloudscaling dot comJuly 26th, 2012LA #openstack meetup 23

    ×