A one stop solution
for Puppet and Openstack
Daniel Lobato Garcia
daniel.lobato.garcia@cern,ch
@eLobatoss
What is CERN
Between Geneva and the
Jura mountains,
straddling the Swiss-
French border
Mission: learn what is the
univers...
Fundamental
questions in
physics
Why do particles have mass?
What is 96% of the universe
made of?
Why isn’t there anti-mat...
8/12/2013 Document reference 5
8/12/2013 Document reference 6
8/12/2013 Document reference 7
8/12/2013 Document reference 8
Current status
• 270 Openstack hypervisors
• 2900 virtual machines
• 300 users
• 14 Puppet masters
• 6 Foreman backend nod...
Goals
• Ramp up to 15K hypervisors – 150-200K
vms in 2015
• Multi-site (Hungary)
10
8/12/2013 Document reference 11
8/12/2013 Document reference 12
Why?
• Unnecessary homebrew stack of tools
• Shift to cloud standards with minimal
customizations
• High turnover – can’t ...
Why?
• Symbiotic relationship with the community
14
Openstack?
• Modular IaaS free open source project
• APIs ~compatible with those of Amazon
15
Openstack Nova
(compute)
Cloud fabric controller
16
Openstack
Keystone (Identity)
RBAC
Integrated with LDAP
Multiple auth* methods
17
Openstack Glance
(Images)
Discovery, registration,
delivery of images
18
Openstack Horizon
(Dashboard)
19
Modules
• Puppet definitions for every use case you
can imagine.
• Dynamic environments
• Hadoop node
• Openstack hypervis...
21
Workflow..?
Modules and Git
• Manifests and hieradata are version
controlled
22
23
Git workflow
Puppet masters
24
Easy cherry pick
25
Git workflow
26
Git workflow
Jens
‘Puppetfiles’
Separate repositories
Makes environments and
creates them on the masters
Foreman
• Lifecycle management tool for VMs and
physical servers
• External Node Classifier – tells the puppet
master what...
28
29
Power operations & Foreman
8/12/2013 Document reference 30
Foreman Proxy
Physical
boxIPMI
Physical
box
IPMI
Physical
box
I...
Openstack VM creation
8/12/2013 Document reference 31
Openstack VM creation
8/12/2013 Document reference 32
Openstack VM creation
8/12/2013 Document reference 33
Scalability experiences
• Split up services
• Puppet – critical vs non critical
34
12 backend nodes
Batch
4 backend nodes
...
Scalability experiences
• Foreman – split into different services
35
ENC
Reports
processing
UI/API
Load balancer
9443 – UI...
Scalability experiences
• Autoscale via alarms (Heat)
• Define situations (i.e: load threshold..)
• Spin up VMs as needed
...
Scalability guidelines
37
github.com
/
cernops
38
39
Secrets provisioning (naïve)
• Use case: provision a db password
41
Secrets provisioning (hiera-gpg)
• Use case: provision a db password
42
Secrets provisioning (hack)
• Use case: provision a db password
43
Secrets provisioning
•Masters
need not
read secrets
44
A One-Stop Solution for Puppet and OpenStack
A One-Stop Solution for Puppet and OpenStack
Upcoming SlideShare
Loading in …5
×

A One-Stop Solution for Puppet and OpenStack

7,512 views

Published on

Throughout the last year, we have been using and developing tools that allow us to have an IaaS where our data center is configured by Puppet and our virtualization and authentication needs are catered by Openstack. RedHat's foreman is our lifecycle management tool which we configured to support both bare metal and Openstack virtual machines. We use git to manage environments and hostgroup configurations and we will tell you how we deal with its security implications, how to store Hieradata secrets. Switching from a homebrew toolchain to open source tools like Facter, Foreman, Openstack has turned out into many contributions to these teams. Nearly everyone at CERN has started to wear the devops hat which brings new challenges in terms of development workflows and scalability.

Daniel Lobato Garcia
Software Engineer, CERN
Daniel Lobato is a developer who has worked in very different environmentst, from data centers and mainframes to startups. Nowadays he has dived into the Agile Infrastructure team at CERN where the design and implementation of the new computing infrastructure is done. As for Puppet, he currently helps RedHat to develop Foreman, a lifecycle management tool for physical and virtual machines. One of his goals at CERN is to knot this tool to all the relevant parts of the infrastructure, which includes Puppet for configuration management, OpenStack for virtualization and authentication, Puppetdb and others. He is sure the source of all computer problems is between the chair and the keyboard.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
7,512
On SlideShare
0
From Embeds
0
Number of Embeds
4,430
Actions
Shares
0
Downloads
108
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

A One-Stop Solution for Puppet and OpenStack

  1. 1. A one stop solution for Puppet and Openstack Daniel Lobato Garcia daniel.lobato.garcia@cern,ch @eLobatoss
  2. 2. What is CERN Between Geneva and the Jura mountains, straddling the Swiss- French border Mission: learn what is the universe made of and how does it work? 3
  3. 3. Fundamental questions in physics Why do particles have mass? What is 96% of the universe made of? Why isn’t there anti-matter in the universe? What was the state of matter after the Big Bang? 4
  4. 4. 8/12/2013 Document reference 5
  5. 5. 8/12/2013 Document reference 6
  6. 6. 8/12/2013 Document reference 7
  7. 7. 8/12/2013 Document reference 8
  8. 8. Current status • 270 Openstack hypervisors • 2900 virtual machines • 300 users • 14 Puppet masters • 6 Foreman backend nodes • Some production services migrating to our cloud – early birds 9
  9. 9. Goals • Ramp up to 15K hypervisors – 150-200K vms in 2015 • Multi-site (Hungary) 10
  10. 10. 8/12/2013 Document reference 11
  11. 11. 8/12/2013 Document reference 12
  12. 12. Why? • Unnecessary homebrew stack of tools • Shift to cloud standards with minimal customizations • High turnover – can’t teach new tools 13
  13. 13. Why? • Symbiotic relationship with the community 14
  14. 14. Openstack? • Modular IaaS free open source project • APIs ~compatible with those of Amazon 15
  15. 15. Openstack Nova (compute) Cloud fabric controller 16
  16. 16. Openstack Keystone (Identity) RBAC Integrated with LDAP Multiple auth* methods 17
  17. 17. Openstack Glance (Images) Discovery, registration, delivery of images 18
  18. 18. Openstack Horizon (Dashboard) 19
  19. 19. Modules • Puppet definitions for every use case you can imagine. • Dynamic environments • Hadoop node • Openstack hypervisor • … you name it 20
  20. 20. 21 Workflow..?
  21. 21. Modules and Git • Manifests and hieradata are version controlled 22
  22. 22. 23 Git workflow Puppet masters
  23. 23. 24 Easy cherry pick
  24. 24. 25 Git workflow
  25. 25. 26 Git workflow Jens ‘Puppetfiles’ Separate repositories Makes environments and creates them on the masters
  26. 26. Foreman • Lifecycle management tool for VMs and physical servers • External Node Classifier – tells the puppet master what a node should look like 27
  27. 27. 28
  28. 28. 29
  29. 29. Power operations & Foreman 8/12/2013 Document reference 30 Foreman Proxy Physical boxIPMI Physical box IPMI Physical box IPMI VMVMVM Openstack Nova API
  30. 30. Openstack VM creation 8/12/2013 Document reference 31
  31. 31. Openstack VM creation 8/12/2013 Document reference 32
  32. 32. Openstack VM creation 8/12/2013 Document reference 33
  33. 33. Scalability experiences • Split up services • Puppet – critical vs non critical 34 12 backend nodes Batch 4 backend nodes Interactive
  34. 34. Scalability experiences • Foreman – split into different services 35 ENC Reports processing UI/API Load balancer 9443 – UI/API 9444 – Reports 9445 – ENC …
  35. 35. Scalability experiences • Autoscale via alarms (Heat) • Define situations (i.e: load threshold..) • Spin up VMs as needed 36
  36. 36. Scalability guidelines 37
  37. 37. github.com / cernops 38
  38. 38. 39
  39. 39. Secrets provisioning (naïve) • Use case: provision a db password 41
  40. 40. Secrets provisioning (hiera-gpg) • Use case: provision a db password 42
  41. 41. Secrets provisioning (hack) • Use case: provision a db password 43
  42. 42. Secrets provisioning •Masters need not read secrets 44

×