How to Puppetize Google Cloud Platform - Katharina Probst, Google, Matt Bookman, Google & Ryan Coleman, Puppet Labs

1. HOWTO:Puppetizing Google Cloud Platform
Katharina Probst, Technical Lead & Manager (Google)
Ryan Coleman, Product Manager (Puppet Labs)

2. Agenda: September 23rd, 2014
Whirlwind tour of Google Cloud Platform
Diving into Google Compute Engine
How Puppet Labs uses Google Cloud Platform
Live Demo
1
2
3
4

3. Agenda: September 23rd, 2014
Whirlwind tour of Google Cloud Platform
Diving into Google Compute Engine
How Puppet Labs uses Google Cloud Platform
Live Demo
1
2
3
4

4. For the past 15 years, Google
has been building out the
world’s fastest, most powerful,
highest quality cloud
infrastructure on the planet.
Images by Connie
Zhou
Why Google Cloud Platform?

5. Building Products that Scale
Google Maps Gmail Google Drive YouTube

6. A Network that Spans the Globe

7. Innovating Software & Driving Technology Forward
MapReduce Dremel Spanner
Big Table Colossus
GFS
Compute
Engine
2002 2004 2006 2008 2010 2012 2013

8. “[Google's] ability to build, organize, and
operate a huge network of servers and fiber-optic
cables with an efficiency and speed that
rocks physics on its heels.
This is what makes Google Google: its
physical network, its thousands of fiber miles,
and those many thousands of servers that, in
aggregate, add up to the mother of all
clouds.”
- Wired
Images by Connie
Zhou

9. Storage
Cloud Storage Cloud SQL Cloud
Datastore
Compute
Compute
App Engine Engine
App Services
BigQuery Cloud
Endpoints
Google Cloud Platform
Cloud DNS

10. Compute
Compute
App Engine Engine
Storage
Cloud Storage Cloud SQL Cloud
Datastore
Power Computation
App Services
BigQuery Cloud
Endpoints
Cloud DNS

11. Storing What You Want, However You Want
Storage
Cloud Storage Cloud SQL Cloud
Datastore
Compute
Compute
App Engine Engine
App Services
BigQuery Cloud
Endpoints
Cloud DNS

12. App Services
BigQuery Cloud
Endpoints
Building Robust, Intelligent Systems
Storage
Cloud Storage Cloud SQL Cloud
Datastore
Compute
Compute
App Engine Engine
Cloud DNS

13. Agenda: September 23rd, 2014
Whirlwind tour of Google Cloud Platform
Diving into Google Compute Engine
How Puppet Labs uses Google Cloud Platform
Live Demo
1
2
3
4

14. Google Compute Engine
• IaaS: VMs, Network, Storage
• Google DNA (speed, scale, reliable, secure)
• Fast Provisioning, Consistent Performance
• Enterprise Ready
• 24x7 Support
• 99.95% monthly SLA
• ISO 27001, SSAE-16 SOC 1,2,3
• Accessible Through
• Web @ https://cloud.google.com/console
• gcloud compute command-line utility
• REST API
• Partners (Commercial and FOSS)

15. Launching 50 virtual machines

16. Block Storage
Persistent Disks

17. Persistent Disk
10 TB
• SSD and "Standard"
• Data and root partitions
• Billed only by capacity (GB/month)
• Performance caps scale linearly with size
• Volume striping is automatic
• Differential snapshots
• Create new PDs based on snapshots

18. PD Use-cases
Root
Stateful root
volume
Root RW
Data
User managed data
volume
RO
Data
Instant distribution of
static content

19. Global Snapshot and Restore
vol1 vol2 vol3
GCS
vol1.
t2
vol2.
t2
vol3.
t3
t1 t1 t1
t1 t1 t1
t1 t1 t1
t1 t1
t1 t1
t2 t2
t2
t2
t3 t3 t3
● Point in time snapshot to Google
Cloud Storage (GCS)
● Differential snapshots
● GCS global replication!
● Restore from snapshot
anywhere in the world
us-central1-a
europe-west1-a

20. Virtual Machines

21. Machine Types Machine Type Virtual Cores
standard
• For workloads with balanced CPU and memory
highmem
• For workloads with higher memory requirements
highcpu
• For workloads with higher CPU requirements
Shared Core Instances
• For inexpensive prototyping and staging
workloads (g1-small, f1-micro)
Memory
(GB)
n1-standard-1 1 3.75
n1-standard-2 2 7.50
n1-standard-4 4 15.00
n1-standard-8 8 30.00
n1-standard-16 16 60.00
n1-highmem-2 2 13.00
n1-highmem-4 4 26.00
n1-highmem-8 8 52.00
n1-highmem-16 16 104.00
n1-highcpu-2 2 1.80
n1-highcpu-4 4 3.60
n1-highcpu-8 8 7.20
n1-highcpu-16 16 14.40

22. Images
+ Custom Kernel Support
(debian backports works great for docker)

23. Sub-hour Billing + Sustained Use Discount
1 minute granularity, 10 minute minimum
$
(30%)
Full billing cycle
Sustained Use Discount
● >25% use, price reduced
● Up to 30% reduction
● No commitment / contract

24. Enhanced Reliability
us-central1-a
scheduled maintenance event
Live Migration
• No downtime during scheduled
datacenter maintenance events
Automatic Restart
• Instances automatically
restarted if subjected to system
events such as hardware failure

25. Networking

26. Images by Connie
Zhou
Networking
• Google's Network
• Projects are isolated private networks
• TCP, UDP, ICMP only
• Multiple private network groups and firewalls
• Tags and address ranges
• Addresses
• public: static or ephemeral
• private: ephemeral with DNS
• Routes, gateways, VPNs, and IP Forwarding

27. Region: us-central1
Target Pool (tp-a)
Load-balancer
• Region based
• Session affinity, hashing options on tuples,
• Protocol, src ip:port, dst ip:port
• Consists of:
us-central1-a
www0
www1
tcp:80 ➔ tp-a
• Target pool: collection of instances
• Forwarding rules: protocol:port, pool
• HTTP health check: optional us-central1-b
www2
www3
Forwarding Rules
tcp:443 ➔ tp-a
Internet
http://googlecloudplatform.blogspot.com/2013/11/compute-engine-load-balancing-hits-1-million-requests-per-second.html

28. ...and

29. Noteworthy
• Regions
• Central United States, Europe, Asia (new!)
• Metadata
• Startup scripts
• SSH Keys
• Tags (instance/network)
• OAuth2 and Scopes
• Access other Google Cloud Platform services
• Ecosystem is growing
• Partners: RightScale, Scalr, New Relic, MongoLab, MapR, and many more...
• Open Source: Chef, Puppet, Salt, Ansible, Vagrant, Docker, CoreOS, fog, libcloud

30. Agenda: September 23rd, 2014
Whirlwind tour of Google Cloud Platform
Diving into Google Compute Engine
How Puppet Labs uses Google Cloud Platform
Live Demo
1
2
3
4

31. Puppet DSL for Google Compute Engine
https://forge.puppetlabs.com/puppetlabs/gce_compute

33. The Puppet Forge
a real world (meta) example

38. Abstraction Layers for Infrastructure as Code
Infrastructure Layer
Role & Profile Layer
Component Layer
Primitive Layer
puppetlabs-gce_compute
custom module
puppetlabs-postgresql
core puppet type/provider

39. Agenda: September 23rd, 2014
Whirlwind tour of Google Cloud Platform
Diving into Google Compute Engine
How Puppet Labs uses Google Cloud Platform
Live Demo
1
2
3
4

40. Demo Architecture
1 3
2 4
us-central1-a
M
us-central1-b
allow tcp:80
Load-balancer
Target-pool
● [M]aster resides in GCE
● Uses gce_compute manifest to:
○ Spin up 4 VMs
○ Networking (firewall, LB)
● New nodes register with Master
● Install site.pp (apache+php+mysql)
● Custom index.php, connects to
Cloud SQL database
● Let's try it out!
Cloud SQL

41. Bring up the Compute Engine environment
puppet apply /etc/puppetlabs/puppet/manifests/gce_up.pp
# puppet apply /etc/puppetlabs/puppet/manifests/gce_up.pp
# ...
# cat !$
1 $prefix = 'pc'
2 $zonea = 'us-central1-a'
3 $zoneb = 'us-central1-b'
4 $region = 'us-central1'
5
6 gce_auth { "graphite-demos": }
7

42. Bring up the Compute Engine environment (cont)
puppet apply /etc/puppetlabs/puppet/manifests/gce_up.pp
8 gce_firewall { "$prefix-firewall":
9 ensure => present,
10 description => 'Allow HTTP',
11 network => 'default',
12 allowed => 'tcp:80',
13 allowed_ip_sources => '0.0.0.0/0',
14 }
15
16 gce_httphealthcheck { "$prefix-http":
17 ensure => present,
18 require => Gce_instance["$prefix-agent-1", "$prefix-agent-2"...],
19 description => 'basic http health check',
20 }
21 ...

43. Bring up the Compute Engine environment (cont)
puppet apply /etc/puppetlabs/puppet/manifests/gce_up.pp
46 gce_disk { [ "$prefix-agent-2", "$prefix-agent-4" ]:
47 ensure => present,
48 size_gb => 10,
49 zone => "$zoneb",
50 source_image => 'debian-7',
51 }
52 ...

44. Bring up the Compute Engine environment (cont)
puppet apply /etc/puppetlabs/puppet/manifests/gce_up.pp
73 define create_instance_zoneb {
74 gce_instance { "${title}":
75 ensure => present,
76 zone => "$zoneb",
77 disks => "${title},boot",
78 require => Gce_disk["${title}"],
79 puppet_service => present,
80 startupscript => 'pe-simplified-agent.sh',
81 metadata => {
82 'pe_role' => 'agent',
83 'pe_master' => "$hostname",
84 'pe_version' => '3.3.2',
85 }
86 }
87 }
88
89 create_instance_zoneb { ["$prefix-agent-2", "$prefix-agent-4"]: }

45. The Demo App
puppet apply /etc/puppetlabs/puppet/manifests/gce_up.pp
1 <html>
2 <title>PuppetConf 2014</title>
3 <body>
4 <h1>PuppetConf 2014: Puppetizing Google Cloud Platform</h1>
5 ...
6 <!-- A couple 'facter' facts -->
7 <ul>This app is being served from host: <b><%= @ hostname %> [<%=
@fqdn %>, 7 <%= @ipaddress %>]</b></ul>
8 <p>Please enter a Nick and Message and watch it appear below:</p>
9 ...
10 <?php
11 $db = mysqli_connect("173.194.254.93", "puppetconf", " my_password",
"puppetconf");
12 ...
13 ?>

46. cloud.google.com
• Try out Google Cloud Platform with a $500 Credit
• Visit https://cloud.google.com/starterpack and use promo code puppet14-con
• Want more?
• Try the demo on your own at https://bit.ly/puppet-gce-demo
• or see http://googlecloudplatform.github.io/compute-video-demo-puppet
• We welcome your help and feedback to make Puppet + GCE even better!
• Find us at gce-discussion@googlegroups.com or
https://github.com/puppetlabs/puppetlabs-gce_compute

47. FIN