OSCON 2010 Cloud Summit presentation. How to to operate in a cloudy world.

1. John Willis
VP of Services
john@opscode.com
twitter.com/botchagalupe
Copyright © 2010 Opscode, Inc - All Rights Reserved 1
Thursday, July 29, 2010

2. IT Management Podcast
DevopsCafe
CloudCafe
Copyright © 2010 Opscode, Inc - All Rights Reserved 2
Thursday, July 29, 2010

3. IT Management Podcast
DevopsCafe
CloudCafe
Copyright © 2010 Opscode, Inc - All Rights Reserved 2
Thursday, July 29, 2010

4. Private Public
SaaS
PaaS PaaS
IaaS IaaS
Managed
Virtualization
hosting
Slide courtesy Alistair Croll - alistair@rednod.com
Thursday, July 29, 2010

5. Private Public
nt t o SaaS
a
w d s,
o u lo u
y c
If l k
PaaS
rs t.PaaS
t a ne f i
i ck o
IaaS p IaaS
Managed
Virtualization
hosting
Slide courtesy Alistair Croll - alistair@rednod.com
Thursday, July 29, 2010

6. Infrastructure as a Service
(IaaS)
Amazon EC2, Rackspace Cloud, Terremark,
Gogrid, Joyent (and nearly every private
cloud built on Zenserver or VMWare.)
Slide courtesy Alistair Croll - alistair@rednod.com
Thursday, July 29, 2010

7. Cloudy
Operations
Thursday, July 29, 2010

8. Pixie Dust!
Copyright © 2010 Opscode, Inc - All Rights Reserved 6
Thursday, July 29, 2010

9. Did They Lie?
Copyright © 2010 Opscode, Inc - All Rights Reserved 7
Thursday, July 29, 2010

10. Did They Lie?
I did not have “cloudy”
relations with that provider
Copyright © 2010 Opscode, Inc - All Rights Reserved 7
Thursday, July 29, 2010

11. Infrastructure is Hard!
Copyright © 2010 Opscode, Inc - All Rights Reserved 8
Thursday, July 29, 2010

12. Fully
Automated Infrastructure
Control Provisioning Release
Orchestration Deploy CI Server Issue tracker
Artifact
Config repository
Dispatcher SCM
management Build
Repository
OS boot/
Scheduler install
Code Workflows
Model Resources Configuration
Events Trending
Asset
Identity
inventory
Trending Reporting Topology
Host naming CMDB
Monitoring Sources
Copyright © 2010 Opscode, Inc - All Rights Reserved 9
Thursday, July 29, 2010

13. Fully
Automated Infrastructure
Control Provisioning Release
Orchestration Deploy CI Server Issue tracker
Artifact
Config repository
Dispatcher SCM
management Build
Repository
OS boot/
Scheduler install
Code Workflows
Model Resources Configuration
Events Trending
Asset
Identity
inventory
Trending Reporting Topology
Host naming CMDB
Monitoring Sources
Copyright © 2010 Opscode, Inc - All Rights Reserved 9
Thursday, July 29, 2010

14. Network Operations
Systems Administrators
Software Developers
Database Administrators
Storage Management
Project Management
Change Management
Continuity Planning
Risk Management
Web Design
Performance
Compliance
( )
Architecture * in no
Tooling particular
Testing order
Security
Reporting
Facilities
Thursday, July 29, 2010

15. Thursday, July 29, 2010
SNAFU
Copyright © 2010 Opscode, Inc - All Rights Reserved 11

16. Cloudy Monitoring
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

17. Cloudy Monitoring
Performance
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

18. Cloudy Monitoring
Performance
Log
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

19. Cloudy Monitoring
Performance
Log
Alerts
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

20. Cloudy Monitoring
Performance
Log
Alerts
Event
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

21. Cloudy Monitoring
Performance
Log
Alerts
Event
Correlation
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

22. Cloudy Monitoring
Performance
Log
Alerts
Event
Correlation
Capacity
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

23. Cloudy Monitoring
Performance
Log
Alerts
Event
Correlation
Capacity
Analytics
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

24. Cloudy Monitoring
Nagios
Performance
Log
Alerts
Event
Correlation
Capacity
Analytics
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

25. Cloudy Monitoring
Nagios
Performance
Collectd
Log
Alerts
Event
Correlation
Capacity
Analytics
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

26. Cloudy Monitoring
Nagios
Performance
Collectd
Log
jcollectd
Alerts
Event
Correlation
Capacity
Analytics
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

27. Cloudy Monitoring
Nagios
Performance
Collectd
Log
jcollectd
Alerts
Ganglia
Event
Correlation
Capacity
Analytics
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

28. Cloudy Monitoring
Nagios
Performance
Collectd
Log
jcollectd
Alerts
Ganglia
Event
Zenoss
Correlation
Capacity
Analytics
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

29. Cloudy Monitoring
Nagios
Performance
Collectd
Log
jcollectd
Alerts
Ganglia
Event
Zenoss
Correlation
JMX
Capacity
Analytics
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

30. Cloudy Monitoring
Nagios
Performance
Collectd
Log
jcollectd
Alerts
Ganglia
Event
Zenoss
Correlation
JMX
Capacity
OpenNMS
Analytics
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

31. Cloudy Monitoring
Nagios
Performance
Collectd
Log
jcollectd
Alerts
Ganglia
Event
Zenoss
Correlation
JMX
Capacity
OpenNMS
Analytics
Munin
Copyright © 2010 Opscode, Inc - All Rights Reserved 12
Thursday, July 29, 2010

32. Cloudy Provisioning
Copyright © 2010 Opscode, Inc - All Rights Reserved 13
Thursday, July 29, 2010

33. Cloudy Provisioning
Provisioning
Copyright © 2010 Opscode, Inc - All Rights Reserved 13
Thursday, July 29, 2010

34. Cloudy Provisioning
Provisioning
Configuration
Copyright © 2010 Opscode, Inc - All Rights Reserved 13
Thursday, July 29, 2010

35. Cloudy Provisioning
Provisioning
Configuration
Systems Integration
Copyright © 2010 Opscode, Inc - All Rights Reserved 13
Thursday, July 29, 2010

36. Provisioning
Nodes
opslb01
opsws01
opsws02
opsdm01
opsds01
opsds02 Copyright © 2010 Opscode, Inc - All Rights Reserved 14
Thursday, July 29, 2010

37. Configuration Management
Roles
loadbalancer
webserver
dbmaster
dbslave
Copyright © 2010 Opscode, Inc - All Rights Reserved 15
Thursday, July 29, 2010

38. Systems Integration
Load Balancer
Recipes
haproxy
Web Server Web Server
apache2
myssql DB Master
DB Slave DB Slave
Disk Disk Disk
Copyright © 2010 Opscode, Inc - All Rights Reserved 16
Thursday, July 29, 2010

39. Role Based Configuration
name "webserver"
description "Systems that serve HTTP traffic"
run_list(
"role[base]",
"recipe[apache2]",
"recipe[apache2::mod_ssl]"
)
default_attributes(
"apache" => {
"listen_ports" => [ "80", "443" ]
}
)
override_attributes(
"apache" => {
"max_children" => "50"
}
)
Copyright © 2010 Opscode, Inc - All Rights Reserved 17
Thursday, July 29, 2010

40. Load Balancer Example
Copyright © 2010 Opscode, Inc - All Rights Reserved 18
Thursday, July 29, 2010

41. Systems Integration
Load Balancer
Web Server Web Server
DB Master
DB Slave DB Slave
Disk Disk Disk
Copyright © 2010 Opscode, Inc - All Rights Reserved 19
Thursday, July 29, 2010

42. Devops
Copyright © 2010 Opscode, Inc - All Rights Reserved 20
Thursday, July 29, 2010

43. Devops
• Culture
Copyright © 2010 Opscode, Inc - All Rights Reserved 20
Thursday, July 29, 2010

44. Devops
•Culture
• Automation
Copyright © 2010 Opscode, Inc - All Rights Reserved 20
Thursday, July 29, 2010

45. Devops
•Culture
• Automation
• Measurement
Copyright © 2010 Opscode, Inc - All Rights Reserved 20
Thursday, July 29, 2010

46. Devops
•Culture
• Automation
• Measurement
• Sharing
Copyright © 2010 Opscode, Inc - All Rights Reserved 20
Thursday, July 29, 2010

47. What Do
Developers
Need?
Copyright © 2010 Opscode, Inc - All Rights Reserved 21
Thursday, July 29, 2010

48. For Developers...
Thursday, July 29, 2010

49. For Developers...
• Self Service Operations
Thursday, July 29, 2010

50. For Developers...
• Self Service Operations
• The infrastructure is the application
(and vice versa)
Thursday, July 29, 2010

51. For Developers...
• Self Service Operations
• The infrastructure is the application
(and vice versa)
• Minimize Bottlenecks
Thursday, July 29, 2010

52. For Developers...
• Self Service Operations
• The infrastructure is the application
(and vice versa)
• Minimize Bottlenecks
• The “Right” Tools
Thursday, July 29, 2010

53. What Does
Operations Need?
Copyright © 2010 Opscode, Inc - All Rights Reserved 23
Thursday, July 29, 2010

54. Copyright © 2010 Opscode, Inc - All Rights Reserved 24
Thursday, July 29, 2010

55. Operations
http://covers.oreilly.com/images/9780596007836/lrg.jpg
Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft
Thursday, July 29, 2010

56. Operations
• Say “Yes”.
http://covers.oreilly.com/images/9780596007836/lrg.jpg
Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft
Thursday, July 29, 2010

57. Operations
• Say “Yes”.
• You never liked rack
and stack that much
anyway.
http://covers.oreilly.com/images/9780596007836/lrg.jpg
Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft
Thursday, July 29, 2010

58. Operations
• Say “Yes”.
• You never liked rack
and stack that much
anyway.
• You have never
been more critical.
http://covers.oreilly.com/images/9780596007836/lrg.jpg
Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft
Thursday, July 29, 2010

59. Operations
• Say “Yes”.
• You never liked rack
and stack that much
anyway.
• You have never
been more critical.
• Just get out of the
way.
http://covers.oreilly.com/images/9780596007836/lrg.jpg
Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft
Thursday, July 29, 2010

60. Agile Infrastructure
Development
Team focus
IDE/Workbench
Agile methodology
Source Control
Copyright © 2010 Opscode, Inc - All Rights Reserved 26
Thursday, July 29, 2010

61. Agile Infrastructure
Operations
Individual focus
Script VI based
Source control?
Waterfall
Copyright © 2010 Opscode, Inc - All Rights Reserved 27
Thursday, July 29, 2010

62. Infrastructure as Code
http://www.flickr.com/photos/wonderlane/2306082998/
Copyright © 2010 Opscode, Inc - All Rights Reserved 28
Thursday, July 29, 2010

63. Infrastructure as Code is...
http://www.flickr.com/photos/kwerfeldein/2634561264/sizes/o/
Copyright © 2010 Opscode, Inc - All Rights Reserved 29
Thursday, July 29, 2010

64. Infrastructure as Code is...
A technical
domain revolving
around building
and managing
infrastructure
programmatically
http://www.flickr.com/photos/kwerfeldein/2634561264/sizes/o/
Copyright © 2010 Opscode, Inc - All Rights Reserved 29
Thursday, July 29, 2010

65. Enable the reconstruction of
the business from nothing
but a source code
repository, an application
data backup, and bare metal
resources.
Copyright © 2010 Opscode, Inc - All Rights Reserved 30
Thursday, July 29, 2010

66. A Tornado Hits Your Data
http://www.flickr.com/photos/gi/518613153/sizes/o/
Thursday, July 29, 2010

67. A Tornado Hits Your Data
• Pause your movie
http://www.flickr.com/photos/gi/518613153/sizes/o/
Thursday, July 29, 2010

68. A Tornado Hits Your Data
• Pause your movie
• Sign into your cloud provider
http://www.flickr.com/photos/gi/518613153/sizes/o/
Thursday, July 29, 2010

69. A Tornado Hits Your Data
• Pause your movie
• Sign into your cloud provider
• Upload your offsite backups
http://www.flickr.com/photos/gi/518613153/sizes/o/
Thursday, July 29, 2010

70. A Tornado Hits Your Data
• Pause your movie
• Sign into your cloud provider
• Upload your offsite backups
• Provision, config and integrate the new servers
http://www.flickr.com/photos/gi/518613153/sizes/o/
Thursday, July 29, 2010

71. A Tornado Hits Your Data
• Pause your movie
• Sign into your cloud provider
• Upload your offsite backups
• Provision, config and integrate the new servers
• Change DNS to point to “Hit by Tornado” page
http://www.flickr.com/photos/gi/518613153/sizes/o/
Thursday, July 29, 2010

72. A Tornado Hits Your Data
• Pause your movie
• Sign into your cloud provider
• Upload your offsite backups
• Provision, config and integrate the new servers
• Change DNS to point to “Hit by Tornado” page
• Restore the customer and application data
http://www.flickr.com/photos/gi/518613153/sizes/o/
Thursday, July 29, 2010

73. A Tornado Hits Your Data
• Pause your movie
• Sign into your cloud provider
• Upload your offsite backups
• Provision, config and integrate the new servers
• Change DNS to point to “Hit by Tornado” page
• Restore the customer and application data
• Remove the “Hit by Tornado” page
http://www.flickr.com/photos/gi/518613153/sizes/o/
Thursday, July 29, 2010

74. A Tornado Hits Your Data
• Pause your movie
• Sign into your cloud provider
• Upload your offsite backups
• Provision, config and integrate the new servers
• Change DNS to point to “Hit by Tornado” page
• Restore the customer and application data
• Remove the “Hit by Tornado” page
• Unpause movie http://www.flickr.com/photos/gi/518613153/sizes/o/
Thursday, July 29, 2010

75. A Tornado Hits Your Data
• Pause your movie
• Sign into your cloud provider
• Upload your offsite backups
• Provision, config and integrate the new servers
• Change DNS to point to “Hit by Tornado” page
• Restore the customer and application data
• Remove the “Hit by Tornado” page
• Unpause movie http://www.flickr.com/photos/gi/518613153/sizes/o/
Thursday, July 29, 2010

76. A Tornado Hits Your Data
• Pause your movie
• Chapter 5
Sign into your cloud provider
Infrastructure
• Upload your offsite backups
as Code
• Provision, config and Jacob the new servers
Adam integrate
• Change DNS to point to “Hit by Tornado” page
• Restore the customer and application data
• Remove the “Hit by Tornado” page
• Unpause movie http://www.flickr.com/photos/gi/518613153/sizes/o/
Thursday, July 29, 2010

77. In a Cloudy World
Your Prime Constraint Should Be
http://www.flickr.com/photos/visualage/2126833132/sizes/o/
Copyright © 2010 Opscode, Inc - All Rights Reserved 32
Thursday, July 29, 2010

78. In a Cloudy World
Your Prime Constraint Should Be
The time it
takes to
restore your
application
data
http://www.flickr.com/photos/visualage/2126833132/sizes/o/
Copyright © 2010 Opscode, Inc - All Rights Reserved 32
Thursday, July 29, 2010

79. Infrastructure as Code
Copyright © 2010 Opscode, Inc - All Rights Reserved 33
Thursday, July 29, 2010

80. Recipies
Applies resources in the order they are specified
http://www.flickr.com/photos/roadsidepictures/2478953342/sizes/o/
Thursday, July 29, 2010

81. Recipies
Applies resources in the order they are specified
• Can include other
recipes.
http://www.flickr.com/photos/roadsidepictures/2478953342/sizes/o/
Thursday, July 29, 2010

82. Recipies
Applies resources in the order they are specified
• Can include other
recipes.
• A DSL like Ruby.
http://www.flickr.com/photos/roadsidepictures/2478953342/sizes/o/
Thursday, July 29, 2010

83. Tale of Two Startups
“Traditional” Operations Operations - The “Secret Sauce”
50 50
40 40
# of Hours
30 30
20 20
Hardware
OS Install
10 10 Config
Upkeep
20 20
15 15
Servers
10 10 New
5 5 Existing
0 0
1 2 3 4 5 6 7 9 10 11 12 1 2 3 4 5 6 7 9 10 11 12
Week # Week #
(http://radar.oreilly.com/archives/2007/10/operations-advantage.html)
Copyright © 2010 Opscode, Inc - All Rights Reserved 35
Thursday, July 29, 2010

84. Tale of Two Startups
“Traditional” Operations Operations - The “Secret Sauce”
50 50
This is the secret of
40 40
Cloud Computing.
Every other virtue stems from
# of Hours
30 30
here.
20 20
Hardware
OS Install
10 10 Config
Upkeep
20 20
15 15
Servers
10 10 New
5 5 Existing
0 0
1 2 3 4 5 6 7 9 10 11 12 1 2 3 4 5 6 7 9 10 11 12
Week # Week #
(http://radar.oreilly.com/archives/2007/10/operations-advantage.html)
Copyright © 2010 Opscode, Inc - All Rights Reserved 35
Thursday, July 29, 2010

85. Thursday, July 29, 2010

86. A Period of Combinatorial
Innovation
Thursday, July 29, 2010

87. A Period of Combinatorial
Innovation
• Abstract and fault tolerant
components
Thursday, July 29, 2010

88. A Period of Combinatorial
Innovation
• Abstract and fault tolerant
components
• Integrated network accessible
services
Thursday, July 29, 2010

89. A Period of Combinatorial
Innovation
• Abstract and fault tolerant
components
• Integrated network accessible
services
• Unlimited infrastructure
Thursday, July 29, 2010

90. Industry Shifts
Be bold-and mighty forces
will come to your aid
Basil King
Copyright © 2010 Opscode, Inc - All Rights Reserved 37
Thursday, July 29, 2010