There’s a lot of talk in the devops world about bringing developer concepts to system administration, and discussion the other way about bringing the awareness of operations to developers, but a lot of the conversation leaves out what is often the most critical part of your technology stack: the database. Perhaps that’s because DBA’s have always had to keep one foot in development and one in production, before there was a devops. Or maybe DBA’s just suck at playing well with others. Bottom line; it doesn’t matter. If you are going to store data, you need a plan that both developers and operations people can understand and embrace. At OmniTI we’ve worked with many of the leaders in the devops movement and we’ve found there are commonalties across these organizations. It’s not so much about the tools, but about the techniques they use to help people break down barriers between different roles and establish a common ownership of technology within their organizations. Monitoring and visibility, managing schema changes and production pushes, optimization, configuration and backups; there are aspects to data storage that bring about unique challenges. You won’t need to adopt all of these techniques to be successful, but it’s time you had a frank conversation about what it takes to make your database truly “webscale”. Slides from PGOpen 2011, But this talk was also delivered at Velocity 2011 as well.

1. Managing Databases in a
{DEVOPS} Environment
/ Robert Treat
1
Friday, September 16, 11

2. Who Am I?
Robert Treat
@robtreat2
xzilla.net
2
Friday, September 16, 11

3. Who Am I?
OmniTI
3
Friday, September 16, 11

4. Who Am I?
“Database
Operations”
4
Friday, September 16, 11

5. Who Am I?
Highly Scaled
Big Data
Heavy Transactions
5
Friday, September 16, 11

6. Who Am I?
Highly Scaled
Big Data
Heavy Transactions
Really Important Stuff
6
Friday, September 16, 11

7. What Is DevOps?
7
Friday, September 16, 11

8. put your development in my ops
8
Friday, September 16, 11

9. put your ops in my development
9
Friday, September 16, 11

10. MINDMELD
10
Friday, September 16, 11

11. 11
Friday, September 16, 11

12. Not A Talk About
DevOps
per se
12
Friday, September 16, 11

13. I like to make up definitions
13
Friday, September 16, 11

14. DevOps Is Basically About
Three Things
14
Friday, September 16, 11

15. Configuration Management
15
Friday, September 16, 11

16. Monitoring
16
Friday, September 16, 11

17. Software Development
17
Friday, September 16, 11

18. Yes
I just offended
everyone
with a
“kanban”
t-shirt
18
Friday, September 16, 11

19. Moving On
19
Friday, September 16, 11

20. Configuration Management
20
Friday, September 16, 11

21. Some SysAdmins get
Very Excited
about
Systems Automation
21
Friday, September 16, 11

22. 22
Friday, September 16, 11

23. meh
23
Friday, September 16, 11

24. Many
DBA’s
Have Like
One Box to
Manage
24
Friday, September 16, 11

25. This Makes Server Automation
Seem Like a Hassle
25
Friday, September 16, 11

26. Now You Have Two Problems
26
Friday, September 16, 11

27. Even With One Box, There Are
Many Things To Configure
27
Friday, September 16, 11

28. database configs, cron jobs, packages, scripts
monitoring, binaries, shell config, pooler
28
Friday, September 16, 11

29. A Lot To Configure
29
Friday, September 16, 11

30. Needs To Be Documented
30
Friday, September 16, 11

31. Needs To Be Documented
or
At Least Reproducible
31
Friday, September 16, 11

32. It Would Be Nice To Know
When Changes Occur In
Your Environment
32
Friday, September 16, 11

33. Configuration Management
Helps You
With All That
33
Friday, September 16, 11

34. Configuration Management
34
Friday, September 16, 11

35. “Configuration”
<-->
“Management”
35
Friday, September 16, 11

36. Even Without Automation
36
Friday, September 16, 11

37. Track Changes
37
Friday, September 16, 11

38. Notification Of Changes
38
Friday, September 16, 11

39. Also Consider
39
Friday, September 16, 11

40. Your
“One Big Database”
40
Friday, September 16, 11

41. Is Probably More Than
Just One Database
41
Friday, September 16, 11

42. data loading
webservers script
writes writes
master
writes
reads
replay replay
slave A slave B
slave
slave slave slave slave slave
reads
load balancer
42
Friday, September 16, 11

43. Failover, Reporting, Staging,
Development, Read Slaves,
And More!
43
Friday, September 16, 11

44. And Then
Someone
Sets Up
A
New Service
44
Friday, September 16, 11

45. Now You Have To Configure
Everything Again For Multiple
Machines
45
Friday, September 16, 11

46. How Long Does It Take You To Stand
Up A New Copy Of Your Database?
46
Friday, September 16, 11

47. How Confident Are You That You
Have Configured Everything
Correctly?
47
Friday, September 16, 11

48. Monitoring For DevOps
48
Friday, September 16, 11

49. Monitor Everything!
49
Friday, September 16, 11

50. System Stuff:
50
Friday, September 16, 11

51. Disk IO, CPU, Network Traffic
Load, Disk Utilization
51
Friday, September 16, 11

52. Database Stuff:
52
Friday, September 16, 11

53. Transactions Per Second
53
Friday, September 16, 11

54. 54
Friday, September 16, 11

55. TPS, Connections, Index Scans,
Seq Scans, Table Size
55
Friday, September 16, 11

56. TPS, Connections, Index Scans,
Seq Scans, Table Size
Cache Hit Ratio
56
Friday, September 16, 11

57. Try Not To Get Hung Up On The
DETAILS
57
Friday, September 16, 11

58. DBA’s Are Sometimes Finicky About
Data Quality
58
Friday, September 16, 11

59. It is better to have USEFUL Metrics
Than to have ACCURATE Metrics
59
Friday, September 16, 11

60. Metrics Enable Capacity Planning
60
Friday, September 16, 11

61. 61
Friday, September 16, 11

62. Metrics Let You Know What
Good Looks Like
62
Friday, September 16, 11

63. Metrics Enable Forensics
63
Friday, September 16, 11

64. Why Did My Site Break?
64
Friday, September 16, 11

65. You Should Also Track Changes
65
Friday, September 16, 11

66. Configuration Settings
Code Pushes
Traffic Spikes
66
Friday, September 16, 11

67. Make It Easy To
Correlate Changes
67
Friday, September 16, 11

68. Expose This To
EVERYONE!
68
Friday, September 16, 11

69. DBAs, SAs, Devs
Support Folks, Finance Team
The Cute New Intern
EVERYONE!
69
Friday, September 16, 11

70. Use Them To Make Decisions
70
Friday, September 16, 11

71. Use Them To Make Decisions
71
Friday, September 16, 11

72. Use Them To Make Decisions
72
Friday, September 16, 11

73. Push Baby Push!
73
Friday, September 16, 11

74. A Bazillion Deploys A Day!
74
Friday, September 16, 11

75. From Idea To Code
In The Blink Of An Eye!
75
Friday, September 16, 11

76. What’s the Problem?
76
Friday, September 16, 11

77. This Doesn’t Work Well
For Databases
77
Friday, September 16, 11

78. Databases Are Fundamentally
Different
78
Friday, September 16, 11

79. They Have Data
79
Friday, September 16, 11

80. Data Is Important
80
Friday, September 16, 11

81. And Hard To Replace
81
Friday, September 16, 11

82. You Can Always Rewrite The Code
82
Friday, September 16, 11

83. 83
Friday, September 16, 11

84. You Can’t Get Everyone To Give You
Their Data Again
84
Friday, September 16, 11

85. 85
Friday, September 16, 11

86. Furthermore
86
Friday, September 16, 11

87. Code Is Static
87
Friday, September 16, 11

88. Code Is Controlled
88
Friday, September 16, 11

89. Data Is Often Dynamic
89
Friday, September 16, 11

90. Control Is In The Hands Of Users
90
Friday, September 16, 11

91. This Makes
Database Administration
More Difficult
91
Friday, September 16, 11

92. We Can’t Eliminate
Change
92
Friday, September 16, 11

93. We Must Manage
Change
93
Friday, September 16, 11

94. Back In The Day
94
Friday, September 16, 11

95. DBA’s Lived In Walled Gardens
95
Friday, September 16, 11

96. Dev’s Send You Requirements
96
Friday, September 16, 11

97. 97
Friday, September 16, 11

98. You Send Them A Schema
98
Friday, September 16, 11

99. 99
Friday, September 16, 11

100. They Send You Modified Requirements
100
Friday, September 16, 11

101. 101
Friday, September 16, 11

102. You Send Them A New Schema
102
Friday, September 16, 11

103. This Goes Back And Forth
103
Friday, September 16, 11

104. This Goes Back And Forth
104
Friday, September 16, 11

105. Dev’s Keep Pushing
Just To Get Things Done
105
Friday, September 16, 11

106. You Try To Temper Things To
Minimize Disruption
106
Friday, September 16, 11

107. Eventually Something
Is Agreed Upon
107
Friday, September 16, 11

108. But This Takes A Long Time
108
Friday, September 16, 11

109. 109
Friday, September 16, 11

110. Many Shops Respond By Eliminating
The DBA’s
110
Friday, September 16, 11

111. But You Still Need
Someone
To Make Things Work
111
Friday, September 16, 11

112. You Need Balance
112
Friday, September 16, 11

113. 113
Friday, September 16, 11

114. Devs Design The Schema
114
Friday, September 16, 11

115. Blasphemy!
115
Friday, September 16, 11

116. Devs
Meet With Project Managers
116
Friday, September 16, 11

117. Devs
Digest Requirements
117
Friday, September 16, 11

118. Devs
Understand The App
118
Friday, September 16, 11

119. Your Job Is To Be Available If They
Have Questions
119
Friday, September 16, 11

120. If You Had A
{Perl|Ruby|Python|Javascript}
Question, You Would
Ask A Senior Dev
120
Friday, September 16, 11

121. DBAs Are The Senior Devs
WRT Databases
121
Friday, September 16, 11

122. Make A Style Guide For Schemas
122
Friday, September 16, 11

123. Schema Changes Become
Migrations
123
Friday, September 16, 11

124. New Stuff, Changed Stuff
124
Friday, September 16, 11

125. Migrations Are Subject To Review
125
Friday, September 16, 11

126. Look For Red Flags
126
Friday, September 16, 11

127. Nulls, Indexes,
Datatypes, Constraints
127
Friday, September 16, 11

128. What Is The Impact?
128
Friday, September 16, 11

129. Locking, Data Rewrites,
Other Stuff That Sucks
129
Friday, September 16, 11

130. 1000 Pushes A Day?
130
Friday, September 16, 11

131. No
131
Friday, September 16, 11

132. Remember Locking?
132
Friday, September 16, 11

133. You Have Peak Times
133
Friday, September 16, 11

134. Build Schema Migration Windows
Around Off-Peak Hours
134
Friday, September 16, 11

135. Once A Week
135
Friday, September 16, 11

136. Once A Day
136
Friday, September 16, 11

137. Just Not During Lunch
137
Friday, September 16, 11

138. You Can’t Eliminate Risk
138
Friday, September 16, 11

139. Minimize Unnecessary Risk
139
Friday, September 16, 11

140. And It’s Worth Pointing Out
140
Friday, September 16, 11

141. Schema Changes Don’t Have To Be
Married To Code Changes
141
Friday, September 16, 11

142. If Your Feature Launches
Next Week
142
Friday, September 16, 11

143. You Can Make Changes This Week
143
Friday, September 16, 11

144. Add Columns Well
In Advance
Of Writing To Them
144
Friday, September 16, 11

145. Stop Writing To Columns
Well Before
Dropping Them
145
Friday, September 16, 11

146. Like Development It’s
Easier
To Digest
146
Friday, September 16, 11

147. Small
Incremental
Changes
147
Friday, September 16, 11

148. DevOps Is A
New Way Of Thinking
148
Friday, September 16, 11

149. DBAs Are The
Original DevOps
149
Friday, September 16, 11

150. It’s Time We Got On
The Same Team
150
Friday, September 16, 11

151. THANK YOU!
#pgopen11
151
Friday, September 16, 11

152. http://www.flickr.com/photos/rubink1/3019322879/
http://www.flickr.com/photos/nettsu/4570198529/
http://www.flickr.com/photos/camknows/5227248712/
http://www.flickr.com/photos/fornal/373418814/
http://www.flickr.com/photos/kathryn_rotondo/473877266/
http://www.flickr.com/photos/remco0808/4121818178/
http://www.flickr.com/photos/john/3807248802/
http://www.flickr.com/photos/facilitybikeclub/3197419294/
http://www.flickr.com/photos/michperu/3969552741/
http://www.flickr.com/photos/winterofdiscontent/4101596626/
http://www.flickr.com/photos/zooboing/4361038612/
http://www.flickr.com/photos/sequoya/2236491391/
http://www.flickr.com/photos/blakespot/4011035061/
http://www.flickr.com/photos/ikhnaton2/533233247/
http://www.flickr.com/photos/stayinsussex/3801588698/
http://www.flickr.com/photos/liamrichardsx/4373285723/
152
Friday, September 16, 11