Managing Databases In A DevOps Environment

Managing Databases in a
{DEVOPS} Environment

/ Robert Treat

1

Friday, September 16, 11

Who Am I?

Robert Treat
@robtreat2
xzilla.net

2


Who Am I?

OmniTI

3


Who Am I?

“Database
Operations”

4


Who Am I?

Highly Scaled
Big Data
Heavy Transactions

5


Who Am I?

Highly Scaled
Big Data
Heavy Transactions

Really Important Stuff
6


What Is DevOps?

7


put your development in my ops

8


put your ops in my development

9


MINDMELD
10


Not A Talk About
DevOps
per se

12


I like to make up deﬁnitions

13


DevOps Is Basically About
Three Things

14


Conﬁguration Management

15


Monitoring

16


Software Development

17


Yes
I just offended
everyone
with a
“kanban”
t-shirt

18


Moving On

19



20


Some SysAdmins get
Very Excited
about
Systems Automation

21


meh

23


Many
DBA’s
Have Like
One Box to
Manage
24


This Makes Server Automation
Seem Like a Hassle

25


Now You Have Two Problems

26


Even With One Box, There Are
Many Things To Conﬁgure

27


database conﬁgs, cron jobs, packages, scripts

monitoring, binaries, shell conﬁg, pooler
28


A Lot To Conﬁgure

29


Needs To Be Documented

30


Needs To Be Documented
or
At Least Reproducible

31


It Would Be Nice To Know
When Changes Occur In
Your Environment

32


Helps You
With All That

33



34


“Conﬁguration”
<-->
“Management”

35


Even Without Automation

36


Track Changes

37


Notiﬁcation Of Changes

38


Also Consider

39


Your
“One Big Database”

40


Is Probably More Than
Just One Database

41


data loading
webservers script

writes writes

master

writes
reads
replay replay
slave A slave B

slave
slave slave slave slave slave

reads

load balancer
42


Failover, Reporting, Staging,
Development, Read Slaves,
And More!

43


And Then
Someone
Sets Up
A
New Service

44


Now You Have To Conﬁgure
Everything Again For Multiple
Machines

45


How Long Does It Take You To Stand
Up A New Copy Of Your Database?

46


How Conﬁdent Are You That You
Have Conﬁgured Everything
Correctly?

47


Monitoring For DevOps

48


Monitor Everything!

49


System Stuff:

50


Disk IO, CPU, Network Trafﬁc
Load, Disk Utilization

51


Database Stuff:

52


Transactions Per Second

53


TPS, Connections, Index Scans,
Seq Scans, Table Size

55


TPS, Connections, Index Scans,
Seq Scans, Table Size
Cache Hit Ratio

56


Try Not To Get Hung Up On The
DETAILS

57


DBA’s Are Sometimes Finicky About
Data Quality

58


It is better to have USEFUL Metrics
Than to have ACCURATE Metrics

59


Metrics Enable Capacity Planning

60


Metrics Let You Know What
Good Looks Like

62


Metrics Enable Forensics

63


Why Did My Site Break?

64


You Should Also Track Changes

65


Conﬁguration Settings
Code Pushes
Trafﬁc Spikes

66


Make It Easy To
Correlate Changes

67


Expose This To
EVERYONE!

68


DBAs, SAs, Devs
Support Folks, Finance Team
The Cute New Intern
EVERYONE!

69


Use Them To Make Decisions

70



71



72


Push Baby Push!

73


A Bazillion Deploys A Day!

74


From Idea To Code
In The Blink Of An Eye!

75


What’s the Problem?

76


This Doesn’t Work Well
For Databases

77


Databases Are Fundamentally
Different

78


They Have Data

79


Data Is Important

80


And Hard To Replace

81


You Can Always Rewrite The Code

82


You Can’t Get Everyone To Give You
Their Data Again

84


Furthermore

86


Code Is Static

87


Code Is Controlled

88


Data Is Often Dynamic

89


Control Is In The Hands Of Users

90


This Makes
Database Administration
More Difﬁcult

91


We Can’t Eliminate
Change

92


We Must Manage
Change

93


Back In The Day

94


DBA’s Lived In Walled Gardens

95


Dev’s Send You Requirements

96


You Send Them A Schema

98


They Send You Modiﬁed Requirements

100


101


You Send Them A New Schema

102


This Goes Back And Forth

103


This Goes Back And Forth

104


Dev’s Keep Pushing
Just To Get Things Done

105


You Try To Temper Things To
Minimize Disruption

106


Eventually Something
Is Agreed Upon

107


But This Takes A Long Time

108


109


Many Shops Respond By Eliminating
The DBA’s

110


But You Still Need
Someone
To Make Things Work

111


You Need Balance

112


113


Devs Design The Schema

114


Blasphemy!

115


Devs
Meet With Project Managers

116


Devs
Digest Requirements

117


Devs
Understand The App

118


Your Job Is To Be Available If They
Have Questions

119


If You Had A
{Perl|Ruby|Python|Javascript}
Question, You Would
Ask A Senior Dev

120


DBAs Are The Senior Devs
WRT Databases

121


Make A Style Guide For Schemas

122


Schema Changes Become
Migrations

123


New Stuff, Changed Stuff

124


Migrations Are Subject To Review

125


Look For Red Flags

126


Nulls, Indexes,
Datatypes, Constraints

127


What Is The Impact?

128


Locking, Data Rewrites,
Other Stuff That Sucks

129


1000 Pushes A Day?

130


No

131


Remember Locking?

132


You Have Peak Times

133


Build Schema Migration Windows
Around Off-Peak Hours

134


Once A Week

135


Once A Day

136


Just Not During Lunch

137


You Can’t Eliminate Risk

138


Minimize Unnecessary Risk

139


And It’s Worth Pointing Out

140


Schema Changes Don’t Have To Be
Married To Code Changes

141


If Your Feature Launches
Next Week

142


You Can Make Changes This Week

143


Add Columns Well
In Advance
Of Writing To Them

144


Stop Writing To Columns
Well Before
Dropping Them

145


Like Development It’s
Easier
To Digest

146


Small
Incremental
Changes

147


DevOps Is A
New Way Of Thinking

148


DBAs Are The
Original DevOps

149


It’s Time We Got On
The Same Team

150


THANK YOU!

#pgopen11

151


http://www.flickr.com/photos/rubink1/3019322879/
http://www.flickr.com/photos/nettsu/4570198529/
http://www.flickr.com/photos/camknows/5227248712/
http://www.flickr.com/photos/fornal/373418814/
http://www.flickr.com/photos/kathryn_rotondo/473877266/
http://www.flickr.com/photos/remco0808/4121818178/
http://www.flickr.com/photos/john/3807248802/
http://www.flickr.com/photos/facilitybikeclub/3197419294/
http://www.flickr.com/photos/michperu/3969552741/
http://www.flickr.com/photos/winterofdiscontent/4101596626/
http://www.flickr.com/photos/zooboing/4361038612/
http://www.flickr.com/photos/sequoya/2236491391/
http://www.flickr.com/photos/blakespot/4011035061/
http://www.flickr.com/photos/ikhnaton2/533233247/
http://www.flickr.com/photos/stayinsussex/3801588698/
http://www.flickr.com/photos/liamrichardsx/4373285723/

152


Managing Databases In A DevOps Environment

More Related Content

Viewers also liked

Similar to Managing Databases In A DevOps Environment

More from Robert Treat

Recently uploaded

Managing Databases In A DevOps Environment