• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Managing Databases In A DevOps Environment
 

Managing Databases In A DevOps Environment

on

  • 3,689 views

There’s a lot of talk in the devops world about bringing developer concepts to system administration, and discussion the other way about bringing the awareness of operations to developers, but a lot ...

There’s a lot of talk in the devops world about bringing developer concepts to system administration, and discussion the other way about bringing the awareness of operations to developers, but a lot of the conversation leaves out what is often the most critical part of your technology stack: the database. Perhaps that’s because DBA’s have always had to keep one foot in development and one in production, before there was a devops. Or maybe DBA’s just suck at playing well with others. Bottom line; it doesn’t matter. If you are going to store data, you need a plan that both developers and operations people can understand and embrace.

At OmniTI we’ve worked with many of the leaders in the devops movement and we’ve found there are commonalties across these organizations. It’s not so much about the tools, but about the techniques they use to help people break down barriers between different roles and establish a common ownership of technology within their organizations.

Monitoring and visibility, managing schema changes and production pushes, optimization, configuration and backups; there are aspects to data storage that bring about unique challenges. You won’t need to adopt all of these techniques to be successful, but it’s time you had a frank conversation about what it takes to make your database truly “webscale”.

Slides from PGOpen 2011, But this talk was also delivered at Velocity 2011 as well.

Statistics

Views

Total Views
3,689
Views on SlideShare
3,661
Embed Views
28

Actions

Likes
10
Downloads
0
Comments
0

6 Embeds 28

http://lanyrd.com 16
http://omniti.com 4
http://paper.li 3
http://www.linkedin.com 2
http://www.omniti.com 2
http://twitter.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Managing Databases In A DevOps Environment Managing Databases In A DevOps Environment Presentation Transcript

    • Managing Databases in a {DEVOPS} Environment / Robert Treat 1Friday, September 16, 11
    • Who Am I? Robert Treat @robtreat2 xzilla.net 2Friday, September 16, 11
    • Who Am I? OmniTI 3Friday, September 16, 11
    • Who Am I? “Database Operations” 4Friday, September 16, 11
    • Who Am I? Highly Scaled Big Data Heavy Transactions 5Friday, September 16, 11
    • Who Am I? Highly Scaled Big Data Heavy Transactions Really Important Stuff 6Friday, September 16, 11
    • What Is DevOps? 7Friday, September 16, 11
    • put your development in my ops 8Friday, September 16, 11
    • put your ops in my development 9Friday, September 16, 11
    • MINDMELD 10Friday, September 16, 11
    • 11Friday, September 16, 11
    • Not A Talk About DevOps per se 12Friday, September 16, 11
    • I like to make up definitions 13Friday, September 16, 11
    • DevOps Is Basically About Three Things 14Friday, September 16, 11
    • Configuration Management 15Friday, September 16, 11
    • Monitoring 16Friday, September 16, 11
    • Software Development 17Friday, September 16, 11
    • Yes I just offended everyone with a “kanban” t-shirt 18Friday, September 16, 11
    • Moving On 19Friday, September 16, 11
    • Configuration Management 20Friday, September 16, 11
    • Some SysAdmins get Very Excited about Systems Automation 21Friday, September 16, 11
    • 22Friday, September 16, 11
    • meh 23Friday, September 16, 11
    • Many DBA’s Have Like One Box to Manage 24Friday, September 16, 11
    • This Makes Server Automation Seem Like a Hassle 25Friday, September 16, 11
    • Now You Have Two Problems 26Friday, September 16, 11
    • Even With One Box, There Are Many Things To Configure 27Friday, September 16, 11
    • database configs, cron jobs, packages, scripts monitoring, binaries, shell config, pooler 28Friday, September 16, 11
    • A Lot To Configure 29Friday, September 16, 11
    • Needs To Be Documented 30Friday, September 16, 11
    • Needs To Be Documented or At Least Reproducible 31Friday, September 16, 11
    • It Would Be Nice To Know When Changes Occur In Your Environment 32Friday, September 16, 11
    • Configuration Management Helps You With All That 33Friday, September 16, 11
    • Configuration Management 34Friday, September 16, 11
    • “Configuration” <--> “Management” 35Friday, September 16, 11
    • Even Without Automation 36Friday, September 16, 11
    • Track Changes 37Friday, September 16, 11
    • Notification Of Changes 38Friday, September 16, 11
    • Also Consider 39Friday, September 16, 11
    • Your “One Big Database” 40Friday, September 16, 11
    • Is Probably More Than Just One Database 41Friday, September 16, 11
    • data loading webservers script writes writes master writes reads replay replay slave A slave B slave slave slave slave slave slave reads load balancer 42Friday, September 16, 11
    • Failover, Reporting, Staging, Development, Read Slaves, And More! 43Friday, September 16, 11
    • And Then Someone Sets Up A New Service 44Friday, September 16, 11
    • Now You Have To Configure Everything Again For Multiple Machines 45Friday, September 16, 11
    • How Long Does It Take You To Stand Up A New Copy Of Your Database? 46Friday, September 16, 11
    • How Confident Are You That You Have Configured Everything Correctly? 47Friday, September 16, 11
    • Monitoring For DevOps 48Friday, September 16, 11
    • Monitor Everything! 49Friday, September 16, 11
    • System Stuff: 50Friday, September 16, 11
    • Disk IO, CPU, Network Traffic Load, Disk Utilization 51Friday, September 16, 11
    • Database Stuff: 52Friday, September 16, 11
    • Transactions Per Second 53Friday, September 16, 11
    • 54Friday, September 16, 11
    • TPS, Connections, Index Scans, Seq Scans, Table Size 55Friday, September 16, 11
    • TPS, Connections, Index Scans, Seq Scans, Table Size Cache Hit Ratio 56Friday, September 16, 11
    • Try Not To Get Hung Up On The DETAILS 57Friday, September 16, 11
    • DBA’s Are Sometimes Finicky About Data Quality 58Friday, September 16, 11
    • It is better to have USEFUL Metrics Than to have ACCURATE Metrics 59Friday, September 16, 11
    • Metrics Enable Capacity Planning 60Friday, September 16, 11
    • 61Friday, September 16, 11
    • Metrics Let You Know What Good Looks Like 62Friday, September 16, 11
    • Metrics Enable Forensics 63Friday, September 16, 11
    • Why Did My Site Break? 64Friday, September 16, 11
    • You Should Also Track Changes 65Friday, September 16, 11
    • Configuration Settings Code Pushes Traffic Spikes 66Friday, September 16, 11
    • Make It Easy To Correlate Changes 67Friday, September 16, 11
    • Expose This To EVERYONE! 68Friday, September 16, 11
    • DBAs, SAs, Devs Support Folks, Finance Team The Cute New Intern EVERYONE! 69Friday, September 16, 11
    • Use Them To Make Decisions 70Friday, September 16, 11
    • Use Them To Make Decisions 71Friday, September 16, 11
    • Use Them To Make Decisions 72Friday, September 16, 11
    • Push Baby Push! 73Friday, September 16, 11
    • A Bazillion Deploys A Day! 74Friday, September 16, 11
    • From Idea To Code In The Blink Of An Eye! 75Friday, September 16, 11
    • What’s the Problem? 76Friday, September 16, 11
    • This Doesn’t Work Well For Databases 77Friday, September 16, 11
    • Databases Are Fundamentally Different 78Friday, September 16, 11
    • They Have Data 79Friday, September 16, 11
    • Data Is Important 80Friday, September 16, 11
    • And Hard To Replace 81Friday, September 16, 11
    • You Can Always Rewrite The Code 82Friday, September 16, 11
    • 83Friday, September 16, 11
    • You Can’t Get Everyone To Give You Their Data Again 84Friday, September 16, 11
    • 85Friday, September 16, 11
    • Furthermore 86Friday, September 16, 11
    • Code Is Static 87Friday, September 16, 11
    • Code Is Controlled 88Friday, September 16, 11
    • Data Is Often Dynamic 89Friday, September 16, 11
    • Control Is In The Hands Of Users 90Friday, September 16, 11
    • This Makes Database Administration More Difficult 91Friday, September 16, 11
    • We Can’t Eliminate Change 92Friday, September 16, 11
    • We Must Manage Change 93Friday, September 16, 11
    • Back In The Day 94Friday, September 16, 11
    • DBA’s Lived In Walled Gardens 95Friday, September 16, 11
    • Dev’s Send You Requirements 96Friday, September 16, 11
    • 97Friday, September 16, 11
    • You Send Them A Schema 98Friday, September 16, 11
    • 99Friday, September 16, 11
    • They Send You Modified Requirements 100Friday, September 16, 11
    • 101Friday, September 16, 11
    • You Send Them A New Schema 102Friday, September 16, 11
    • This Goes Back And Forth 103Friday, September 16, 11
    • This Goes Back And Forth 104Friday, September 16, 11
    • Dev’s Keep Pushing Just To Get Things Done 105Friday, September 16, 11
    • You Try To Temper Things To Minimize Disruption 106Friday, September 16, 11
    • Eventually Something Is Agreed Upon 107Friday, September 16, 11
    • But This Takes A Long Time 108Friday, September 16, 11
    • 109Friday, September 16, 11
    • Many Shops Respond By Eliminating The DBA’s 110Friday, September 16, 11
    • But You Still Need Someone To Make Things Work 111Friday, September 16, 11
    • You Need Balance 112Friday, September 16, 11
    • 113Friday, September 16, 11
    • Devs Design The Schema 114Friday, September 16, 11
    • Blasphemy! 115Friday, September 16, 11
    • Devs Meet With Project Managers 116Friday, September 16, 11
    • Devs Digest Requirements 117Friday, September 16, 11
    • Devs Understand The App 118Friday, September 16, 11
    • Your Job Is To Be Available If They Have Questions 119Friday, September 16, 11
    • If You Had A {Perl|Ruby|Python|Javascript} Question, You Would Ask A Senior Dev 120Friday, September 16, 11
    • DBAs Are The Senior Devs WRT Databases 121Friday, September 16, 11
    • Make A Style Guide For Schemas 122Friday, September 16, 11
    • Schema Changes Become Migrations 123Friday, September 16, 11
    • New Stuff, Changed Stuff 124Friday, September 16, 11
    • Migrations Are Subject To Review 125Friday, September 16, 11
    • Look For Red Flags 126Friday, September 16, 11
    • Nulls, Indexes, Datatypes, Constraints 127Friday, September 16, 11
    • What Is The Impact? 128Friday, September 16, 11
    • Locking, Data Rewrites, Other Stuff That Sucks 129Friday, September 16, 11
    • 1000 Pushes A Day? 130Friday, September 16, 11
    • No 131Friday, September 16, 11
    • Remember Locking? 132Friday, September 16, 11
    • You Have Peak Times 133Friday, September 16, 11
    • Build Schema Migration Windows Around Off-Peak Hours 134Friday, September 16, 11
    • Once A Week 135Friday, September 16, 11
    • Once A Day 136Friday, September 16, 11
    • Just Not During Lunch 137Friday, September 16, 11
    • You Can’t Eliminate Risk 138Friday, September 16, 11
    • Minimize Unnecessary Risk 139Friday, September 16, 11
    • And It’s Worth Pointing Out 140Friday, September 16, 11
    • Schema Changes Don’t Have To Be Married To Code Changes 141Friday, September 16, 11
    • If Your Feature Launches Next Week 142Friday, September 16, 11
    • You Can Make Changes This Week 143Friday, September 16, 11
    • Add Columns Well In Advance Of Writing To Them 144Friday, September 16, 11
    • Stop Writing To Columns Well Before Dropping Them 145Friday, September 16, 11
    • Like Development It’s Easier To Digest 146Friday, September 16, 11
    • Small Incremental Changes 147Friday, September 16, 11
    • DevOps Is A New Way Of Thinking 148Friday, September 16, 11
    • DBAs Are The Original DevOps 149Friday, September 16, 11
    • It’s Time We Got On The Same Team 150Friday, September 16, 11
    • THANK YOU! #pgopen11 151Friday, September 16, 11
    • http://www.flickr.com/photos/rubink1/3019322879/ http://www.flickr.com/photos/nettsu/4570198529/ http://www.flickr.com/photos/camknows/5227248712/ http://www.flickr.com/photos/fornal/373418814/ http://www.flickr.com/photos/kathryn_rotondo/473877266/ http://www.flickr.com/photos/remco0808/4121818178/ http://www.flickr.com/photos/john/3807248802/ http://www.flickr.com/photos/facilitybikeclub/3197419294/ http://www.flickr.com/photos/michperu/3969552741/ http://www.flickr.com/photos/winterofdiscontent/4101596626/ http://www.flickr.com/photos/zooboing/4361038612/ http://www.flickr.com/photos/sequoya/2236491391/ http://www.flickr.com/photos/blakespot/4011035061/ http://www.flickr.com/photos/ikhnaton2/533233247/ http://www.flickr.com/photos/stayinsussex/3801588698/ http://www.flickr.com/photos/liamrichardsx/4373285723/ 152Friday, September 16, 11