Data, the way that we process it and store it, is one of many important aspects of IT. Data is the lifeblood of our organizations, supporting real-time business processes and decision-making. For our DevOps strategy to be truly effective we must be able to safely and quickly evolve production databases, just as we safely and quickly evolve production code. Yet for many organizations their data sources prove to be less than trustworthy and their data-oriented development efforts little more than productivity sinkholes. We can, and must, do better. This presentation begins with a collection of agile principles for data professionals and of data principles for agile developers - the first step in working together is to understand and appreciate the priorities and strengths of the people that we work with. Our focus is on a collection of practices that enable development teams to easily and safely evolve and deploy databases. These techniques include agile data modeling, database refactoring, database regression testing, continuous database integration, and continuous database deployment. We also work through operational strategies required of production databases to support your DevOps strategy. If data sources aren’t an explicit part of your DevOps strategy then you’re not really doing DevOps, are you?

1. Data DevOps
Scott W. Ambler
Consulting Methodologist | Agile Data Coach
Ambysoft Inc.
© Ambysoft Inc. All rights reserved.
1
Download these slides at Slideshare.net/ScottWAmbler/

2. Agenda
© Ambysoft Inc. All rights reserved. 2
• Data DevOps
• Agile data development techniques
• Agile data operations techniques
• Parting thoughts
• Questions and answers

3. Scott Ambler
© Ambysoft Inc. All rights reserved. 3
• scott@scottambler.com
• Twitter: @scottwambler
• linkedin.com/in/sambler/
Consulting
Methodologist
Ambysoft.com
Thought Leader
AgileData.org
Thought Leader
AgileModeling.com

4. © Ambysoft Inc. All rights reserved. 4
Data
Dev
Data
Ops

5. Defining Data DevOps
Data Development (Data Dev)
The effort to develop and evolve the data aspects of your solutions
Data Operations (Data Ops)
The effort to operate, support, and govern the data aspects of your solutions
Data DevOps
The streamlined combination of Data Dev + Data Ops, part of your overall DevOps
strategy
© Ambysoft Inc. All rights reserved. 5

6. © Ambysoft Inc. All rights reserved. 6
Data
Dev
Data
Ops
Agile data
modeling
Database
refactoring
Continuous
database
integration (CDI)
Automated
database
regression
testing
Continuous database
deployment (CDD)
Operational data
quality assurance
Data
security
Critical Data DevOps Techniques

7. © Ambysoft Inc. All rights reserved. 7
Data modeling is the act of exploring
data-oriented structures
Evolutionary data modeling is data
modeling performed in an iterative
and incremental manner
Agile data modeling is evolutionary
data modeling done in a collaborative
manner
Source: AgileData.org/essays/agileDataModeling.html
Agile Data Modeling
Construction Sprint
JIT Modeling
Detailed physical data
modeling (to generate DDL)
Sprint 0:
Initial Modeling
High-level conceptual modeling
Test-Driven
Development
Detailed specification

8. © Ambysoft Inc. All rights reserved. 8
Source: AgileData.org/essays/lookAheadDataAnalysis.html
Agile Data Modeling: Look-Ahead Data Analysis
There are several factors that will determine how far ahead you need to perform look-ahead data analysis:
• The complexity of the data source(s).
• Your ability to gain access to the data source(s).
• The difficulty of the question being asked.
• The skill, experience, and knowledge of the data analyst(s).
• The availability of the data analyst(s).
• Your data profiling tools.
Sometimes it takes several days, even weeks, to perform data analytics before you can implement
a question story.

9. © Ambysoft Inc. All rights reserved. 9
A database refactoring is a simple change to a database
schema that improves its design while retaining both its
behavioral and informational semantics
A database schema includes structural aspects such as
table and view definitions; functional aspects such as
stored procedures and triggers; and informational
aspects such as the data itself
Source: AgileData.org/essays/databaseRefactoring.html
Database Refactoring

10. © Ambysoft Inc. All rights reserved. 10
Source: AgileData.org/essays/databaseRefactoring.html
Customer
CustomerID <<PK>>
Fname
Customer
CustomerID <<PK>>
Fname
FirstName
SynchronizeFirstName()
Customer
CustomerID <<PK>>
FirstName
Original
Schema:
Interim
Schema:
Final
Schema:
Database Refactoring: Rename Column

11. © Ambysoft Inc. All rights reserved. 11
• Part of building the system is building the
database (if it changed)
• Challenge: Tests SHOULD put the database back
into a known state, but sometimes don’t
• You will want to rebuild the (non-production)
database from scratch every so often
• Challenge: Database accesses take time
• Some test suites will test against DB mocks
• You still need to test the actual database
occasionally
• Source:
AgileData.org/essays/continuousIntegration.html
Continuous Database Integration (CDI)

12. © Ambysoft Inc. All rights reserved. 12
Source: AgileData.org/essays/databaseTesting.html
Database Regression Testing

13. © Ambysoft Inc. All rights reserved. 13
Continuous Deployment between Sandboxes
Development Integration 1 Integration N Production
• Each integration DB must know it’s version, so as to support database refactoring
• CDI differs between sandboxes
• Development sandboxes often implement database mocks
• Integration sandboxes should test against the database
• Production has limited tests and checks for performance reasons
Source: AgileData.org/essays/sandboxes.html

14. Operational Data Quality Assurance
There are many techniques available to ensure operational data quality
• Data transformation
• Data masking
• Data stewardship
• Data privacy
• Data governance
• Data source backup
• Data lineage
• Data history
• Data cleansing
Challenges:
• Increasing volume of data
• Increasing data arrival rate
© Ambysoft Inc. All rights reserved. 14

15. Data Security
There are a myriad of data security
issues to consider:
• Data encryption
• Data privacy
• Access control
• Secure storage
• Secure deletion
• And many more…
© Ambysoft Inc. All rights reserved. 15

16. Parting Thoughts
© Ambysoft Inc. All rights reserved. 16

17. © Ambysoft Inc. All rights reserved. 17
The increasing pace of change,
increasing complexity,
and increasing volume of data
demands nothing less than
complete data agility

18. Thank You!
© Ambysoft Inc. All rights reserved. 18
• scott@scottambler.com
• Twitter: @scottwambler
• linkedin.com/in/sambler/
Consulting Methodologist
Ambysoft.com
Thought Leader
AgileData.org
Thought Leader
AgileModeling.com

19. The Agile Data Site
AgileData.org
Key articles:
• Data Technical Debt: How to Address Quality Problems in Data Sources
• Database Techniques Stack
• The Agile Data Architect
© Ambysoft Inc. All rights reserved. 19

20. Look-Ahead Data Analysis: Agile
© Ambysoft Inc. All rights reserved. 20
Scenario: You want to implement three question stories in sprint #9
You need to:
• Have a definition of ready (DoR) indicating the amount of data analysis work required
• Guesstimate the amount of data analysis required for each one, and then perform the analysis
sufficiently before sprint #9
• Have sufficient capacity to perform look-ahead data analysis
• Interleave data analysis for other sprints into the work of the people performing it
Note: Staffing your team with specialists will exacerbate work scheduling challenges. Consider
generalizing specialists (AgileModeling.com/essays/generalizingSpecialists.htm) instead.

21. Look-Ahead Data Analysis: Continuous Delivery
© Ambysoft Inc. All rights reserved. 21
Development – QS 9a
Development – QS 9b
Development – QS 9c
Scenario: You want to implement the same three question stories
• You are not constrained by organizing the work into sprints
• Development = Data analysis + other implementation work
• Work can be brought into the team as capacity permits
• Value is delivered when it is available
• Average cycle time to deliver stories is shorter
• Staffing your team with generalizing specialists may enable you to swarm around data analysis
work and shorten the development time of a question story

22. Agile Data Ways of Thinking (WoT)
© Ambysoft Inc. All rights reserved. 22

23. Agile Data Ways of Thinking (WoT)
AgileData.org/essays/philosophies.html
Look Beyond Data
Collaborate Closely
Be Quality Infected
Embrace Evolution
Be Enterprise Aware
Fit-For-Purpose
Everyone Agile
© Ambysoft Inc. All rights reserved. 23