Here’s the thing about data: it’s sticky, often rigid, and rarely feels agile. Yes, there are patterns that help increase the agility of data. Ruby on Rails, for example, has data migrations that not just allow but actively encourage incremental schema design. However all too often we hear about relational databases becoming a defacto API between subsystems, and thus resistant to change. It’s not just relational databases. Even supposedly unstructured data stored as key value pairs can be difficult to change if every piece of code that uses the data has duplicated logic to manage the semantic meaning of the data. Further, in-database logic such as rules, stored procedures, or user defined functions can be remarkably difficult to unit test. Finally, there are the often strict data governance requirements that necessitate keeping tight authorization control. Application developers have a wealth of tools and practices available to support incremental delivery. System administrators have DevOps tools and practices to support repeatable, automated operations. Where are the equivalents for data-centric work? Bringing agility to your data strategy may feel like an impossible goal, but it is possible. In this talk we consider the various ways in which data can impede agility, and how to make data strategies more agile-friendly.

1. © Copyright 2018 Pivotal Software, Inc. All rights Reserved. Version 1.0
Elisabeth Hendrickson
deliver:Agile
May 2, 2018
Austin, TX
Agility for Data

2. What Does Agility Feel Like?

3. Deliver a continuous stream of value
at a sustainable pace
while adapting to the changing
needs of the business
Is it outcomes?

4. Eliminate friction
Or … is it an experience?
between idea and delivery.
! "

5. What does friction look like?
#! "

6. Data is Messy

7. Expect the unexpected
$null

8. Test data is bigand hard to manage.
%

9. &
In-database logic (e.g. a stored procedure)
(almost always) necessitates
slow end-to-end tests

10. &
Re-setting to a known state can
be slow, difficult, and
lead to test flakiness.
'

11. Provisioning test environments
is often a ticket-driven activity

12. Schemas are often de facto APIs

13. The change control board must
approve schema changes
when the schema is the de facto
API between sub-systems.

14. The prime directive of data is keep it safe

15. Confession: everything
I know about iterative
database development,
I learned from Rails.

16. Schema migrations
http://edgeguides.rubyonrails.org/active_record_migrations.html
class AddLikestoAnimals < ActiveRecord::Migration[5.0]
def up
add_column :animals, :likes, :string
end
def down
remove_column :animals, :likes, :string
end
end

17. Test fixtures
http://guides.rubyonrails.org/testing.html#the-low-down-on-fixtures
duck:
name: Ducky
species: rubber duck
likes: floating, pairing
ellie:
name: Ellie
species: dog, rescue mix
likes: sunbeams, sticks

18. Test data factories
http://21clouds.com/ruby-on-rails-tutorial-better-testing-with-factorygirl/
FactoryGirl.define do
factory :duck do
name ”Mrs. Mallard"
species ”Duck"
likes ”Paddling"
end
end
# ...elsewhere in the code...
ducky = create(:duck)

19. It’s all code

20. So What Do We Do?

21. Refactor the monolith
App 3
App 1 App 2
App 4
SP1
SP2
SP3
SP
serviceservice service
App 1 App 2 App 3 App 4
Independent services
Own their data
Act as an API

22. Invest in carefully curated test data
and generators versioned with the code.

23. Invest in schema migrations
to support incremental database
development (even if your framework
doesn’t include them).

24. Eliminate the need for tickets with
automation to enable self-service
(enabled by modern PaaS / cloud offerings).

25. Embrace data specialists
as part of your balanced team.

26. But What About… ?

27. General guideline:
Adapt the principles and practices for
agility in a stateless world
and apply them in a data context.

28. Mandated constraints? 5 Whys.
Manual steps? Automate.
Silos? Balanced Team.
Brittle? Abstract.
Big bang events? Increments.
Late-breaking surprises? Fail fast.
! "

29. Transforming How The World Builds Software
© Copyright 2017 Pivotal Software, Inc. All rights Reserved.