This document discusses democratizing data and including it as part of the DevOps toolchain. It argues that data should be made available as a service and provisioned in a secure and compliant manner to empower developers. The document recommends using a DataOps approach and platform like Delphix to virtualize data from various sources and provision production-like test data for developers in an automated way. This helps overcome issues like long test data provisioning times and lack of access to production data, improving the delivery pipeline. Case studies of insurance and banking clients adopting this approach are also presented.

1. My Code, My Environment,
and Yes, My Data
Empowering the Developer with Data
Sanjeev Sharma
VP, Data Modernization

2. © 2019 All Rights Reserved. 2
$ whoami
• 20+ Years in Software Development
and Delivery
• Past: IBM Distinguished Engineer and
CTO for DevOps Adoption (2013 –
2018)
• Author of two DevOps books:
• DevOps For Dummies:
https://ibm.biz/BdsPMX
• The DevOps Adoption Playbook:
http://amzn.to/2hH7rt2
• Blog: https://sdarchitect.blog
• @sd_architect

3. © 2019 All Rights Reserved. 3
1874 - Solid Wood 1947 - Laminated Wood 1968 – Steel
Racquet
1993 – Graphite
Racquet
2008 – Aerodynamic
Racquet
Innovation drives the Democratization of Technology

4. © 2019 All Rights Reserved. 4
DevOps: Democratizing the CI/CD Pipeline
• Democratize the Capabilities across the CI/CD pipeline as services consumable by Practitioners
• ‘Shift-Left’ availability of capabilities across delivery pipeline
• Raise the abstraction of the role of Specialists to deliver their capabilities as services
• Enable better collaboration and communication across functional areas
• More effective controls and governance
Continuous Integration Continuous Delivery
Shift Left Test
Shift Left Ops
Culture
Development SCM Build
Package
Repo
Deploy Testing Staging Production FeedbackPlanning Manage

5. © 2019 All Rights Reserved. 5
Automate Infrastructure
Automate Data
Automate SDLC
Innovate
5
Democratize Data
Innovate
Democratize SDLC
Democratize Infrastructure
Building a CI/CD Pipeline

6. © 2019 All Rights Reserved. 6
6
Democratizing Data
DATA

7. © 2019 All Rights Reserved. 7
7
Democratizing Data with Delphix

8. © 2019 All Rights Reserved. 8
8
Democratizing Data with Delphix
Become Data Source Agnostic
Make Data Available Self Service
Manage Data Like Code
Mitigate Data Privacy &
Compliance Risks
Include Data Management in the
DevOps toolchain

9. Democratizing DevOps
Capabilities

10. © 2019 All Rights Reserved. 10
Overcoming Cultural Inertia
1. Organizations are not monolithic or homogeneous, with disparate
processes, technology stacks, org structures and culture
2. Teams lack trust and collaboration across Functional Siloes. They are not
organized for Shared Responsibility
3. Governance Overhead and Manual Approval Processes are major
impediments to Agility and Speed
DevOps Works fine in co-located ‘2-pizza’ Teams, but not at
Enterprise Scale

11. © 2019 All Rights Reserved. 11
If you are doing 2-
week Sprints, but it
takes 3-weeks to
get Test Datasets,
how long are your
Sprints?
Democratizing Test Data

12. © 2019 All Rights Reserved. 12
It is not possible to
patch the software of
a missile AFTER it
has been launched
Quality is dependent on testing with Production-like Data

13. © 2019 All Rights Reserved. 13
Overcoming Cultural Inertia
• Deliver Data as a Service
• Include DBAs in your Squads and
Tribes
• Make Data Accessible and Agile, in
a secure and compliant manner
Everyone is responsible for delivering
Business Value to Users

14. Democratizing Data in
the CI/CD pipeline with
DataOps

15. © 2019 All Rights Reserved. 15
Barriers to Delivering Enterprise Data with Speed and Scale
15
• Massive and continuously
growing data volumes
• Provisioning cost and delays
• Growing data risk
management and regulatory
compliance burden
DATA HAS INERTIA
• Resides in heterogeneous
sources
• Scattered across on-
premises and public/hybrid
clouds
• Managed through custom,
complex and/or manual
workflows
DATA IS COMPLEX
• Locked in packaged apps or
zero-touch production
systems
• Incompatible toolchains and
workflows
• Organizational silos and
cultural inertia
DATA IS SILOED

16. © 2019 All Rights Reserved. 16
A collaborative data management practice
focused on improving the communication,
integration, and automation of data flows
between data managers and data consumers
across an organization.
- Gartner
DataOps

17. © 2019 All Rights Reserved. 17
DataOps for Secure and Compliant Test Data via Self Service
17
Applications
Files
Databases
DATA POD
On-Prem
Cloud
CONNECT
APIs
VIRTUALIZE
Sync
Compress
Provision
SECURE
Compliance
Policy
Masking
MANAGE
Distribute
Audit & Report
Manage
D B A
Developer
QA
Analyst
Data
Scientist
On-Prem Any Server Private Cloud Public Cloud

18. © 2019 All Rights Reserved. 18
Delphix DataOps Platform plugs the toolchain gap
Design Develop Test
Deploy
Source Code
Management
Continuous
Integration
Environment
Configuration
Release
Automation
Test
Automation
Test Data Management
DistributeMask

19. © 2019 All Rights Reserved. 19
INTEGRATION TEST SYSTEM TEST PRODUCTION
CODE
Virtual DB
CODE
Virtual DB
CODE
Virtual DB Production
CODE
DEV
DEV
DEV
Secure Data
Deployment
Code and
Infrastructure
Deployment
Provision/Refresh
Data
Sync and Mask
…
Trigger BuildPush
Delphix Plugin
Secure Test Data Management in the Delivery Pipeline
19
Provision/Refresh
Data

20. © 2019 All Rights Reserved. 20
DataOps for Enterprise DevOps Adoption
INTEGRATION TEST
SYSTEM TEST

21. © 2019 All Rights Reserved. 21
DataOps Adoption Scenarios
• Insurance Client:
– Had 3 Test environments that were shared by 35 project teams. The three environments were hosted on
static infrastructure with limited compute and storage
– Went to 35 test environments (one for each team) with Virtualized Data provisioned by Delphix, hosted on
the same static infrastructure
• Global Bank:
– Development teams had dynamic Test Environments. However, they has limited access. Production-like
Test Data was not available to Developers. Database refresh took 6-7 days per request
– Today all Databases (50+) are provisioned via Delphix into 3,000+ Dev and Test environments. Every
development stream (Git branch) has its own Database
– Data provisioning governed via automated policy based ‘guardrails’
• Hotel Chain:
– Data was not a part of DevOps toolchain. Database provisioning was done manually and only subsets of
data was made available to Developers and Testers
– Data is a core service of the DevOps toolchain. Database provisioning is part of the Jenkins pipeline and
fully available to developers and testers via self-service

22. Value Stream Mapping
the CI/CD Pipeline

23. © 2019 All Rights Reserved. 23
Making the Delivery Pipeline Lean
Idea/Feature/Bug Fix/
Enhancement
Production
Development Build QA SIT UAT Prod
PMO
Requirements/
Analyst
Developer
CustomersLine of Business
Build
Engineer
QA Team Integration Tester User/Tester Operations
Artifact Repository
Deployment Engineer
Release Management
Code Repository
Deploy
Get Feedback
Infrastructure as Code/
Cloud Patterns
Feedback
Customer or
Customer Surrogate
Data
Tasks
Artifacts
Value Stream Mapping to
Identify:
• Waste
• Wait
• Rework
Video : DevOps Value
Stream Mapping

24. © 2019 All Rights Reserved. 24
24
• Review the current state
o Business goals, IT goals, current
initiatives
o Requirements
o Environments
o Repositories
o Data Sources /Data Architecture
o Roles / Organization
o Metrics
o Other
• Prioritize Waste, Wait states and
Rework
• Create a first pass at a roadmap to
address inefficiencies
The whiteboard
Next Step: DevOps Value Stream Mapping Workshop