Data governance teams attempt to apply manual control at various points for consistency and quality of the data. By thinking of our machine learning data pipelines as compilers that convert data into executable functions and leveraging data version control, data governance and engineering teams can engineer the data together, filing bugs against data versions, applying quality control checks to the data compilers, and other activities. This talk illustrates how innovations are poised to drive process and cultural changes to data governance, leading to order-of-magnitude improvements.

1. The Rise of DataOps
From the ashes of Data Governance
Ryan Gross

2. 2 Pariveda Solutions, Inc. Confidential & Proprietary.
Fear driven
Governance
Does not work

3. © Pariveda Solutions. Confidential & Proprietary.3
Governance is often the weakest pillar in a Modern Data
Enterprise, holding back the realization of value
No organization-wide plan or
vision guiding efforts and
consequently, ROI is unknown
or poorly justified
Current state process defined,
focused on supporting current
state efforts and baseline
financial estimates of
projected costs and initial
value forecasts
Vision of future state defined,
focused on incremental
improvements inclusive of
estimates that consider
operations
Vision and guiding principles,
cost and value considerations
together drive all technology
efforts to reach future state
Data Warehouse –
Analytics are descriptive and
immature
Data Swamp –
Descriptive analytics,
reporting, and visualization
Data Lake –
Experimentation and
beginning of predictive
analytics
Data Platform –
Optimization
for increased scale of use-
cases
Lack of well-defined data
stewardship and management
Data stewardship defined,
manage requests and issues
reactively
Proactive data stewardship
and management as code
Data stewardship and
management is
operationalized
Uncontrolled Reactive Proactive Resilient
Governance
Value
Platform
Most organization’s
governance efforts are
stuck in a loop from
reactive  uncontrolled
Attempt to apply legacy,
manual governance
practices to address issues
Governance efforts fail to
gain traction, leading to a
return to uncontrolled state

4. Data Governance Guideposts haven’t
changed
Data Quality Data Accessibility Data SecurityCompliance Availability

5. Traditional Data Governance is heavily dependent on human
intervention to manage, creating business decision bottlenecks
Chief Data Officer
Key Business Unit LeadsLead Data Stewards
Data Governance Council
Lead Data Stewards
Data Project Groups
Data Custodians
Data Stewards
Issues
Guidance
Initiatives
Initiatives
Traditional Approach in 2014
Recreated: http://datagovernanceaus.com.au/data-governance-what-is-it/
Lead Data Stewards
Chief Data Officer
Data Governance Council
Lead Data Stewards Key Business Unit Leads
Data Project Groups
Data Custodians
Data Stewards
Issues
Guidance
Initiatives
Initiatives
Traditional Approach in 2020
Data Governance Headcount, Meetings
& Quality Over Time
Data Quality

6. © Pariveda Solutions. Confidential & Proprietary.6
To fix Data
Governance,
we need to
change our
Mental Model

7. © Pariveda Solutions. Confidential & Proprietary.7
In other words, maybe
we need to reframe the
problem…

8. 8 Pariveda Solutions, Inc. Confidential & Proprietary.
With
Machine
Learning,
Data
Writes the
Code

9. Once Upon a Time…

10. © Pariveda Solutions. Confidential & Proprietary.10
The ability to
compile a set of
input code to
executable
outputs
Version control
systems to
keep track of
the input code
Two core innovations created the discipline of software engineering:
+

11. 11 Pariveda Solutions, Inc. Confidential & Proprietary.
1. Higher level languages (scala, python)
2. Automated Unit & Integration Tests
3. Static Analysis
4. Continuous Integration
5. Refactoring
All modern software engineering builds on
these fundamental constructs:

12. 12 Pariveda Solutions, Inc. Confidential & Proprietary.
The ability to compile input code
to executable outputs
Version control systems to keep
track of the input code
DevOps: Bring Compilers and Source Control to Infrastructure

13. 13 Pariveda Solutions, Inc. Confidential & Proprietary.
Let’s Look at DevOps
The same process of increasing
improvements has repeated itself, with
cloud native approaches and
continuous deployment becoming the
norm

14. 14 Pariveda Solutions, Inc. Confidential & Proprietary.
So getting back to
data…

15. 15 Pariveda Solutions, Inc. Confidential & Proprietary.
The ability to compile input code
to executable outputs
Version control systems to keep
track of the input code
DataOps: Bring Compilers and Source Control to the world of Data
If this is the
source
code…
…and this is the
resulting
operation
… then the pipeline
is the compiler

16. We still don’t really
understand how
data writes code
• This is why we have data
scientists experiment to
figure out the logic
• Later data engineers come
in later to build the
optimizers

17. © Pariveda Solutions. Confidential & Proprietary.17
Use Software Development Best Practices to manage what we
don’t fully understand
Innovation Pipelines provide a means for companies to
deliver value while automating testing and monitoring
quality.Source: The DataOps Cookbook

18. Define
Everything as
Code
to reduce risk
increase
quality, and
build trust
Access&Privacy
Defines the requirements
to access the data
outputs produced by
each pipeline stage
Dependencies
Defines all libraries that this
component depends on to
execute / test without actually
including the libraries in SCM.
PipelineCode
The functional code for
this component. This
code should be separated
out so that pure business
logic lives in a library &
platform specific code
calls the lib.
CloudEnvironments
CloudFormation Templates define the
infrastructure that will be created to
deploy this component. Data cloning
or test data management provide the
datasets that enable testing
LogicTests
Test code to
ensure proper
function of the
business logic.
These capture
edge cases that
may not be in
the real data
DeployPipeline
Jenkins File definitions
include the pipeline of build
steps required to
successfully get this
component into production.
DataTests
Test code that
ensures input
data is correct
and outputs are
properly
configured.
By ensuring that every
aspect of developing
analytics solutions is
captured and tracked as
code, it becomes much
more clear which change
introduced a failure
Everything
as Code

19. © Pariveda Solutions. Confidential & Proprietary.19
Change Management
• Cloud Vendor Selection
• Resilient Solution Architecture
• Integrated Enterprise Solution
• Infrastructure Automation
• DevOps Process Definition & Change Plan
• Solution Evolution & Cost Optimization
Make it all discoverable automatically
Automate documentation and cataloging of data so data
producers don’t need to curate and document where and what
the data is
Empower self-service and self discovery
Make data discoverable and transparent so data consumers
can access and take advantage of it without having to talk to
the data producers.
Rapidly realize deep business insights
Focus is on the value of the data and how it can be used to drive timely
actionable insights at the speed of business not IT.
Remove Inefficiency and Thrash
Manual manipulation and management of data is
notoriously time consuming and inefficient
Onboard new sources in a day
New data sources can historically take weeks or
months, use the principles of DevOps to deploy
new sources from POC to Production in days or
hours
Write no incremental code
Bring new data sources in through configuration and automated
data introspection. Build flexible platforms that don’t need
expensive and error prone custom development to interpret new
data. Save custom development for algorithms that add
demonstrable business value.
DATA OPS
AS
GOVERNANCE
What if you could…
All the while providing data compliance and security.

20. © Pariveda Solutions. Confidential & Proprietary.20
New technology is available support DataOps Governance
Data Quality Data Discovery Data
Security
Data
Availability
Visual data wrangling
Pipeline Testing AWS LakeFormation
Compliance
ML Data Catalogs Compliance Catalogs
Rapid Exploration
AWS Athena
FederatedQuery
Privacy Tools
Observability Tools Access Management
AmazonKeyManagementServiceData Versioning
Encryption

21. Ingest Diff Model Enhance Transform
Production
Data Sources
Production Data Platform
Data Lake
Data Pipeline
Banking Data
Bloomberg, Dow Jones
Test Data Development Data Platform
Data Lake
Data Pipeline
Ingest Diff Model Enhance TransformDiff Model Enhance Transform
Raw Modeled Enhanced Products Data Mart
Test Banking Data
Test Bloomberg, Dow
Jones Data
test
test
test
test
test
Define Access Policies, Test end-to-end
Data Mart
Design right-to-left
Test Data for
NYSE
Raw Modeled Enhanced Products
Ingest Prototype
BI
Real time
AI
BI
Real time
AI
Dash-
boards
Dash-
boards

22. © Pariveda Solutions. Confidential & Proprietary.22
Add tests to control what you don’t know

23. That’s great…
…now how do I get started?

24. © Pariveda Solutions. Confidential & Proprietary.24
How to ramp up of data operations roles & processes
Leverage the data platform to capture
governance metrics and enforce constraints
Over time, the data management team will work more closely with
data science to build in governance during the experimentation loop
Same team applies governance once
feasible, valuable prototypes exist
People will be more likely to participate in the governance
process if they have a hand in the value being generated
Build data management team with a focus
on enabling use cases
The more people contribute to the initial Data Operations
design, the more they will feel invested in seeing it succeed
Reporting on the value of data insights and risk
avoidance drives continued motivation
Data Operations is a continually ongoing activity, people need to be
thoroughly invested to maintain participation
1
2
3
4
The gap between experiment start and
time necessary to learn the reasoning,
processes, tooling, and rules for the
governance aspects of data operations
OPPORTUNITIES PROTOTYPES
METRICSINSIGHTS
IDENTIFY,
ASSESS AND
PRIORITIZE
EXPERIMENT
AND LEARN
BUILD, TEST
AND RUN

25. © Pariveda Solutions. Confidential & Proprietary.25
A DataOps-enabled platform will support the entire Data Science Lifecycle
Data Catalog Data Lake Storage Data Pipelines
Experiment & Learn Deploy, Test, and Run

26. As you mature, you will be able to take on more complexity
Platform
Processes
Modern
ToolingFoundation
Data Forge
Framework
Data Ops
Roles/People
Organization
Technology and
Infrastructure
HigherlevelsofDataMaturity
Differentiated Business Value driven from your Data
Modern Data
Management

27. © Pariveda Solutions. Confidential & Proprietary.27
Manage trust in addition to minimizing Data Issues

28. 28 Pariveda Solutions, Inc. Confidential & Proprietary.
1. Data environment management
2. Access & Privacy as Code
3. Test management
4. Continuous Deployment & Compliance
A similar process will play out with innovations
built on the foundation of DataOps