The document discusses DataOps as an evolution of Agile and DevOps, aimed at improving collaboration between data teams and the broader business to enhance data utilization for actionable insights. It outlines the benefits and principles of DataOps, emphasizing the importance of processes, people, culture, and technology in its implementation. Additionally, it provides a case study of Facebook's DataOps journey and highlights essential techniques for successful DataOps delivery.

1. Starting Your Modern
DataOps Journey
Lucas Stone
Solutions Engineer
01.12.2020

2. The DataOps Story
Pre
2000
2007
Waterfall
Linear approach
Good for projects where end state is well-defined
(e.g. physical infrastructure)
Not as good where product is continually
changing / developing (e.g. software)
Agile
Began with “Agile Manifesto”
Designed for production of software products
Response to how rapid business
requirements could change
Emphasised iteration
DevOps
Even given the rise of Agile, dev and
ops teams remained siloed
DevOps aimed to bring them together
Set of practices to release high quality
code and faster
DataOps
Bring the DevOps approach to data function
Align data science / management and
operations teams
Ensures the business can fully leverage
data and convert into actionable insight
2000 2014

3. Where is DataOps in the “Hype Cycle”?

4. DataOps Search Trends: Past 5 Years

5. DataOps
DATA PRODUCTION VALUE
DEVELOPMENT
Quality
Data + Operations
Ensure that data value is delivered to business as soon as possible
Intersection of the Value and Innovation Pipelines
Value
Pipeline
Innovation Pipeline
IDEA
Quality

6. DataOps Misconceptions
A technology
Although there are a set of technologies that
are commonly used to support its
implementation within an organisation
Restricted
Either to 1) “big data” – the scale of data and
complexity doesn’t preclude benefits 2) only
advanced data science applications (e.g. ML)
More than just
DevOps for Data!
It brings together 3 distinct elements:
Agile development, DevOps, and
Statistical Process Control (Lean)
A methodology
It brings together a number of principles
and practices around the way an
organization manages and processes data
DataOps is… DataOps is not…

7. DataOps – What are the benefits?
Faster
Deployment & Feedback
Consumers of data analytics will get what
they need faster and be able to feedback
more often, creating virtuous circle with
business requirements as the starting point
Happier Colleagues
Introduction of DataOps will mean those involved in
the process are more able to quickly see the positive
impact of their work leading to more engaged and
productive teams
Higher Data Quality
Increased automation (particularly of testing) and
standardised processes will lead to high Data
Quality, which will in turn lead to better insights
generated from Machine Learning models
Collaboration
DataOps promotes collaboration,
communication, and coordination between
teams that may otherwise remained siloed

8. Which companies use DataOps?
Please see Qubole’s Creating a Data-Driven Enterprise with DataOps ebook for further
information on how each of these organisations implements DataOps

9. Case Study: Facebook & Apache Hive
Stage 2
Created a Hadoop data
lake and developed Hive
to make it more
accessible – data team
evolved from a service
team to building self-
service platforms for data
extraction
Stage 4
Developed Uis that were
easy for business users to
understand and use to
independently extract data
- data becomes fully
democratised
Stage 1
No structure around data
requests, rather the
business would request on
an ad-hoc basis – the data
function would act as a
service
Stage 3
Combined metadata
services with Hive
allowing users to look at
data and metadata –
however, data still not
accessible to non-tech
users

10. What is required to begin implementing DataOps?
Processes
It is then important to establish clear processes
including who is “RACI”. Those responsible should
receive appropriate training. Measuring process
effectiveness with appropriate KPIs is also crucial
People & Culture
The foundation for introducing DataOps lies firstly
with buy-in from the key stakeholder groups
particularly the business so that business
requirements are understood
Technologies
Once the correct culture and processes have been
established, an organisation can introduce tooling to
support related activities, notably automation, testing,
and orchestration
2
1
3

11. People: Stakeholders Groups
Data Consumers
Those who will use data to perform
analysis and extract insights to then
deliver to those in the business who
can use these to drive value
Data Suppliers
Those managing the integrity of
Authoritative Data Sources to ensure
data quality and availability
Data Preparers
Those who build data pipelines
linking one source to another as well
as managing its transformation into a
usable format for Data Consumers
Business
Other parts of the organisation would
not use DataOps – rather they rely
on and benefit from better outputs in
terms of insights / BI / analytics and
convey Business Requirements
Data Ops builds two crucial bridges, firstly between the business and technology
functions, secondly within the data function itself

12. People: Ingraining a DataOps Culture
Push from the Top
Cultural changes must be endorsed by
senior management both within and
outside of the data function before being
pushed down to individual teams
Embrace the Process
Acknowledge that change won’t happen
over night and that improvements will be
incremental – allow a realistic timeframe for
the process of implementing data ops
Remove silos
Breaking down organisational barriers between
Data Suppliers, Preparers, Consumers, and the
Business will be crucial to the smooth flow of
data to those making decisions
Emphasise Data
Data should be front and centre of
strategic decision making for DataOps to
realise its full potential – this should be
embedded as a company value
Invest in Tools
Carefully selecting a complementary set of
technologies underpinning the implementation of
DataOps is essential as will be providing the
relevant training to upskill your teams
1
2
3
4
5

13. Processes: Building a “Data Supply Chain”
Data
Suppliers
Data
Preparers
Data
Consumers
The
Business
Source Owner, DBA, Infrastructure and Ops Personnel,
Application Admins + Developers
Data Engineers, Data Architects, Data Stewards,
Integration Architects + Developers, Data Modelers
Machine Learning Model Developers, Data Scientists
HR, Finance, Strategy, Operations etc.
Data Product Managers
Business Analysts
Data Security Teams
Data Privacy Officers

14. Technology: Agile, Collaboration,
Automation, Infrastructure as Code
Agile
Small but frequent deliveries
of new features
Constant feedback loop
between the business and tech
Version Control to decrease risk
and increase productivity
Job / issue tracking to ensure
even minor feedback captured
Collaboration
Automation
Continuous Integration,
Deployment, Delivery
Automate testing and speed up
getting code into production
Infrastructure as Code
Manage IT infrastructure
using code
Make changes to
existing infrastructure
much more easily

15. CloverDX & DataOps
Increased deployment frequency
Package, share, and reuse any
functionality you design
Automated testing
Incorporate data quality tests and build in error
handling to your data pipelines
Consistent metadata and version control
CloverDX is easy to integrate with most VC tools,
metadata can easily be tracked and visualized
Monitoring
The CloverDX server has a monitoring suite that
can be applied to individual jobs or whole
business processes
Collaboration across all stakeholders
CloverDX’s visual design allows technical and
non-technical users to “speak the same language”
Gartner identified 5 “key techniques” that will support with the delivery of DataOps –
Clover can support “Data Preparers” with each

16. Upcoming Webinar
Code Management with
Version Control in CloverDX
December 8th
11am EST / 4pm GMT / 5pm CET
Register
Q&A