SlideShare a Scribd company logo
© Stonebranch 2022. All rights reserved.
ORCHESTRATE
the Flow of Data Across
Data Pipelines
May 3, 2022
Ravi Murugesan
Sr. Solution
Engineer
Scott Davis
Global Vice President
2
© Stonebranch 2022. All rights reserved.
DevOps Orchestration Layer
01
What is a Data Pipeline
02
How to Orchestrate a Data Pipeline
03
Data Pipeline Orchestration Demo
04
Questions and Answers
05
Agenda
© Stonebranch 2022. All rights reserved.
About Data Pipelines
3
Scott Davis
Global Vice President
4
© Stonebranch 2022. All rights reserved.
Vendor Landscape for DataOps – From Gartner
Orchestrators
Specialists
Portfolio Cloud Service Providers Servware (Services & Software) System Integrators
Integration Cataloging Governance
MDM Analytics-Ready Enterprise Data Management
Industrial Data Data Quality Observability
Continuous Delivery Accelerators Privacy & Access Control
* Based on “Gartner Data and Analytics Essentials: DataOps,” by Robert Robert Thanaraj
5
© Stonebranch 2022. All rights reserved.
Data Pipeline: Simple View
6
© Stonebranch 2022. All rights reserved.
Software & Tools By Stage
Dashboards
Looker, Tableau, Qlik, Power
BI, SAP BusinessObjects
Embedded Analytics
Sisense, Looker, Cube.js
Augmented Analytics
Throughspot, Outlier,
Anodot, Sisu
App Frameworks
Plotly Dash, Streamlit
Custom Apps
SMS Messages / Emails
Data Science &
Machine Learning
Databricks, SAS, MathWork,
Domino, Dataiku, DataRobot,
TIBCO Software, Spark,
RapidMiner, H2O.AI, AWS, GCP
AI, Azure ML, IBM Watson
Studio, Cloudera, Alteryx,
TensorFlow, Anaconda
Data Lake
Databricks Delta Lake,
Iceberg, Hudi, Hive Acid
Data Lake
within Cloud Storage
AWS S3, Google Cloud
Storage, HDFS,
Azure Data Lake Store
Data Warehouse
Snowflake, BigQuery, Spark,
AWS Redshift, Qubole, SAP
BW, SAP DWC, Oracle ADW,
Hive, Cloudera (for Hadoop)
ETL
(Extract, Transform, Load)
Informatica, IBM, SAP Data
Services, Oracle OWB, SAS,
Talend, AWS Glue, Azure Data
Factory, Pentaho, GCP Data
Fusion
Stream Data Processing
ELT
Kafka, Flink, Storm, GCP
Pub/Sub
Applications / ERP
Oracle, Salesforce, SAP,
ServiceNow
IoT Devices / Sensors
Stream Data
Website & Mobile Apps
Stream Data, Online
Transaction
Cloud Storage
AWS S3, Google Cloud
Storage, Azure
Data Sources Data Integration & Ingestion Data Store Analyze / Computation Delivery
How Do Enterprises Orchestrate Today?
7
© Stonebranch 2022. All rights reserved.
Common Ways to
Connect Data Tools
Within the Pipeline
Point-to-Point
Integrations
Custom
Scripts
Don’t Connect
(Manual Movement)
How Do Enterprises Orchestrate Today?
8
© Stonebranch 2022. All rights reserved.
Common Ways to
Connect Data Tools
Within the Pipeline
Point-to-Point
Integrations
Custom
Scripts
Don’t Connect
(Manual Movement)
Benefits of Proper
Orchestration Solutions
Centralized
View
Root-Cause
Issues
Proactive
Support
Achieve
Scale
Automation Pain Points
Common Ways to
Connect Data Tools
Within the Pipeline
Point-to-Point
Integrations
Custom
Scripts
Don’t Connect
(Manual Movement)
How Do Enterprises Orchestrate Today?
9
© Stonebranch 2022. All rights reserved.
Benefits of Proper
Orchestration Solutions
Centralized
View
Root-Cause
Issues
Proactive
Support
Achieve
Scale
In-Built
Schedulers
Open-Source
Schedulers
Cloud
Schedulers
Legacy On-Prem
Focused Schedulers
Can’t schedule jobs
in other tools
Often batch- or time-
based automation
Focus on their
own ecosystems
Can’t automate jobs in both
on-prem and cloud systems,
i.e., no hybrid IT automation
Data Pipeline
Orchestration
© Stonebranch 2022. All rights reserved. 10
11
© Stonebranch 2022. All rights reserved.
Data Pipeline Orchestration How to accomplish the real-time automation
and file transfers needed to manage the
entire data pipeline.
Data Pipeline Orchestration
Orchestration
How to accomplish the real-time automation
and file transfers needed to manage the
entire data pipeline.
• Centrally schedule and
orchestrate automated processes within
each tool along the entire data pipeline
• Use APIs or Agents to control the various
tools used within each stage
12
© Stonebranch 2022. All rights reserved.
Data Pipeline Orchestration
Orchestration
How to accomplish the real-time automation
and file transfers needed to manage the
entire data pipeline.
• Centrally schedule and
orchestrate automated processes within
each tool along the entire data pipeline
• Use APIs or Agents to control the various
tools used within each stage
What you achieve with this approach:
• Observability of the logs and data for
governance and security
• DataOps lifecycle management (Dev-Test-
Prod) - including simulations
• Centralized control and visibility with
visual workflows
• Quickly root-cause issues with proactive
alerts when something fails
13
© Stonebranch 2022. All rights reserved.
Data Pipeline Orchestration
Orchestration
How to accomplish the real-time automation
and file transfers needed to manage the
entire data pipeline.
• Centrally schedule and
orchestrate automated processes within
each tool along the entire data pipeline
• Use APIs or Agents to control the various
tools used within each stage
What you achieve with this approach:
• Observability of the logs and data for
governance and security
• DataOps lifecycle management (Dev-Test-
Prod) - including simulations
• Centralized control and visibility with
visual workflows
• Quickly root-cause issues with proactive
alerts when something fails
14
© Stonebranch 2022. All rights reserved.
Orchestration
15
© Stonebranch 2022. All rights reserved.
Driven by SOAP
Cloud DevOps ERP/Apps
IaaS
UAC
SOAP
META-ORCHESTRATION
16
© Stonebranch 2022. All rights reserved.
Self-Service
Automation
Centralized collaboration
platform for data,
developers, and
operations
IT ops teams gain
operational visibility
Data teams approve and
trigger automated workflows
& pipelines from common
business applications
Data Pipeline
Putting the Ops in DataOps
17
© Stonebranch 2022. All rights reserved.
For Enterprises Ready for the Next Level of Maturity
Develop/
Orchestrate
Test /
Simulate
Production
/ Deploy
Continuous Improvement Continuous Deployment
Development Controller Production Controller
Develop/
Orchestrate
Test /
Simulate
Production
/ Deploy
Continuous Improvement Continuous Deployment
Development Controller Production Controller
Putting the Ops in DataOps
18
© Stonebranch 2022. All rights reserved.
For Enterprises Ready for the Next Level of Maturity
Web
GUI
As
Code
Via in-built
capabilities
Promotion
Options
Via third-party
repositories like
GitHub
Data Pipeline Orchestration Demo
Ravi Murugesan
Sr. Solution Engineer
© Stonebranch 2022. All rights reserved. 19
© Stonebranch 2022. All rights reserved. 20
Demonstration
Update Visual Dashboard from Multiple Data Sources (both on-prem and cloud-based)
Live orchestration of a data pipeline,
including
• Sources (cloud, on-prem, apps)
• Ingestion, transformation (Informatica)
• Stores (Azure blob, Snowflake)
• Delivery (Tableau)
One of the Largest Global Food & Beverage Manufacturers in the World
Customer Use Case
21
Customer Use Case: Overview
One of the Largest Global Food & Beverage Manufacturers in the World
Evolution & Goal
• Goal: Orchestrate the full pipeline end-to-end
• Objective: Identify a platform that could connect all their critical data tools
Overall Strategy
• On-prem to cloud digital transformation
• Implemented an enterprise analytics data management environment
• Hub-and-spoke model to help keep regional resource groups and services segregated
• Approved services are first developed and deployed at the hub level, with further spoke
deployment via containers
Original Approach
• Their data pipeline for the enterprise data management environment with Azure Data Factory
• Azure Data Factory worked well in an Azure environment
• It served as an entry point for the project
• The Challenge: Data Factory did not integrate with their full stack of solutions used along the
data pipeline
22
© Stonebranch 2022. All rights reserved.
Data Pipeline Orchestration
One of the Largest Global Food & Beverage Manufacturers in the World
Achieving Their Goal
• Secure and robust file transfer
• DataOps: define pipelines as code and gain lifecycle
management (test/dev/prod) capabilities
• Integrate diverse data pipelines that are built using
various cloud-based and on-prem services and tools
• For operations: visibility into the process, improve SLAs,
real-time monitoring, alerting
• Unified view to design and orchestrate workflows
across multiple cloud and on-prem applications
Orchestration
Databases
23
© Stonebranch 2022. All rights reserved.
© Stonebranch 2022. All rights reserved.
Data Pipeline Orchestration Solution
Universal Automation
Center
24
Real Time Hybrid IT Automation
25
© Stonebranch 2022. All rights reserved.
Universal Automation Center Platform
A Platform Approach
Orchestrating IT processes from on-prem,
to cloud, to containerized microservices
Find. Deploy. Extend.
• Download extensions
• Share extensions
• Community driven
• Constant additions (monthly)
• Large Data Pipeline Focus
• Rapid creation of new integrations
Orchestration = Integration
26
© Stonebranch 2022. All rights reserved.
What to Look for in a Data Pipeline Orchestration Solution
27
© Stonebranch 2022. All rights reserved.
Summary
Who is this for?
• Want to keep using existing data tools, but are ready to graduate from opensource
schedulers to enterprise grade platforms
• Would like a single platform to connect Data Teams, Developers, IT Ops, and Cloud Ops
teams – to help scale their data program
• Need to operationalize DataOps methodologies to gain speed and improve data quality
• Want to gain full visibility across the entire pipeline – to move quickly when issue arise
• Have a growing or changing data tool landscape, and need the ability to rapidly build
new integrations (or download pre-existing integrations)
• Need to enable data scientists or business users with simple self-service capabilities
via the platform or third-party tools like ServiceNow, Microsoft Teams, or Slack
• Bonus: Want a central IT automation and orchestration platform (beyond data pipeline
orchestration) to support cloud automation, on-prem automation, traditional job
scheduling, and DevOps orchestration
© Stonebranch 2022. All rights reserved. 28
© Stonebranch 2022. All rights reserved. 29
Q & A
Scott Davis
Global Vice President
scott.davis@stonebranch.com
Stonebranch - Atlanta, USA
Ravi Murugesan
Sr. Solution Engineer
ravi.murugesan@stonebranch.com
Stonebranch – Frankfurt, Germany
Thank You
© Stonebranch 2022. All rights reserved.

More Related Content

What's hot

Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Dr. Arif Wider
 
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
DATAVERSITY
 
Data mesh
Data meshData mesh
Data mesh
ManojKumarR41
 
How a Semantic Layer Makes Data Mesh Work at Scale
How a Semantic Layer Makes  Data Mesh Work at ScaleHow a Semantic Layer Makes  Data Mesh Work at Scale
How a Semantic Layer Makes Data Mesh Work at Scale
DATAVERSITY
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4
Databricks
 
Data Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital TransformationData Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital Transformation
DATAVERSITY
 
Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3
Jeffrey T. Pollock
 
Data Governance Best Practices
Data Governance Best PracticesData Governance Best Practices
Data Governance Best Practices
DATAVERSITY
 
Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)
James Serra
 
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
DataScienceConferenc1
 
DataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven OrganizationsDataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven Organizations
Ellen Friedman
 
Data Quality Best Practices
Data Quality Best PracticesData Quality Best Practices
Data Quality Best Practices
DATAVERSITY
 
Introducing Databricks Delta
Introducing Databricks DeltaIntroducing Databricks Delta
Introducing Databricks Delta
Databricks
 
Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?
DATAVERSITY
 
Building Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft AzureBuilding Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft Azure
Dmitry Anoshin
 
Data Warehousing Trends, Best Practices, and Future Outlook
Data Warehousing Trends, Best Practices, and Future OutlookData Warehousing Trends, Best Practices, and Future Outlook
Data Warehousing Trends, Best Practices, and Future Outlook
James Serra
 
Data Architecture Best Practices for Advanced Analytics
Data Architecture Best Practices for Advanced AnalyticsData Architecture Best Practices for Advanced Analytics
Data Architecture Best Practices for Advanced Analytics
DATAVERSITY
 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data Architecture
Databricks
 
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Tristan Baker
 
Gartner: Master Data Management Functionality
Gartner: Master Data Management FunctionalityGartner: Master Data Management Functionality
Gartner: Master Data Management Functionality
Gartner
 

What's hot (20)

Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
 
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
 
Data mesh
Data meshData mesh
Data mesh
 
How a Semantic Layer Makes Data Mesh Work at Scale
How a Semantic Layer Makes  Data Mesh Work at ScaleHow a Semantic Layer Makes  Data Mesh Work at Scale
How a Semantic Layer Makes Data Mesh Work at Scale
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4
 
Data Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital TransformationData Architecture Strategies: Data Architecture for Digital Transformation
Data Architecture Strategies: Data Architecture for Digital Transformation
 
Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3
 
Data Governance Best Practices
Data Governance Best PracticesData Governance Best Practices
Data Governance Best Practices
 
Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)
 
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
 
DataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven OrganizationsDataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven Organizations
 
Data Quality Best Practices
Data Quality Best PracticesData Quality Best Practices
Data Quality Best Practices
 
Introducing Databricks Delta
Introducing Databricks DeltaIntroducing Databricks Delta
Introducing Databricks Delta
 
Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?Data Warehouse or Data Lake, Which Do I Choose?
Data Warehouse or Data Lake, Which Do I Choose?
 
Building Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft AzureBuilding Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft Azure
 
Data Warehousing Trends, Best Practices, and Future Outlook
Data Warehousing Trends, Best Practices, and Future OutlookData Warehousing Trends, Best Practices, and Future Outlook
Data Warehousing Trends, Best Practices, and Future Outlook
 
Data Architecture Best Practices for Advanced Analytics
Data Architecture Best Practices for Advanced AnalyticsData Architecture Best Practices for Advanced Analytics
Data Architecture Best Practices for Advanced Analytics
 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data Architecture
 
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
 
Gartner: Master Data Management Functionality
Gartner: Master Data Management FunctionalityGartner: Master Data Management Functionality
Gartner: Master Data Management Functionality
 

Similar to Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines

Cloud-Native Data: What data questions to ask when building cloud-native apps
Cloud-Native Data: What data questions to ask when building cloud-native appsCloud-Native Data: What data questions to ask when building cloud-native apps
Cloud-Native Data: What data questions to ask when building cloud-native apps
VMware Tanzu
 
InfoSphere BigInsights
InfoSphere BigInsightsInfoSphere BigInsights
InfoSphere BigInsights
Wilfried Hoge
 
Self-Service Analytics with Guard Rails
Self-Service Analytics with Guard RailsSelf-Service Analytics with Guard Rails
Self-Service Analytics with Guard Rails
Denodo
 
2022 Trends in Enterprise Analytics
2022 Trends in Enterprise Analytics2022 Trends in Enterprise Analytics
2022 Trends in Enterprise Analytics
DATAVERSITY
 
Big Data Ready Enterprise
Big Data Ready Enterprise Big Data Ready Enterprise
Big Data Ready Enterprise
DataWorks Summit/Hadoop Summit
 
Enabling Next Gen Analytics with Azure Data Lake and StreamSets
Enabling Next Gen Analytics with Azure Data Lake and StreamSetsEnabling Next Gen Analytics with Azure Data Lake and StreamSets
Enabling Next Gen Analytics with Azure Data Lake and StreamSets
Streamsets Inc.
 
Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...
Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...
Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...
DataStax Academy
 
Simplifying Real-Time Architectures for IoT with Apache Kudu
Simplifying Real-Time Architectures for IoT with Apache KuduSimplifying Real-Time Architectures for IoT with Apache Kudu
Simplifying Real-Time Architectures for IoT with Apache Kudu
Cloudera, Inc.
 
Horses for Courses: Database Roundtable
Horses for Courses: Database RoundtableHorses for Courses: Database Roundtable
Horses for Courses: Database Roundtable
Eric Kavanagh
 
About CDAP
About CDAPAbout CDAP
About CDAP
Cask Data
 
Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Implement a Universal Data Distribution Architecture to Manage All Streaming ...Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Timothy Spann
 
Migrating from Oracle to Postgres
Migrating from Oracle to PostgresMigrating from Oracle to Postgres
Migrating from Oracle to Postgres
EDB
 
Re-Platforming Applications for the Cloud
Re-Platforming Applications for the CloudRe-Platforming Applications for the Cloud
Re-Platforming Applications for the Cloud
Carter Wickstrom
 
Hadoop and Your Enterprise Data Warehouse
Hadoop and Your Enterprise Data WarehouseHadoop and Your Enterprise Data Warehouse
Hadoop and Your Enterprise Data Warehouse
Edgar Alejandro Villegas
 
Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...
Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...
Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...
Denodo
 
Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...
Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...
Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...
MapR Technologies
 
A Successful Journey to the Cloud with Data Virtualization
A Successful Journey to the Cloud with Data VirtualizationA Successful Journey to the Cloud with Data Virtualization
A Successful Journey to the Cloud with Data Virtualization
Denodo
 
Government and Education Webinar: Optimizing Database Performance
Government and Education Webinar: Optimizing Database PerformanceGovernment and Education Webinar: Optimizing Database Performance
Government and Education Webinar: Optimizing Database Performance
SolarWinds
 
Build and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus Webinar
Build and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus WebinarBuild and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus Webinar
Build and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus Webinar
Impetus Technologies
 
CI/CD for a Data Platform
CI/CD for a Data PlatformCI/CD for a Data Platform
CI/CD for a Data Platform
Codit
 

Similar to Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines (20)

Cloud-Native Data: What data questions to ask when building cloud-native apps
Cloud-Native Data: What data questions to ask when building cloud-native appsCloud-Native Data: What data questions to ask when building cloud-native apps
Cloud-Native Data: What data questions to ask when building cloud-native apps
 
InfoSphere BigInsights
InfoSphere BigInsightsInfoSphere BigInsights
InfoSphere BigInsights
 
Self-Service Analytics with Guard Rails
Self-Service Analytics with Guard RailsSelf-Service Analytics with Guard Rails
Self-Service Analytics with Guard Rails
 
2022 Trends in Enterprise Analytics
2022 Trends in Enterprise Analytics2022 Trends in Enterprise Analytics
2022 Trends in Enterprise Analytics
 
Big Data Ready Enterprise
Big Data Ready Enterprise Big Data Ready Enterprise
Big Data Ready Enterprise
 
Enabling Next Gen Analytics with Azure Data Lake and StreamSets
Enabling Next Gen Analytics with Azure Data Lake and StreamSetsEnabling Next Gen Analytics with Azure Data Lake and StreamSets
Enabling Next Gen Analytics with Azure Data Lake and StreamSets
 
Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...
Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...
Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassa...
 
Simplifying Real-Time Architectures for IoT with Apache Kudu
Simplifying Real-Time Architectures for IoT with Apache KuduSimplifying Real-Time Architectures for IoT with Apache Kudu
Simplifying Real-Time Architectures for IoT with Apache Kudu
 
Horses for Courses: Database Roundtable
Horses for Courses: Database RoundtableHorses for Courses: Database Roundtable
Horses for Courses: Database Roundtable
 
About CDAP
About CDAPAbout CDAP
About CDAP
 
Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Implement a Universal Data Distribution Architecture to Manage All Streaming ...Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Implement a Universal Data Distribution Architecture to Manage All Streaming ...
 
Migrating from Oracle to Postgres
Migrating from Oracle to PostgresMigrating from Oracle to Postgres
Migrating from Oracle to Postgres
 
Re-Platforming Applications for the Cloud
Re-Platforming Applications for the CloudRe-Platforming Applications for the Cloud
Re-Platforming Applications for the Cloud
 
Hadoop and Your Enterprise Data Warehouse
Hadoop and Your Enterprise Data WarehouseHadoop and Your Enterprise Data Warehouse
Hadoop and Your Enterprise Data Warehouse
 
Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...
Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...
Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...
 
Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...
Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...
Hadoop in 2015: Keys to Achieving Operational Excellence for the Real-Time En...
 
A Successful Journey to the Cloud with Data Virtualization
A Successful Journey to the Cloud with Data VirtualizationA Successful Journey to the Cloud with Data Virtualization
A Successful Journey to the Cloud with Data Virtualization
 
Government and Education Webinar: Optimizing Database Performance
Government and Education Webinar: Optimizing Database PerformanceGovernment and Education Webinar: Optimizing Database Performance
Government and Education Webinar: Optimizing Database Performance
 
Build and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus Webinar
Build and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus WebinarBuild and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus Webinar
Build and Manage Hadoop & Oracle NoSQL DB Solutions- Impetus Webinar
 
CI/CD for a Data Platform
CI/CD for a Data PlatformCI/CD for a Data Platform
CI/CD for a Data Platform
 

More from DATAVERSITY

Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...
Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...
Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...
DATAVERSITY
 
Data at the Speed of Business with Data Mastering and Governance
Data at the Speed of Business with Data Mastering and GovernanceData at the Speed of Business with Data Mastering and Governance
Data at the Speed of Business with Data Mastering and Governance
DATAVERSITY
 
Exploring Levels of Data Literacy
Exploring Levels of Data LiteracyExploring Levels of Data Literacy
Exploring Levels of Data Literacy
DATAVERSITY
 
Building a Data Strategy – Practical Steps for Aligning with Business Goals
Building a Data Strategy – Practical Steps for Aligning with Business GoalsBuilding a Data Strategy – Practical Steps for Aligning with Business Goals
Building a Data Strategy – Practical Steps for Aligning with Business Goals
DATAVERSITY
 
Make Data Work for You
Make Data Work for YouMake Data Work for You
Make Data Work for You
DATAVERSITY
 
Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?
DATAVERSITY
 
Data Catalogs Are the Answer – What Is the Question?
Data Catalogs Are the Answer – What Is the Question?Data Catalogs Are the Answer – What Is the Question?
Data Catalogs Are the Answer – What Is the Question?
DATAVERSITY
 
Data Modeling Fundamentals
Data Modeling FundamentalsData Modeling Fundamentals
Data Modeling Fundamentals
DATAVERSITY
 
Showing ROI for Your Analytic Project
Showing ROI for Your Analytic ProjectShowing ROI for Your Analytic Project
Showing ROI for Your Analytic Project
DATAVERSITY
 
Is Enterprise Data Literacy Possible?
Is Enterprise Data Literacy Possible?Is Enterprise Data Literacy Possible?
Is Enterprise Data Literacy Possible?
DATAVERSITY
 
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
DATAVERSITY
 
Emerging Trends in Data Architecture – What’s the Next Big Thing?
Emerging Trends in Data Architecture – What’s the Next Big Thing?Emerging Trends in Data Architecture – What’s the Next Big Thing?
Emerging Trends in Data Architecture – What’s the Next Big Thing?
DATAVERSITY
 
Data Governance Trends - A Look Backwards and Forwards
Data Governance Trends - A Look Backwards and ForwardsData Governance Trends - A Look Backwards and Forwards
Data Governance Trends - A Look Backwards and Forwards
DATAVERSITY
 
Data Governance Trends and Best Practices To Implement Today
Data Governance Trends and Best Practices To Implement TodayData Governance Trends and Best Practices To Implement Today
Data Governance Trends and Best Practices To Implement Today
DATAVERSITY
 
2023 Trends in Enterprise Analytics
2023 Trends in Enterprise Analytics2023 Trends in Enterprise Analytics
2023 Trends in Enterprise Analytics
DATAVERSITY
 
Data Strategy Best Practices
Data Strategy Best PracticesData Strategy Best Practices
Data Strategy Best Practices
DATAVERSITY
 
Who Should Own Data Governance – IT or Business?
Who Should Own Data Governance – IT or Business?Who Should Own Data Governance – IT or Business?
Who Should Own Data Governance – IT or Business?
DATAVERSITY
 
Data Management Best Practices
Data Management Best PracticesData Management Best Practices
Data Management Best Practices
DATAVERSITY
 
MLOps – Applying DevOps to Competitive Advantage
MLOps – Applying DevOps to Competitive AdvantageMLOps – Applying DevOps to Competitive Advantage
MLOps – Applying DevOps to Competitive Advantage
DATAVERSITY
 
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...
DATAVERSITY
 

More from DATAVERSITY (20)

Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...
Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...
Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...
 
Data at the Speed of Business with Data Mastering and Governance
Data at the Speed of Business with Data Mastering and GovernanceData at the Speed of Business with Data Mastering and Governance
Data at the Speed of Business with Data Mastering and Governance
 
Exploring Levels of Data Literacy
Exploring Levels of Data LiteracyExploring Levels of Data Literacy
Exploring Levels of Data Literacy
 
Building a Data Strategy – Practical Steps for Aligning with Business Goals
Building a Data Strategy – Practical Steps for Aligning with Business GoalsBuilding a Data Strategy – Practical Steps for Aligning with Business Goals
Building a Data Strategy – Practical Steps for Aligning with Business Goals
 
Make Data Work for You
Make Data Work for YouMake Data Work for You
Make Data Work for You
 
Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?Data Catalogs Are the Answer – What is the Question?
Data Catalogs Are the Answer – What is the Question?
 
Data Catalogs Are the Answer – What Is the Question?
Data Catalogs Are the Answer – What Is the Question?Data Catalogs Are the Answer – What Is the Question?
Data Catalogs Are the Answer – What Is the Question?
 
Data Modeling Fundamentals
Data Modeling FundamentalsData Modeling Fundamentals
Data Modeling Fundamentals
 
Showing ROI for Your Analytic Project
Showing ROI for Your Analytic ProjectShowing ROI for Your Analytic Project
Showing ROI for Your Analytic Project
 
Is Enterprise Data Literacy Possible?
Is Enterprise Data Literacy Possible?Is Enterprise Data Literacy Possible?
Is Enterprise Data Literacy Possible?
 
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
 
Emerging Trends in Data Architecture – What’s the Next Big Thing?
Emerging Trends in Data Architecture – What’s the Next Big Thing?Emerging Trends in Data Architecture – What’s the Next Big Thing?
Emerging Trends in Data Architecture – What’s the Next Big Thing?
 
Data Governance Trends - A Look Backwards and Forwards
Data Governance Trends - A Look Backwards and ForwardsData Governance Trends - A Look Backwards and Forwards
Data Governance Trends - A Look Backwards and Forwards
 
Data Governance Trends and Best Practices To Implement Today
Data Governance Trends and Best Practices To Implement TodayData Governance Trends and Best Practices To Implement Today
Data Governance Trends and Best Practices To Implement Today
 
2023 Trends in Enterprise Analytics
2023 Trends in Enterprise Analytics2023 Trends in Enterprise Analytics
2023 Trends in Enterprise Analytics
 
Data Strategy Best Practices
Data Strategy Best PracticesData Strategy Best Practices
Data Strategy Best Practices
 
Who Should Own Data Governance – IT or Business?
Who Should Own Data Governance – IT or Business?Who Should Own Data Governance – IT or Business?
Who Should Own Data Governance – IT or Business?
 
Data Management Best Practices
Data Management Best PracticesData Management Best Practices
Data Management Best Practices
 
MLOps – Applying DevOps to Competitive Advantage
MLOps – Applying DevOps to Competitive AdvantageMLOps – Applying DevOps to Competitive Advantage
MLOps – Applying DevOps to Competitive Advantage
 
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...
 

Recently uploaded

Supervised Learning (Data Science).pptx
Supervised Learning  (Data Science).pptxSupervised Learning  (Data Science).pptx
Supervised Learning (Data Science).pptx
TARIKU ENDALE
 
Universidad Autónoma de Madrid degree offer diploma Transcript
Universidad Autónoma de Madrid  degree offer diploma TranscriptUniversidad Autónoma de Madrid  degree offer diploma Transcript
Universidad Autónoma de Madrid degree offer diploma Transcript
taqyea
 
NPS_Presentation_V3.pptx it is regarding National pension scheme
NPS_Presentation_V3.pptx it is regarding National pension schemeNPS_Presentation_V3.pptx it is regarding National pension scheme
NPS_Presentation_V3.pptx it is regarding National pension scheme
ASISHSABAT3
 
Machine learning _new.pptx for a presentation
Machine learning _new.pptx for a presentationMachine learning _new.pptx for a presentation
Machine learning _new.pptx for a presentation
RahulS66654
 
ch8_multiplexing cs553 st07 slide share ss
ch8_multiplexing cs553 st07 slide share ssch8_multiplexing cs553 st07 slide share ss
ch8_multiplexing cs553 st07 slide share ss
MinThetLwin1
 
Cómo hemos implementado semántica de "Exactly Once" en nuestra base de datos ...
Cómo hemos implementado semántica de "Exactly Once" en nuestra base de datos ...Cómo hemos implementado semántica de "Exactly Once" en nuestra base de datos ...
Cómo hemos implementado semántica de "Exactly Once" en nuestra base de datos ...
javier ramirez
 
PHENOMENOLOGY and Interpretive phenomenological analysis
PHENOMENOLOGY and Interpretive phenomenological analysisPHENOMENOLOGY and Interpretive phenomenological analysis
PHENOMENOLOGY and Interpretive phenomenological analysis
CharmoliApumKhrime
 
Universidad de Alcalá degree offer diploma Transcript
Universidad de Alcalá  degree offer diploma TranscriptUniversidad de Alcalá  degree offer diploma Transcript
Universidad de Alcalá degree offer diploma Transcript
taqyea
 
Sunshine Coast University diploma
Sunshine Coast University diplomaSunshine Coast University diploma
Sunshine Coast University diploma
cwavvyy
 
Universitat Oberta de Catalunya degree offer diploma Transcript
Universitat Oberta de Catalunya  degree offer diploma TranscriptUniversitat Oberta de Catalunya  degree offer diploma Transcript
Universitat Oberta de Catalunya degree offer diploma Transcript
taqyea
 
BIGPPTTTTTTTTtttttttttttttttttttttt.pptx
BIGPPTTTTTTTTtttttttttttttttttttttt.pptxBIGPPTTTTTTTTtttttttttttttttttttttt.pptx
BIGPPTTTTTTTTtttttttttttttttttttttt.pptx
RajdeepPaul47
 
Simon Fraser University degree offer diploma Transcript
Simon Fraser University  degree offer diploma TranscriptSimon Fraser University  degree offer diploma Transcript
Simon Fraser University degree offer diploma Transcript
taqyea
 
Universidad de Cádiz degree offer diploma Transcript
Universidad de Cádiz  degree offer diploma TranscriptUniversidad de Cádiz  degree offer diploma Transcript
Universidad de Cádiz degree offer diploma Transcript
taqyea
 
Niagara College degree offer diploma Transcript
Niagara College  degree offer diploma TranscriptNiagara College  degree offer diploma Transcript
Niagara College degree offer diploma Transcript
taqyea
 
Victoria University degree offer diploma Transcript
Victoria University  degree offer diploma TranscriptVictoria University  degree offer diploma Transcript
Victoria University degree offer diploma Transcript
taqyea
 
Sin Involves More Than You Might Think (We'll Explain)
Sin Involves More Than You Might Think (We'll Explain)Sin Involves More Than You Might Think (We'll Explain)
Sin Involves More Than You Might Think (We'll Explain)
sapna sharmap11
 
Universidad de Barcelona degree offer diploma Transcript
Universidad de Barcelona  degree offer diploma TranscriptUniversidad de Barcelona  degree offer diploma Transcript
Universidad de Barcelona degree offer diploma Transcript
taqyea
 
Australian Catholic University degree offer diploma Transcript
Australian Catholic University  degree offer diploma TranscriptAustralian Catholic University  degree offer diploma Transcript
Australian Catholic University degree offer diploma Transcript
taqyea
 
the unexpected potential of Dijkstra's Algorithm
the unexpected potential of Dijkstra's Algorithmthe unexpected potential of Dijkstra's Algorithm
the unexpected potential of Dijkstra's Algorithm
huseindihon
 
From Clues to Connections: How Social Media Investigators Expose Hidden Networks
From Clues to Connections: How Social Media Investigators Expose Hidden NetworksFrom Clues to Connections: How Social Media Investigators Expose Hidden Networks
From Clues to Connections: How Social Media Investigators Expose Hidden Networks
Milind Agarwal
 

Recently uploaded (20)

Supervised Learning (Data Science).pptx
Supervised Learning  (Data Science).pptxSupervised Learning  (Data Science).pptx
Supervised Learning (Data Science).pptx
 
Universidad Autónoma de Madrid degree offer diploma Transcript
Universidad Autónoma de Madrid  degree offer diploma TranscriptUniversidad Autónoma de Madrid  degree offer diploma Transcript
Universidad Autónoma de Madrid degree offer diploma Transcript
 
NPS_Presentation_V3.pptx it is regarding National pension scheme
NPS_Presentation_V3.pptx it is regarding National pension schemeNPS_Presentation_V3.pptx it is regarding National pension scheme
NPS_Presentation_V3.pptx it is regarding National pension scheme
 
Machine learning _new.pptx for a presentation
Machine learning _new.pptx for a presentationMachine learning _new.pptx for a presentation
Machine learning _new.pptx for a presentation
 
ch8_multiplexing cs553 st07 slide share ss
ch8_multiplexing cs553 st07 slide share ssch8_multiplexing cs553 st07 slide share ss
ch8_multiplexing cs553 st07 slide share ss
 
Cómo hemos implementado semántica de "Exactly Once" en nuestra base de datos ...
Cómo hemos implementado semántica de "Exactly Once" en nuestra base de datos ...Cómo hemos implementado semántica de "Exactly Once" en nuestra base de datos ...
Cómo hemos implementado semántica de "Exactly Once" en nuestra base de datos ...
 
PHENOMENOLOGY and Interpretive phenomenological analysis
PHENOMENOLOGY and Interpretive phenomenological analysisPHENOMENOLOGY and Interpretive phenomenological analysis
PHENOMENOLOGY and Interpretive phenomenological analysis
 
Universidad de Alcalá degree offer diploma Transcript
Universidad de Alcalá  degree offer diploma TranscriptUniversidad de Alcalá  degree offer diploma Transcript
Universidad de Alcalá degree offer diploma Transcript
 
Sunshine Coast University diploma
Sunshine Coast University diplomaSunshine Coast University diploma
Sunshine Coast University diploma
 
Universitat Oberta de Catalunya degree offer diploma Transcript
Universitat Oberta de Catalunya  degree offer diploma TranscriptUniversitat Oberta de Catalunya  degree offer diploma Transcript
Universitat Oberta de Catalunya degree offer diploma Transcript
 
BIGPPTTTTTTTTtttttttttttttttttttttt.pptx
BIGPPTTTTTTTTtttttttttttttttttttttt.pptxBIGPPTTTTTTTTtttttttttttttttttttttt.pptx
BIGPPTTTTTTTTtttttttttttttttttttttt.pptx
 
Simon Fraser University degree offer diploma Transcript
Simon Fraser University  degree offer diploma TranscriptSimon Fraser University  degree offer diploma Transcript
Simon Fraser University degree offer diploma Transcript
 
Universidad de Cádiz degree offer diploma Transcript
Universidad de Cádiz  degree offer diploma TranscriptUniversidad de Cádiz  degree offer diploma Transcript
Universidad de Cádiz degree offer diploma Transcript
 
Niagara College degree offer diploma Transcript
Niagara College  degree offer diploma TranscriptNiagara College  degree offer diploma Transcript
Niagara College degree offer diploma Transcript
 
Victoria University degree offer diploma Transcript
Victoria University  degree offer diploma TranscriptVictoria University  degree offer diploma Transcript
Victoria University degree offer diploma Transcript
 
Sin Involves More Than You Might Think (We'll Explain)
Sin Involves More Than You Might Think (We'll Explain)Sin Involves More Than You Might Think (We'll Explain)
Sin Involves More Than You Might Think (We'll Explain)
 
Universidad de Barcelona degree offer diploma Transcript
Universidad de Barcelona  degree offer diploma TranscriptUniversidad de Barcelona  degree offer diploma Transcript
Universidad de Barcelona degree offer diploma Transcript
 
Australian Catholic University degree offer diploma Transcript
Australian Catholic University  degree offer diploma TranscriptAustralian Catholic University  degree offer diploma Transcript
Australian Catholic University degree offer diploma Transcript
 
the unexpected potential of Dijkstra's Algorithm
the unexpected potential of Dijkstra's Algorithmthe unexpected potential of Dijkstra's Algorithm
the unexpected potential of Dijkstra's Algorithm
 
From Clues to Connections: How Social Media Investigators Expose Hidden Networks
From Clues to Connections: How Social Media Investigators Expose Hidden NetworksFrom Clues to Connections: How Social Media Investigators Expose Hidden Networks
From Clues to Connections: How Social Media Investigators Expose Hidden Networks
 

Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines

  • 1. © Stonebranch 2022. All rights reserved. ORCHESTRATE the Flow of Data Across Data Pipelines May 3, 2022 Ravi Murugesan Sr. Solution Engineer Scott Davis Global Vice President
  • 2. 2 © Stonebranch 2022. All rights reserved. DevOps Orchestration Layer 01 What is a Data Pipeline 02 How to Orchestrate a Data Pipeline 03 Data Pipeline Orchestration Demo 04 Questions and Answers 05 Agenda
  • 3. © Stonebranch 2022. All rights reserved. About Data Pipelines 3 Scott Davis Global Vice President
  • 4. 4 © Stonebranch 2022. All rights reserved. Vendor Landscape for DataOps – From Gartner Orchestrators Specialists Portfolio Cloud Service Providers Servware (Services & Software) System Integrators Integration Cataloging Governance MDM Analytics-Ready Enterprise Data Management Industrial Data Data Quality Observability Continuous Delivery Accelerators Privacy & Access Control * Based on “Gartner Data and Analytics Essentials: DataOps,” by Robert Robert Thanaraj
  • 5. 5 © Stonebranch 2022. All rights reserved. Data Pipeline: Simple View
  • 6. 6 © Stonebranch 2022. All rights reserved. Software & Tools By Stage Dashboards Looker, Tableau, Qlik, Power BI, SAP BusinessObjects Embedded Analytics Sisense, Looker, Cube.js Augmented Analytics Throughspot, Outlier, Anodot, Sisu App Frameworks Plotly Dash, Streamlit Custom Apps SMS Messages / Emails Data Science & Machine Learning Databricks, SAS, MathWork, Domino, Dataiku, DataRobot, TIBCO Software, Spark, RapidMiner, H2O.AI, AWS, GCP AI, Azure ML, IBM Watson Studio, Cloudera, Alteryx, TensorFlow, Anaconda Data Lake Databricks Delta Lake, Iceberg, Hudi, Hive Acid Data Lake within Cloud Storage AWS S3, Google Cloud Storage, HDFS, Azure Data Lake Store Data Warehouse Snowflake, BigQuery, Spark, AWS Redshift, Qubole, SAP BW, SAP DWC, Oracle ADW, Hive, Cloudera (for Hadoop) ETL (Extract, Transform, Load) Informatica, IBM, SAP Data Services, Oracle OWB, SAS, Talend, AWS Glue, Azure Data Factory, Pentaho, GCP Data Fusion Stream Data Processing ELT Kafka, Flink, Storm, GCP Pub/Sub Applications / ERP Oracle, Salesforce, SAP, ServiceNow IoT Devices / Sensors Stream Data Website & Mobile Apps Stream Data, Online Transaction Cloud Storage AWS S3, Google Cloud Storage, Azure Data Sources Data Integration & Ingestion Data Store Analyze / Computation Delivery
  • 7. How Do Enterprises Orchestrate Today? 7 © Stonebranch 2022. All rights reserved. Common Ways to Connect Data Tools Within the Pipeline Point-to-Point Integrations Custom Scripts Don’t Connect (Manual Movement)
  • 8. How Do Enterprises Orchestrate Today? 8 © Stonebranch 2022. All rights reserved. Common Ways to Connect Data Tools Within the Pipeline Point-to-Point Integrations Custom Scripts Don’t Connect (Manual Movement) Benefits of Proper Orchestration Solutions Centralized View Root-Cause Issues Proactive Support Achieve Scale
  • 9. Automation Pain Points Common Ways to Connect Data Tools Within the Pipeline Point-to-Point Integrations Custom Scripts Don’t Connect (Manual Movement) How Do Enterprises Orchestrate Today? 9 © Stonebranch 2022. All rights reserved. Benefits of Proper Orchestration Solutions Centralized View Root-Cause Issues Proactive Support Achieve Scale In-Built Schedulers Open-Source Schedulers Cloud Schedulers Legacy On-Prem Focused Schedulers Can’t schedule jobs in other tools Often batch- or time- based automation Focus on their own ecosystems Can’t automate jobs in both on-prem and cloud systems, i.e., no hybrid IT automation
  • 10. Data Pipeline Orchestration © Stonebranch 2022. All rights reserved. 10
  • 11. 11 © Stonebranch 2022. All rights reserved. Data Pipeline Orchestration How to accomplish the real-time automation and file transfers needed to manage the entire data pipeline.
  • 12. Data Pipeline Orchestration Orchestration How to accomplish the real-time automation and file transfers needed to manage the entire data pipeline. • Centrally schedule and orchestrate automated processes within each tool along the entire data pipeline • Use APIs or Agents to control the various tools used within each stage 12 © Stonebranch 2022. All rights reserved.
  • 13. Data Pipeline Orchestration Orchestration How to accomplish the real-time automation and file transfers needed to manage the entire data pipeline. • Centrally schedule and orchestrate automated processes within each tool along the entire data pipeline • Use APIs or Agents to control the various tools used within each stage What you achieve with this approach: • Observability of the logs and data for governance and security • DataOps lifecycle management (Dev-Test- Prod) - including simulations • Centralized control and visibility with visual workflows • Quickly root-cause issues with proactive alerts when something fails 13 © Stonebranch 2022. All rights reserved.
  • 14. Data Pipeline Orchestration Orchestration How to accomplish the real-time automation and file transfers needed to manage the entire data pipeline. • Centrally schedule and orchestrate automated processes within each tool along the entire data pipeline • Use APIs or Agents to control the various tools used within each stage What you achieve with this approach: • Observability of the logs and data for governance and security • DataOps lifecycle management (Dev-Test- Prod) - including simulations • Centralized control and visibility with visual workflows • Quickly root-cause issues with proactive alerts when something fails 14 © Stonebranch 2022. All rights reserved.
  • 15. Orchestration 15 © Stonebranch 2022. All rights reserved. Driven by SOAP Cloud DevOps ERP/Apps IaaS UAC SOAP META-ORCHESTRATION
  • 16. 16 © Stonebranch 2022. All rights reserved. Self-Service Automation Centralized collaboration platform for data, developers, and operations IT ops teams gain operational visibility Data teams approve and trigger automated workflows & pipelines from common business applications Data Pipeline
  • 17. Putting the Ops in DataOps 17 © Stonebranch 2022. All rights reserved. For Enterprises Ready for the Next Level of Maturity Develop/ Orchestrate Test / Simulate Production / Deploy Continuous Improvement Continuous Deployment Development Controller Production Controller
  • 18. Develop/ Orchestrate Test / Simulate Production / Deploy Continuous Improvement Continuous Deployment Development Controller Production Controller Putting the Ops in DataOps 18 © Stonebranch 2022. All rights reserved. For Enterprises Ready for the Next Level of Maturity Web GUI As Code Via in-built capabilities Promotion Options Via third-party repositories like GitHub
  • 19. Data Pipeline Orchestration Demo Ravi Murugesan Sr. Solution Engineer © Stonebranch 2022. All rights reserved. 19
  • 20. © Stonebranch 2022. All rights reserved. 20 Demonstration Update Visual Dashboard from Multiple Data Sources (both on-prem and cloud-based) Live orchestration of a data pipeline, including • Sources (cloud, on-prem, apps) • Ingestion, transformation (Informatica) • Stores (Azure blob, Snowflake) • Delivery (Tableau)
  • 21. One of the Largest Global Food & Beverage Manufacturers in the World Customer Use Case 21
  • 22. Customer Use Case: Overview One of the Largest Global Food & Beverage Manufacturers in the World Evolution & Goal • Goal: Orchestrate the full pipeline end-to-end • Objective: Identify a platform that could connect all their critical data tools Overall Strategy • On-prem to cloud digital transformation • Implemented an enterprise analytics data management environment • Hub-and-spoke model to help keep regional resource groups and services segregated • Approved services are first developed and deployed at the hub level, with further spoke deployment via containers Original Approach • Their data pipeline for the enterprise data management environment with Azure Data Factory • Azure Data Factory worked well in an Azure environment • It served as an entry point for the project • The Challenge: Data Factory did not integrate with their full stack of solutions used along the data pipeline 22 © Stonebranch 2022. All rights reserved.
  • 23. Data Pipeline Orchestration One of the Largest Global Food & Beverage Manufacturers in the World Achieving Their Goal • Secure and robust file transfer • DataOps: define pipelines as code and gain lifecycle management (test/dev/prod) capabilities • Integrate diverse data pipelines that are built using various cloud-based and on-prem services and tools • For operations: visibility into the process, improve SLAs, real-time monitoring, alerting • Unified view to design and orchestrate workflows across multiple cloud and on-prem applications Orchestration Databases 23 © Stonebranch 2022. All rights reserved.
  • 24. © Stonebranch 2022. All rights reserved. Data Pipeline Orchestration Solution Universal Automation Center 24
  • 25. Real Time Hybrid IT Automation 25 © Stonebranch 2022. All rights reserved. Universal Automation Center Platform A Platform Approach Orchestrating IT processes from on-prem, to cloud, to containerized microservices
  • 26. Find. Deploy. Extend. • Download extensions • Share extensions • Community driven • Constant additions (monthly) • Large Data Pipeline Focus • Rapid creation of new integrations Orchestration = Integration 26 © Stonebranch 2022. All rights reserved.
  • 27. What to Look for in a Data Pipeline Orchestration Solution 27 © Stonebranch 2022. All rights reserved.
  • 28. Summary Who is this for? • Want to keep using existing data tools, but are ready to graduate from opensource schedulers to enterprise grade platforms • Would like a single platform to connect Data Teams, Developers, IT Ops, and Cloud Ops teams – to help scale their data program • Need to operationalize DataOps methodologies to gain speed and improve data quality • Want to gain full visibility across the entire pipeline – to move quickly when issue arise • Have a growing or changing data tool landscape, and need the ability to rapidly build new integrations (or download pre-existing integrations) • Need to enable data scientists or business users with simple self-service capabilities via the platform or third-party tools like ServiceNow, Microsoft Teams, or Slack • Bonus: Want a central IT automation and orchestration platform (beyond data pipeline orchestration) to support cloud automation, on-prem automation, traditional job scheduling, and DevOps orchestration © Stonebranch 2022. All rights reserved. 28
  • 29. © Stonebranch 2022. All rights reserved. 29 Q & A Scott Davis Global Vice President scott.davis@stonebranch.com Stonebranch - Atlanta, USA Ravi Murugesan Sr. Solution Engineer ravi.murugesan@stonebranch.com Stonebranch – Frankfurt, Germany
  • 30. Thank You © Stonebranch 2022. All rights reserved.